summary refs log tree commit diff
path: root/tests/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Attempt to delete more duplicate rows in receipts_linearized table. (#14915)Patrick Cloke2023-02-011-1/+3
| | | | | | | | | The previous assumption was that the stream_id column was unique (for a room ID, receipt type, user ID tuple), but this turned out to be incorrect. Now find the max stream ID, then map this back to a database-specific row identifier and delete other rows which match the (room ID, receipt type, user ID) tuple, but *not* the row ID.
* Make sqlite database migrations transactional again, part two (#14926)Sean Quah2023-01-311-0/+96
| | | | | | | | | | | | | | | | #14910 fixed the regression introduced by #13873 where sqlite database migrations would no longer run inside a transaction. However, it committed the transaction before Synapse updated its bookkeeping of which migrations have been run, which means that migrations may be run again after they have completed successfully. Leave the transaction open at the end of `executescript`, to restore the old, correct behaviour. Also make the PostgreSQL behaviour consistent with SQLite. Fixes #14909. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix type hints in typing edu unit tests (#14886)Andrew Morgan2023-01-261-1/+4
|
* Skip processing stats for broken rooms. (#14873)Patrick Cloke2023-01-231-34/+54
| | | | | | | * Skip processing stats for broken rooms. * Newsfragment * Use a custom exception.
* Always notify replication when a stream advances (#14877)Erik Johnston2023-01-201-0/+4
| | | This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
* Wait for streams to catch up when processing HTTP replication. (#14820)Erik Johnston2023-01-181-11/+9
| | | | This should hopefully mitigate a class of races where data gets out of sync due a HTTP replication request racing with the replication streams.
* Re-enable some linting (#14821)Erik Johnston2023-01-121-3/+3
| | | | | | | * Re-enable some linting * Newsfile * Remove comment
* Allow selecting "prejoin" events by state keys (#14642)David Robertson2022-12-131-622/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Declare new config * Parse new config * Read new config * Don't use trial/our TestCase where it's not needed Before: ``` $ time trial tests/events/test_utils.py > /dev/null real 0m2.277s user 0m2.186s sys 0m0.083s ``` After: ``` $ time trial tests/events/test_utils.py > /dev/null real 0m0.566s user 0m0.508s sys 0m0.056s ``` * Helper to upsert to event fields without exceeding size limits. * Use helper when adding invite/knock state Now that we allow admins to include events in prejoin room state with arbitrary state keys, be a good Matrix citizen and ensure they don't accidentally create an oversized event. * Changelog * Move StateFilter tests should have done this in #14668 * Add extra methods to StateFilter * Use StateFilter * Ensure test file enforces typed defs; alphabetise * Workaround surprising get_current_state_ids * Whoops, fix mypy
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-121-1/+1
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Revert the deletion of stale devices due to performance issues. (#14662)reivilibre2022-12-121-3/+1
|
* Add optional ICU support for user search (#14464)Brendan Abolivier2022-12-121-0/+43
| | | | | | | Fixes #13655 This change uses ICU (International Components for Unicode) to improve boundary detection in user search. This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
* Require types in tests.storage. (#14646)Patrick Cloke2022-12-0933-330/+483
| | | | Adds missing type hints to `tests.storage` package and does not allow untyped definitions.
* Delete stale non-e2e devices for users, take 2 (#14595)Erik Johnston2022-12-091-1/+3
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Fix a long-standing bug where the user directory would return 1 more row ↵reivilibre2022-12-071-0/+6
| | | | than requested. (#14631)
* Merge branch 'release-v1.73' into developDavid Robertson2022-12-011-3/+1
|\
| * Revert "POC delete stale non-e2e devices for users (#14038)" (#14582)David Robertson2022-11-291-3/+1
| |
* | Aggregate unread notif count query for badge count calculation (#14255)Nick Mills-Barrett2022-11-301-8/+39
|/ | | | | | Fetch the unread notification counts used by the badge counts in push notifications for all rooms at once (instead of fetching them per room).
* POC delete stale non-e2e devices for users (#14038)Erik Johnston2022-11-291-1/+3
| | | | | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Remove legacy Prometheus metrics names. They were deprecated in Synapse ↵reivilibre2022-11-241-3/+4
| | | | v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
* Track unconverted device list outbound pokes using a position instead (#14516)Sean Quah2022-11-221-2/+1
| | | | | | | | | | | | | | | | | | | | | When a local device list change is added to `device_lists_changes_in_room`, the `converted_to_destinations` flag is set to `FALSE` and the `_handle_new_device_update_async` background process is started. This background process looks for unconverted rows in `device_lists_changes_in_room`, copies them to `device_lists_outbound_pokes` and updates the flag. To update the `converted_to_destinations` flag, the database performs a `DELETE` and `INSERT` internally, which fragments the table. To avoid this, track unconverted rows using a `(stream ID, room ID)` position instead of the flag. From now on, the `converted_to_destinations` column indicates rows that need converting to outbound pokes, but does not indicate whether the conversion has already taken place. Closes #14037. Signed-off-by: Sean Quah <seanq@matrix.org>
* Reintroduce #14376, with bugfix for monoliths (#14468)David Robertson2022-11-161-10/+152
| | | | | | | | | | | | | | | | | | | | | | * Add tests for StreamIdGenerator * Drive-by: annotate all defs * Revert "Revert "Remove slaved id tracker (#14376)" (#14463)" This reverts commit d63814fd736fed5d3d45ff3af5e6d3bfae50c439, which in turn reverted 36097e88c4da51fce6556a58c49bd675f4cf20ab. This restores the latter. * Fix StreamIdGenerator not handling unpersisted IDs Spotted by @erikjohnston. Closes #14456. * Changelog Co-authored-by: Nick Mills-Barrett <nick@fizzadar.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix background updates failing to add unique indexes on receipts (#14453)Sean Quah2022-11-161-0/+209
| | | | | | | | | | | | | | | | | | | | | | As part of the database migration to support threaded receipts, there is a possible window in between `73/08thread_receipts_non_null.sql.postgres` removing the original unique constraints on `receipts_linearized` and `receipts_graph` and the `reeipts_linearized_unique_index` and `receipts_graph_unique_index` background updates from `72/08thread_receipts.sql` completing where the unique constraints on `receipts_linearized` and `receipts_graph` are missing. Any emulated upserts on these tables must therefore be performed with a lock held, otherwise duplicate rows can end up in the tables when there are concurrent emulated upserts. Fix the missing lock. Note that emulated upserts no longer happen by default on sqlite, since the minimum supported version of sqlite supports native upserts by default now. Finally, clean up any duplicate receipts that may have crept in before trying to create the `receipts_graph_unique_index` and `receipts_linearized_unique_index` unique indexes. Signed-off-by: Sean Quah <seanq@matrix.org>
* Drop support for Postgres 10 in full text search code. (#14397)Patrick Cloke2022-11-091-52/+17
|
* Fix tests for change in PostgreSQL 14 behavior change. (#14310)Patrick Cloke2022-10-271-4/+12
| | | | | | | PostgreSQL 14 changed the behavior of `websearch_to_tsquery` to improve some behaviour. The tests were hitting those edge-cases about handling of hanging double quotes. This fixes the tests to take into account the PostgreSQL version.
* Unified search query syntax using the full-text search capabilities of the ↵James Salter2022-10-251-0/+213
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | underlying DB. (#11635) Support a unified search query syntax which leverages more of the full-text search of each database supported by Synapse. Supports, with the same syntax across Postgresql 11+ and Sqlite: - quoted "search terms" - `AND`, `OR`, `-` (negation) operators - Matching words based on their stem, e.g. searches for "dog" matches documents containing "dogs". This is achieved by - If on postgresql 11+, pass the user input to `websearch_to_tsquery` - If on sqlite, manually parse the query and transform it into the sqlite-specific query syntax. Note that postgresql 10, which is close to end-of-life, falls back to using `phraseto_tsquery`, which only supports a subset of the features. Multiple terms separated by a space are implicitly ANDed. Note that: 1. There is no escaping of full-text syntax that might be supported by the database; e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this as accidental functionality and depend on something we don't guarantee. 2. English text is assumed for stemming. To support other languages, either the target language needs to be known at the time of indexing the message (via room metadata, or otherwise), or a separate index for each language supported could be created. Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
* Remove `_get_events_cache` check optimisation from `_have_seen_events_dict` ↵Andrew Morgan2022-10-181-12/+0
| | | | (#14161)
* Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke2022-10-171-33/+85
| | | Gated behind an experimental configuration flag.
* Stop getting missing `prev_events` after we already know their signature is ↵Eric Eastwood2022-10-151-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | invalid (#13816) While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place. Related to - https://github.com/matrix-org/synapse/issues/13622 - https://github.com/matrix-org/synapse/pull/13635 - https://github.com/matrix-org/synapse/issues/13676 Part of https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures. With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now. For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761 To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid. - `backfill` - `outgoing-federation-request` `/backfill` - `_check_sigs_and_hash_and_fetch` - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill - ❗ `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - ❗ `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - `_process_pulled_events` - `_process_pulled_event` for each validated event - ❗ Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it - `_get_state_ids_after_missing_prev_event` - `outgoing-federation-request` `/state_ids` - ❗ `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again - ❗ `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
* Accept threaded receipts for events related to the root event. (#14174)Patrick Cloke2022-10-141-0/+111
| | | | | | | | | The root node of a thread (and events related to it) are considered "part of a thread" when validating receipts. This allows clients which show the root node in both the main timeline and the threaded timeline to easily send receipts in either. Note that threaded notifications are not created for these events, these events created notifications on the main timeline.
* Use threaded receipts when fetching events for push. (#13878)Patrick Cloke2022-10-041-18/+39
| | | | Update the HTTP and email pushers to consider threaded read receipts when fetching unread events.
* Recursively fetch the thread for receipts & notifications. (#13824)Patrick Cloke2022-10-041-0/+100
| | | | | | Consider an event to be part of a thread if you can follow a chain of relations up to a thread root. Part of MSC3773 & MSC3771.
* Mark events as read using threaded read receipts from MSC3771. (#13877)Patrick Cloke2022-10-041-1/+188
| | | | Applies the proper logic for unthreaded and threaded receipts to either apply to all events in the room or only events in the same thread, respectively.
* Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke2022-10-041-2/+167
| | | | | | | | When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
* Update mypy and mypy-zope, attempt 3 (#13993)David Robertson2022-09-301-1/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Revert "Update mypy and mypy-zope (#13925)"David Robertson2022-09-301-6/+0
| | | | This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
* Update mypy and mypy-zope (#13925)David Robertson2022-09-301-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix overflows in /messages backfill calculation (#13936)David Robertson2022-09-301-12/+49
| | | | | | | | * Reproduce bug * Compute `least_function` first * Substitute `least_function` with an f-string * Bugfix: avoid overflow Co-authored-by: Eric Eastwood <erice@element.io>
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-291-1/+101
| | | | used (using MSC3866) (#13556)
* Clarify that a method returns only unthreaded receipts. (#13937)Patrick Cloke2022-09-291-36/+38
| | | | | By renaming it and updating the docstring. Additionally, refactors a method which is used only by tests.
* Limit and filter the number of backfill points to get from the database (#13879)Eric Eastwood2022-09-281-22/+58
| | | | | | | | | There is no need to grab thousands of backfill points when we only need 5 to make the `/backfill` request with. We need to grab a few extra in case the first few aren't visible in the history. Previously, we grabbed thousands of backfill points from the database, then sorted and filtered them in the app. Fetching the 4.6k backfill points for `#matrix:matrix.org` from the database takes ~50ms - ~570ms so it's not like this saves a lot of time 🤷. But it might save us more time now that `get_backfill_points_in_room`/`get_insertion_event_backward_extremities_in_room` are more complicated after https://github.com/matrix-org/synapse/pull/13635 This PR moves the filtering and limiting to the SQL query so we just have less data to work with in the first place. Part of https://github.com/matrix-org/synapse/issues/13356
* Persist CreateRoom events to DB in a batch (#13800)Shay2022-09-281-2/+6
|
* Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood2022-09-271-48/+104
| | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
* Improve tests for get_unread_push_actions_for_user_in_range_*. (#13893)Patrick Cloke2022-09-261-16/+72
| | | | | * Adds a docstring. * Reduces a small amount of duplicated code. * Improves tests.
* Only try to backfill event if we haven't tried before recently (#13635)Eric Eastwood2022-09-231-4/+477
| | | | | | | | | | Only try to backfill event if we haven't tried before recently (exponential backoff). No need to keep trying the same backfill point that fails over and over. Fix https://github.com/matrix-org/synapse/issues/13622 Fix https://github.com/matrix-org/synapse/issues/8451 Follow-up to https://github.com/matrix-org/synapse/pull/13589 Part of https://github.com/matrix-org/synapse/issues/13356
* Accept & store thread IDs for receipts (implement MSC3771). (#13782)Patrick Cloke2022-09-232-8/+29
| | | | Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.
* Require SQLite >= 3.27.0 (#13760)David Robertson2022-09-091-1/+0
|
* Add timestamp to user's consent (#13741)Dirk Klimpel2022-09-081-7/+26
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Rename the `EventFormatVersions` enum values so that they line up with room ↵reivilibre2022-09-072-2/+2
| | | | version numbers. (#13706)
* Disable calculating unread counts unless the config flag is enabled. (#13694)Patrick Cloke2022-09-011-22/+20
| | | | | | | | This avoids doing work that will never be used (since the resulting unread counts will never be sent in a /sync response). The negative of doing this is that unread counts will be incorrect when the feature is initially enabled.
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-22/+12
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel2022-08-301-2/+2
|
* Add forgotten status to Room Details API (#13503)Dirk Klimpel2022-08-171-0/+70
|
* Strengthen tests about deleted old push actions. (#13471)Patrick Cloke2022-08-091-0/+15
|
* Support stable identifiers for MSC2285: private read receipts. (#13273)Šimon Brandner2022-08-051-17/+38
| | | | | This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-1/+6
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-211-0/+261
|
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-211-5/+5
|
* Use a real room in the notification rotation tests. (#13260)Patrick Cloke2022-07-151-116/+79
| | | | Instead of manually inserting fake data. This fixes some issues with having to manually calculate stream orderings and other oddities.
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-152-5/+5
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Don't pull out state in `compute_event_context` for unconflicted state (#13267)Erik Johnston2022-07-141-55/+0
|
* Fix notification count after a highlighted message (#13223)Erik Johnston2022-07-081-0/+7
| | | | | Fixes #13196 Broke by #13005
* Remove obsolete RoomEventsStoreTestCase (#13200)Petr Vaněk2022-07-071-69/+0
| | | | | | | | | | All tests are prefixed with `STALE_` and therefore they are silently skipped. They were moved to `STALE_` in version `v0.5.0` in commit 2fcce3b3c508 - `Remove stale tests`. Tests from `RoomEventsStoreTestCase` class are not used for last 8 years, I believe the best would be to remove them entirely. Signed-off-by: Petr Vaněk <arkamar@atlas.cz>
* Merge tag 'v1.62.0rc3' into developAndrew Morgan2022-07-041-5/+5
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.62.0rc3 (2022-07-04) ============================== Bugfixes -------- - Update the version of the [ldap3 plugin](https://github.com/matrix-org/matrix-synapse-ldap3/) included in the `matrixdotorg/synapse` DockerHub images and the Debian packages hosted on `packages.matrix.org` to 0.2.1. This fixes [a bug](https://github.com/matrix-org/matrix-synapse-ldap3/pull/163) with usernames containing uppercase characters. ([\#13156](https://github.com/matrix-org/synapse/issues/13156)) - Fix a bug introduced in Synapse 1.62.0rc1 affecting unread counts for users on small servers. ([\#13168](https://github.com/matrix-org/synapse/issues/13168))
| * Fix stuck notification counts on small servers (#13168)Erik Johnston2022-07-041-5/+5
| |
* | Merge remote-tracking branch 'origin/release-v1.62' into developPatrick Cloke2022-06-301-7/+5
|\|
| * Fix unread counts on large servers (#13140)Erik Johnston2022-06-301-7/+5
| |
* | Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)Šimon Brandner2022-06-291-0/+69
| | | | | | | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* | Improve performance of getting unread counts in rooms (#13119)Erik Johnston2022-06-291-0/+2
|/
* Fix serialization errors when rotating notifications (#13118)Erik Johnston2022-06-281-5/+30
|
* Add type hints to event push actions tests. (#13099)Patrick Cloke2022-06-171-11/+17
|
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-151-3/+21
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-063-13/+21
|
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-011-1/+58
|
* Remove remaining bits of groups code. (#12936)Patrick Cloke2022-06-011-1/+0
| | | | | | * Update worker docs to remove group endpoints. * Removes an unused parameter to `ApplicationService`. * Break dependency between media repo and groups. * Avoid copying `m.room.related_groups` state events during room upgrades.
* Rename storage classes (#12913)Erik Johnston2022-05-317-22/+31
|
* LockStore: fix acquiring a lock via `LockStore.try_acquire_lock` (#12832)Sumner Evans2022-05-301-0/+54
| | | Signed-off-by: Sumner Evans <sumner@beeper.com>
* Additional constants for EDU types. (#12884)Patrick Cloke2022-05-271-2/+5
| | | Instead of hard-coding strings in many places.
* Add storage and module API methods to get monthly active users and their ↵Matt C2022-05-271-0/+83
| | | | appservices (#12838)
* Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-15/+28
|
* Add some type hints to tests files (#12833)Dirk Klimpel2022-05-232-2/+2
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Remove code which updates `application_services_state.last_txn` (#12680)Richard van der Hoff2022-05-171-21/+6
| | | | This column is unused as of #12209, so let's stop writing to it.
* Track in memory events using weakrefs (#10533)Erik Johnston2022-05-171-0/+25
|
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-1/+1
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* Stop writing to `event_reference_hashes` (#12679)Richard van der Hoff2022-05-101-9/+0
| | | | | This table is never read, since #11794. We stop writing to it; in future we can drop it altogether.
* Update mypy to 0.950 and fix complaints (#12650)David Robertson2022-05-061-5/+5
|
* Add a consistency check on events read from the database (#12620)Richard van der Hoff2022-05-031-22/+37
| | | | | | | I've seen a few errors which can only plausibly be explained by the calculated event id for an event being different from the ID of the event in the database. It should be cheap to check this, so let's do so and raise an exception.
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-271-5/+4
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Mark remote device list updates as already handled (#12557)Erik Johnston2022-04-261-1/+1
|
* Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db` (#12529)Sean Quah2022-04-251-2/+119
| | | | | | | | Multiple calls to `EventsWorkerStore._get_events_from_cache_or_db` can reuse the same database fetch, which is initiated by the first call. Ensure that cancelling the first call doesn't cancel the other calls sharing the same database fetch. Signed-off-by: Sean Quah <seanq@element.io>
* Await un-partial-stating after a partial-state join (#12399)Richard van der Hoff2022-04-212-0/+130
| | | | | | When we join a room via the faster-joins mechanism, we end up with "partial state" at some points on the event DAG. Many parts of the codebase need to wait for the full state to load. So, we implement a mechanism to keep track of which events have partial state, and wait for them to be fully-populated.
* Only send out device list updates for our own users (#12465)Erik Johnston2022-04-141-3/+3
| | | Broke in #12365
* Process device list updates asynchronously (#12365)Erik Johnston2022-04-121-20/+27
|
* Optimise `_update_client_ips_batch_txn` to batch together database ↵reivilibre2022-04-081-16/+57
| | | | | operations. (#12252) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-1/+3
| | | | provided (#12370)
* Add type hints to some tests files (#12371)Dirk Klimpel2022-04-053-36/+50
|
* Track device list updates per room. (#12321)Erik Johnston2022-04-041-4/+10
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Add type hints for `tests/unittest.py`. (#12347)Richard van der Hoff2022-04-012-4/+5
| | | In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-013-51/+27
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Default to `private` room visibility rather than `public` when a client does ↵reivilibre2022-04-011-1/+3
| | | | not specify one, according to spec. (#12350)
* Use a sequence to generate AS transaction IDs, drop `last_txn` AS state (#12209)Nick Mills-Barrett2022-04-011-69/+2
| | | | | | | | Switching to a sequence means there's no need to track `last_txn` on the AS state table to generate new TXN IDs. This also means that there is no longer contention between the AS scheduler and AS handler on updates to the `application_services_state` table, which will prevent serialization errors during the complete AS txn transaction.
* Move single-use methods out of `TestCase` (#12348)Richard van der Hoff2022-04-011-0/+16
| | | | These methods are only used by a single testcase, so they shouldn't be cluttering up the base `TestCase` class.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-4/+13
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Start application service stream token tracking from 1 (#12193)Andrew Morgan2022-03-301-2/+2
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Refuse to start if DB has an unsafe locale (#12262)Shay2022-03-231-0/+46
|
* Add type hints to tests files. (#12256)Dirk Klimpel2022-03-212-60/+68
|
* Handle cancellation in `DatabasePool.runInteraction()` (#12199)Sean Quah2022-03-161-0/+58
| | | | | | | | | | | | To handle cancellation, we ensure that `after_callback`s and `exception_callback`s are always run, since the transaction will complete on another thread regardless of cancellation. We also wait until everything is done before releasing the `CancelledError`, so that logging contexts won't get used after they have been finished. Signed-off-by: Sean Quah <seanq@element.io>
* Use the ignored_users table to test event visibility & sync. (#12225)Patrick Cloke2022-03-151-0/+17
| | | | | Instead of fetching the raw account data and re-parsing it. The ignored_users table is a denormalised version of the account data for quick searching.
* Add tests for database transaction callbacks (#12198)Sean Quah2022-03-151-1/+103
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add config settings for background update parameters (#11980)Shay2022-03-111-16/+237
|
* Support stable identifiers for MSC3440: Threading (#12151)Patrick Cloke2022-03-101-11/+9
| | | | The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.
* Remove dead code in `tests/storage/test_database.py` (#12197)Sean Quah2022-03-091-16/+0
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Fix a bug in background updates wherein background updates are never run ↵Shay2022-03-071-2/+2
| | | | using the default batch size (#12157)
* Back out in-flight state caching changes. (#12126)reivilibre2022-03-021-454/+0
|
* Order in-flight state group queries in biggest-first order (#11610)reivilibre2022-03-011-1/+103
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-2811-69/+69
|
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-241-4/+4
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix non-strings in the `event_search` table (#12037)Sean Quah2022-02-241-2/+115
| | | | | | | Don't attempt to add non-string `value`s to `event_search` and add a background update to clear out bad rows from `event_search` when using sqlite. Signed-off-by: Sean Quah <seanq@element.io>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-2334-52/+52
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Cap the number of in-flight requests for state from a single group (#11608)reivilibre2022-02-221-0/+69
|
* Add more tests for in-flight state query duplication. (#12033)reivilibre2022-02-221-21/+171
|
* Track and deduplicate in-flight requests to `_get_state_for_groups`. (#10870)reivilibre2022-02-181-0/+133
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix bug in `StateFilter.return_expanded()` and add some tests. (#12016)reivilibre2022-02-181-0/+109
|
* Fix incorrect `get_rooms_for_user` for remote user (#11999)Erik Johnston2022-02-151-0/+107
| | | | | | | When the server leaves a room the `get_rooms_for_user` cache is not correctly invalidated for the remote users in the room. This means that subsequent calls to `get_rooms_for_user` for the remote users would incorrectly include the room (it shouldn't be included because the server no longer knows anything about the room).
* Implement MSC3706: partial state in `/send_join` response (#11967)Richard van der Hoff2022-02-121-4/+4
| | | | | | | | | | | | * Make `get_auth_chain_ids` return a Set It has a set internally, and a set is often useful where it gets used, so let's avoid converting to an intermediate list. * Minor refactors in `on_send_join_request` A little bit of non-functional groundwork * Implement MSC3706: partial state in /send_join response
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-1/+3
| | | | helper. (#11615)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-9/+17
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.51.0rc2' into developAndrew Morgan2022-01-241-6/+24
|\ | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc2 (2022-01-24) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
| * Fix logic for dropping old events in fed queue (#11806)Andrew Morgan2022-01-241-6/+24
| | | | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-211-1/+4
|/ | | | ... and start populating them for new events
* Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-141-1/+159
|\
| * Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre2022-01-131-1/+111
| | | | | | | | | | fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
| * Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre2022-01-121-1/+49
| | | | | | | | | | be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-131-10/+16
|/ | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre2021-12-291-2/+2
| | | | | exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Refactor `tests.util.setup_test_homeserver` and ↵reivilibre2021-12-212-2/+3
| | | | `tests.server.setup_test_homeserver`. (#11503)
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-5/+7
|
* Use mock from standard library (#11588)V024602021-12-201-9/+8
| | | Instead of the backported version.
* Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-1/+3
|
* Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-141-1/+1
| | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* Revert accidental commits to develop.Olivier Wilkinson (reivilibre)2021-12-032-3/+2
|
* Move `tests.utils.setup_test_homeserver` to `tests.server`Olivier Wilkinson (reivilibre)2021-12-032-2/+3
| | | | | | | It had no users. We have just taken the identity of a previous function but don't provide the same behaviour, so we need to fix this in the next commit...
* Clean up tests.storage.test_appservice (#11492)reivilibre2021-12-021-177/+168
|
* Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)reivilibre2021-12-021-17/+10
|
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-021-48/+92
|
* Add a license header and comment. (#11479)Patrick Cloke2021-12-011-0/+15
|
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-293-16/+97
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-1/+138
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-2/+2
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-4/+4
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix incorrect return value in tests. (#11359)Patrick Cloke2021-11-161-1/+1
|
* Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+51
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-1/+51
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-3/+6
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-0/+207
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-0/+69
| | | Fixes #11252
* Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+74
|
* Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-011-0/+206
| | | | (#11179)
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+90
| | | Fixes: #9346
* Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-44/+33
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-133-3/+570
| | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-062-2/+2
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-066-60/+10
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-16/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-0/+34
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-0/+34
| |
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-041-2/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Refactor user directory tests (#10935)David Robertson2021-09-301-4/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|/
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+0
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-232-10/+6
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-17/+29
| | | Also enables Mypy for related tests.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-0/+1
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+98
| | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-177-7/+7
|
* Improve event caching code (#10119)Erik Johnston2021-08-041-0/+50
| | | | Ensure we only load an event from the DB once when the same event is requested multiple times at once.
* Fix codestyle CI from #10440 (#10511)reivilibre2021-08-021-1/+1
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Prune inbound federation queues if they get too long (#10390)Erik Johnston2021-08-021-0/+57
|
* Allow setting transaction limit for db connections (#10440)Toni Spets2021-08-021-0/+36
| | | | | | | Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
* Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-261-3/+3
| | | | (#10254)
* Fix dropping locks on shut down (#10433)Erik Johnston2021-07-201-0/+13
|
* Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-132-6/+4
| | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-134-13/+5
|
* Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+100
| | | This adds a simple best effort locking mechanism that works cross workers.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-2/+11
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-013-0/+122
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-2/+6
|
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-071-3/+1
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+2
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-2/+2
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1429-29/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-097-11/+7
|
* Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel2021-04-091-2/+2
| | | | | | | | | | | | | Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-8/+16
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Merge remote-tracking branch 'origin/develop' into rav/drop_py35Richard van der Hoff2021-04-0810-499/+264
|\
| * Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke2021-04-0610-499/+264
| |
* | remove unused param on `make_tuple_comparison_clause`Richard van der Hoff2021-04-081-2/+1
| |
* | Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-081-11/+1
|/
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-14/+8
|
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+73
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* Add a basic test for purging rooms. (#9541)Patrick Cloke2021-03-081-26/+45
| | | | | Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
* Add the shadow-banning status to the display user admin API. (#9400)Dirk Klimpel2021-02-171-0/+1
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1612-47/+89
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-6/+106
|
* Make chain cover index bg update go faster (#9124)Erik Johnston2021-01-151-31/+186
| | | | | | We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.
* Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-0/+114
|
* Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-121-0/+26
| | | This only applies if the user's data is to be erased.
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-112-24/+697
|
* Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-0/+120
| | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-0/+334
| | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* Use the simple dictionary in full text search for the user directory (#8959)Brendan Abolivier2020-12-171-0/+23
| | | | | * Use the simple dictionary in fts for the user directory * Clarify naming
* Make search statement in List Room and User Admin API case-insensitive (#8931)Dirk Klimpel2020-12-171-0/+7
|
* Add number of local devices to Room Details Admin API (#8886)Dirk Klimpel2020-12-111-0/+26
|
* Pass room_id to get_auth_chain_difference (#8879)Erik Johnston2020-12-041-8/+10
| | | | | This is so that we can choose which algorithm to use based on the room ID.
* Remove redundant mockingRichard van der Hoff2020-12-022-16/+3
|
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-024-4/+4
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Correctly handle unpersisted events when calculating auth chain difference. ↵Erik Johnston2020-12-021-0/+5
| | | | | | | | | (#8827) We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.
* Remove deprecated `/_matrix/client/*/admin` endpoints (#8785)Dirk Klimpel2020-11-251-1/+1
| | | These are now only available via `/_synapse/admin/v1`.
* Merge pull request #8761 from matrix-org/rav/test_request_renderingRichard van der Hoff2020-11-171-2/+1
|\ | | | | Make `make_request` actually render the request
| * Remove redundant calls to `render()`Richard van der Hoff2020-11-161-2/+1
| |
* | Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-30/+0
|/
* fixup testRichard van der Hoff2020-11-161-0/+1
|
* Add a `custom_headers` param to `make_request` (#8760)Richard van der Hoff2020-11-161-6/+7
| | | | Some tests want to set some custom HTTP request headers, so provide a way to do that before calling requestReceived().
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-6/+4
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-0/+56
|
* Don't instansiate Requester directly (#8614)Erik Johnston2020-10-223-7/+7
|
* Combine the two sets of tests for CacheDescriptorRichard van der Hoff2020-10-211-228/+0
|
* Send some ephemeral events to appservices (#8437)Will Hunt2020-10-151-4/+4
| | | Optionally sends typing, presence, and read receipt information to appservices.
* Merge pull request #8548 from matrix-org/rav/deferred_cacheRichard van der Hoff2020-10-151-72/+1
|\ | | | | Rename Cache to DeferredCache, and related changes
| * Combine the two sets of DeferredCache testsRichard van der Hoff2020-10-141-72/+0
| |
| * move DeferredCache into its own moduleRichard van der Hoff2020-10-141-1/+2
| |
| * Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-5/+5
| |
* | Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-2/+2
|/ | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
* Make event persisters periodically announce position over replication. (#8499)Erik Johnston2020-10-121-9/+16
| | | | | Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress. This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
* Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-071-0/+1
| | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-5/+9
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Make token serializing/deserializing async (#8427)Erik Johnston2020-09-301-3/+6
| | | The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
* Rewrite BucketCollectorRichard van der Hoff2020-09-301-8/+11
| | | | | | | | | | | | This was a bit unweildy for what I wanted: in particular, I wanted to assign each measurement straight into a bucket, rather than storing an intermediate Counter which didn't do any bucketing at all. I've replaced it with something that is hopefully a bit easier to use. (I'm not entirely sure what the difference between a HistogramMetricFamily and a GaugeHistogramMetricFamily is, but given our counters can go down as well as up the latter *sounds* more accurate?)
* Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-6/+4
|
* Don't table scan events on worker startup (#8419)Erik Johnston2020-09-291-0/+18
| | | | | | | | | | | | | | | | | | | | * Fix table scan of events on worker startup. This happened because we assumed "new" writers had an initial stream position of 0, so the replication code tried to fetch all events written by the instance between 0 and the current position. Instead, set the initial position of new writers to the current persisted up to position, on the assumption that new writers won't have written anything before that point. * Consider old writers coming back as "new". Otherwise we'd try and fetch entries between the old stale token and the current position, even though it won't have written any rows. Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Filter out appservices from mau count (#8404)Will Hunt2020-09-291-1/+16
| | | This is an attempt to fix #8403.
* Add checks for postgres sequence consistency (#8402)Erik Johnston2020-09-281-3/+19
|
* Fix MultiWriteIdGenerator's handling of restarts. (#8374)Erik Johnston2020-09-241-8/+111
| | | | | | | | | | | | | | | | | | | On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.
* Use `async with` for ID gens (#8383)Erik Johnston2020-09-231-25/+41
| | | This will allow us to hit the DB after we've finished using the generated stream ID.
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Fix `MultiWriterIdGenerator.current_position`. (#8257)Erik Johnston2020-09-081-0/+50
| | | | | It did not correctly handle IDs finishing being persisted out of order, resulting in the `current_position` lagging until new IDs are persisted.
* Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-082-13/+5
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-14 22:22:16 +0000