summary refs log tree commit diff
path: root/synapse/storage/schema (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix sqlite `user_filters` upgrade (#15817)Shay2023-06-273-3/+91
|
* Merge branch 'master' into developErik Johnston2023-06-071-1/+21
|\
| * Fix schema delta error in 1.85 (#15739)Erik Johnston2023-06-071-2/+14
| | | | | | | | Some users seem to have multiple rows per user / room with a null thread ID, which we need to handle.
| * Fix schema delta error in 1.85 (#15738)Erik Johnston2023-06-071-0/+8
| | | | | | | | | | | | | | | | There appears to be a race where you can end up with entries in `event_push_summary` with both a `NULL` and `main` thread ID. Fixes #15736 Introduced in #15597
* | `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-023-1/+191
|/ | | | `profiles` and `user_filters` (#15649)
* Fix a performance issue introduced in Synapse v1.83.0 which meant that ↵reivilibre2023-05-311-0/+20
| | | | | | | | | | | | | purging rooms was very slow and database-intensive. (#15693) * Add indices required to efficiently validate new foreign key constraints on stream_ordering * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Set thread_id column to non-null for ↵Patrick Cloke2023-05-266-0/+216
| | | | | | | | | | | | | | | | event_push_{actions,actions_staging,summary} (#15597) Updates the database schema to require a thread_id (by adding a constraint that the column is non-null) for event_push_actions, event_push_actions_staging, and event_push_actions_summary. For PostgreSQL we add the constraint as NOT VALID, then VALIDATE the constraint a background job to avoid locking the table during an upgrade. Each table is updated as a separate schema delta to avoid deadlocks between them. For SQLite we simply rebuild the table & copy the data.
* Update code to refer to "workers". (#15606)Patrick Cloke2023-05-161-1/+1
| | | | A bunch of comments and variables are out of date and use obsolete terms.
* Add not null constraint to column `full_user_id` of tables `profiles` and ↵Shay2023-05-165-2/+72
| | | | `user_filters` (#15537)
* Revert "Set thread_id column to non-null for ↵Erik Johnston2023-05-124-170/+0
| | | | | event_push_{actions,actions_staging,summary} (#15437)" (#15580) This reverts commit a7b3e9ce65335e452de216cb42b9e724e8f3ad1d.
* Set thread_id column to non-null for ↵Patrick Cloke2023-05-034-0/+170
| | | | | | | | | | | | | event_push_{actions,actions_staging,summary} (#15437) Updates the database schema to require a thread_id (by adding a constraint that the column is non-null) for event_push_actions, event_push_actions_staging, and event_push_actions_summary. For PostgreSQL we add the constraint as NOT VALID, then VALIDATE the constraint a background job to avoid locking the table during an upgrade. For SQLite we simply rebuild the table & copy the data.
* Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-031-0/+24
| | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add an admin API endpoint to support per-user feature flags (#15344)Shay2023-04-281-0/+27
|
* Add type hints to schema deltas (#15497)Patrick Cloke2023-04-2727-131/+126
| | | | | | | | | | Cleans-up the schema delta files: * Removes no-op functions. * Adds missing type hints to function parameters. * Fixes any issues with type hints. This also renames one (very old) schema delta to avoid a conflict that mypy complains about.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-263-1/+44
|
* Re re introduce membership tables event stream ordering (#15356)Nick Mills-Barrett2023-04-254-4/+141
|
* Experimental support for MSC3970: per-device transaction IDs (#15318)Quentin Gliech2023-04-251-0/+53
|
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+17
|
* Speed up membership queries for users with forgotten rooms (#15385)Erik Johnston2023-04-041-0/+19
|
* Revert "Set thread_id column to non-null for ↵Olivier Wilkinson (reivilibre)2023-03-314-153/+3
| | | | | | | | event_push_{actions,actions_staging,summary} (#15350)" This reverts commit 2a234b788e2b5706ee83cf8eb86dfd004bc7c166. See #15359 for context.
* Set thread_id column to non-null for ↵Patrick Cloke2023-03-304-3/+153
| | | | | | | | | | | event_push_{actions,actions_staging,summary} (#15350) Clean-up from adding the thread_id column, which was initially null but backfilled with values. It is desirable to require it to now be non-null. In addition to altering this column to be non-null, we clean up obsolete background jobs, indexes, and just-in-time updating code.
* Revert "Reintroduce membership tables event stream ordering (#15128)" (#15347)Erik Johnston2023-03-293-109/+4
| | | This reverts commit e6af49fbea939d9e69ed05e0a0ced5948c722ea4.
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-241-0/+19
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Reintroduce membership tables event stream ordering (#15128)Nick Mills-Barrett2023-03-243-4/+109
| | | | | | | | | | | | | | | | | | * Add `event_stream_ordering` column to membership state tables Specifically this adds the column to `current_state_events`, `local_current_membership` and `room_memberships`. Each of these tables is regularly joined with the `events` table to get the stream ordering and denormalising this into each table will yield significant query performance improvements once used. * Make denormalised `event_stream_ordering` columns foreign keys * Add comment in schema file explaining new denormalised columns * Add triggers to enforce consistency of `event_stream_ordering` columns * Re-order purge room tables to account for foreign keys * Bump schema version to 75 Co-authored-by: David Robertson <david.m.robertson1@gmail.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Preparatory work to fix the user directory assuming that any remote ↵reivilibre2023-03-161-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | membership state events represent a profile change. [rei:userdirpriv] (#14755) * Remove special-case method for new memberships only, use more generic method * Only collect profiles from state events in public rooms * Add a table to track stale remote user profiles * Add store methods to set and delete rows in this new table * Mark remote profiles as stale when a member state event comes in to a private room * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Simplify by removing Optionality of `event_id` * Replace names and avatars with None if they're set to dodgy things I think this makes more sense anyway. * Move schema delta to 74 (I missed the boat?) * Turns out these can be None after all --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Add schema comments about the `destinations` and `destination_rooms` tables. ↵reivilibre2023-03-151-0/+52
| | | | (#15247)
* Bump schema version (#15036)David Robertson2023-02-091-2/+7
| | | | | | | | * Bump schema version This should have been included in f10caa73eee0caa91cf373966104d1ededae2aee (and #14979). * Changelog
* Revert "Add `event_stream_ordering` column to membership state tables (#14979)"David Robertson2023-02-071-21/+0
| | | | This reverts commit 5fdc12f482c68e2cdbb78d7db5de2cfe621720d4.
* Add `event_stream_ordering` column to membership state tables (#14979)Nick Mills-Barrett2023-02-071-0/+21
| | | | | | | | | | | | This adds an `event_stream_ordering` column to `current_state_events`, `local_current_membership` and `room_memberships`. Each of these tables is regularly joined with the `events` table to get the stream ordering and denormalising this into each table will yield significant query performance improvements once used. Includes a background job to populate these values from the `events` table. Same idea as https://github.com/matrix-org/synapse/pull/13703. Signed off by Nick @ Beeper (@fizzadar).
* Drop unused table `presence` (#14825)Dirk Klimpel2023-01-131-0/+17
|
* Add index to improve performance of the `/timestamp_to_event` endpoint used ↵reivilibre2023-01-111-0/+17
| | | | for jumping to a specific date in the timeline of a room. (#14799)
* Faster remote room joins: stream the un-partial-stating of events over ↵reivilibre2022-12-142-0/+54
| | | | replication. [rei:frrj/streams/unpsr] (#14545)
* Delete event_push_summary_unique_index again. (#14669)Patrick Cloke2022-12-141-0/+33
| | | | | | | | | | | | | | | | | | if a Synapse deployment upgraded (from < 1.62.0 to >= 1.70.0) then it is possible for schema deltas to run before background updates causing drift in the database schema due to: 1. A delta registered a background update to create an index. 2. A delta dropped the above index if it exists (but it yet exist won't since the background job hasn't run). 3. The code assumed the index was dropped. To fix this we: 1. Cancel the background update which could create the index. 2. Drop the index again. 3. Drop a related index which is dropped by the background update.
* Rebuild the user directory and stats tables. (#14643)Patrick Cloke2022-12-081-0/+29
| | | | | | | | | | | Due to the various fixes to the StreamChangeCache it is not safe to trust the information in the user directory or room/user stats tables. Rebuild them as background jobs. In particular see da777207528513c858395758bf4c023da2c2c1a3 (#14639), and 6a8310f3dfe77acf59df2fe3e88a71b85b9b3ecc (#14435). Maybe also be related to fac8a38525387e344e3595a092578e0ffedd49ae (#14592).
* Faster remote room joins: stream the un-partial-stating of rooms over ↵reivilibre2022-12-052-0/+52
| | | | replication. [rei:frrj/streams/unpsr] (#14473)
* Add another index to `device_lists_changes_in_room` (#14534)Erik Johnston2022-11-231-0/+20
| | | | This helps avoid reading unnecessarily large amounts of data from the table when querying with a set of room IDs.
* Track unconverted device list outbound pokes using a position instead (#14516)Sean Quah2022-11-221-0/+53
| | | | | | | | | | | | | | | | | | | | | When a local device list change is added to `device_lists_changes_in_room`, the `converted_to_destinations` flag is set to `FALSE` and the `_handle_new_device_update_async` background process is started. This background process looks for unconverted rows in `device_lists_changes_in_room`, copies them to `device_lists_outbound_pokes` and updates the flag. To update the `converted_to_destinations` flag, the database performs a `DELETE` and `INSERT` internally, which fragments the table. To avoid this, track unconverted rows using a `(stream ID, room ID)` position instead of the flag. From now on, the `converted_to_destinations` column indicates rows that need converting to outbound pokes, but does not indicate whether the conversion has already taken place. Closes #14037. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix PostgreSQL sometimes using table scans for `event_search` (#14409)Sean Quah2022-11-101-0/+33
| | | | | | | | | | | PostgreSQL may underestimate the number of distinct `room_id`s in `event_search`, which can cause it to use table scans for queries for multiple rooms. Fix this by setting `n_distinct` on the column. Resolves #14402. Signed-off-by: Sean Quah <seanq@matrix.org>
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-0/+35
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Unified search query syntax using the full-text search capabilities of the ↵James Salter2022-10-251-0/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | underlying DB. (#11635) Support a unified search query syntax which leverages more of the full-text search of each database supported by Synapse. Supports, with the same syntax across Postgresql 11+ and Sqlite: - quoted "search terms" - `AND`, `OR`, `-` (negation) operators - Matching words based on their stem, e.g. searches for "dog" matches documents containing "dogs". This is achieved by - If on postgresql 11+, pass the user input to `websearch_to_tsquery` - If on sqlite, manually parse the query and transform it into the sqlite-specific query syntax. Note that postgresql 10, which is close to end-of-life, falls back to using `phraseto_tsquery`, which only supports a subset of the features. Multiple terms separated by a space are implicitly ANDed. Note that: 1. There is no escaping of full-text syntax that might be supported by the database; e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this as accidental functionality and depend on something we don't guarantee. 2. English text is assumed for stemming. To support other languages, either the target language needs to be known at the time of indexing the message (via room metadata, or otherwise), or a separate index for each language supported could be created. Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
* Update the thread_id right before use (in case the bg update hasn't ↵Patrick Cloke2022-10-183-134/+8
| | | | | | | | finished) (#14222) This avoids running a forced-update of a null thread_id rows. An index is added (in the background) to hopefully make this easier in the future.
* When restarting a partial join resync, prioritise the server which actioned ↵David Robertson2022-10-181-0/+18
| | | | a partial join (#14126)
* Add an API for listing threads in a room. (#13394)Patrick Cloke2022-10-131-0/+30
| | | | | | | | | Implement the /threads endpoint from MSC3856. This is currently unstable and behind an experimental configuration flag. It includes a background update to backfill data, results from the /threads endpoint will be partial until that finishes.
* Mark events as read using threaded read receipts from MSC3771. (#13877)Patrick Cloke2022-10-042-0/+99
| | | | Applies the proper logic for unthreaded and threaded receipts to either apply to all events in the room or only events in the same thread, respectively.
* Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke2022-10-044-3/+152
| | | | | | | | When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
* Clear out old rows from `event_push_actions_staging` (#14020)Erik Johnston2022-10-033-0/+47
| | | On matrix.org we have ~5 million stale rows in `event_push_actions_staging`, let's add a background job to make sure we clear them out.
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-291-0/+20
| | | | used (using MSC3866) (#13556)
* Handle remote device list updates during partial join (#13913)Erik Johnston2022-09-281-0/+28
| | | | | | | c.f. #12993 (comment), point 3 This stores all device list updates that we receive while partial joins are ongoing, and processes them once we have the full state. Note: We don't actually process the device lists in the same ways as if we weren't partially joined. Instead of updating the device list remote cache, we simply notify local users that a change in the remote user's devices has happened. I think this is safe as if the local user requests the keys for the remote user and we don't have them we'll simply fetch them as normal.
* Add new columns tracking when we partial-joined (#13892)David Robertson2022-09-271-0/+23
|
* Snapshot schema 72 (#13873)David Robertson2022-09-266-0/+2054
| | | Including another batch of fixes to the schema dump script
* Track device IDs for pushers (#13831)Brendan Abolivier2022-09-211-0/+20
| | | Second half of the MSC3881 implementation
* Support enabling/disabling pushers (from MSC3881) (#13799)Brendan Abolivier2022-09-211-0/+16
| | | Partial implementation of MSC3881
* Add support to purge rows from MSC2716 and other tables when purging a room ↵Eric Eastwood2022-09-162-0/+24
| | | | | | | | | | | (#13825) `event_failed_pull_attempts` added in https://github.com/matrix-org/synapse/pull/13589 MSC2716 related tables added in: - https://github.com/matrix-org/synapse/pull/10245/files#diff-3d42dfb44d02f7de3aada105e0bdc1cc9dd7f953cbf0f36c5d0f50827bf0320aR1 - Renamed in https://github.com/matrix-org/synapse/pull/10838/files#diff-2730bfbe9e688b55e46f9371aefe67dac2bd2b2b7d9d6b92774eea1fcfae156dR1 - https://github.com/matrix-org/synapse/pull/10498/files#diff-c52bbfbb5921a3f6f023b24343668479d966fac164f13b7c39d2197ce3afa7a5R1
* Keep track when we try and fail to process a pulled event (#13589)Eric Eastwood2022-09-142-0/+31
| | | | | | | | | | | | | | We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
* Update event push action and receipt tables to support threads. (#13753)Patrick Cloke2022-09-145-1/+155
| | | | | | | | | | | | | | | Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` tables. This will notifications to be segmented by the thread in a future pull request. The `thread_id` column stores the root event ID or the special value `"main"`. The `thread_id` column for `event_push_actions` and `event_push_summary` is backfilled with `"main"` for all existing rows. New entries into `event_push_actions` and `event_push_actions_staging` will get the proper thread ID. `receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar, except `NULL` is a special value meaning the receipt is "unthreaded". See MSC3771 and MSC3773 for where this data will be useful.
* Use partial indices on SQLIte. (#13802)Patrick Cloke2022-09-141-0/+56
| | | | | | | Partial indices have been supported since SQLite 3.8, but Synapse now requires >= 3.27, so we can enable support for them. This requires rebuilding previous indices which were partial on PostgreSQL, but not on SQLite.
* Remove incorrect migration file from `state` logical DB (#13788)David Robertson2022-09-141-37/+0
| | | | | | | | | | | | | * Remove incorrect migration file from `state` logical DB The table `ex_outlier_stream` is part of the `main` logical DB; it should not have been created in the `state` logical DB. We remove this migration now as a tidy-up. Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new migration, because some (most) instances of Synapse host both of these logical DBs on the same DB cluster. * Changelog
* Make sequence `cache_invalidation_stream_seq` begin at `2` (#13766)Mathieu Velten2022-09-132-0/+24
| | | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Updates to the schema dump script (#13770)David Robertson2022-09-131-0/+4
|
* Add receipts event stream ordering (#13703)Nick Mills-Barrett2022-09-131-0/+19
|
* Remove check current state membership up to date (#13745)Nick Mills-Barrett2022-09-121-0/+52
| | | | | | | * Remove checks for membership column in current_state_events * Add schema script to force through the `current_state_events_membership` background job Contributed by Nick @ Beeper (@fizzadar).
* Add timestamp to user's consent (#13741)Dirk Klimpel2022-09-081-0/+16
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add a schema delta to drop unstable private read receipts. (#13692)Patrick Cloke2022-09-011-0/+19
| | | | Otherwise they'll be leaked due to the filtering code only respecting the stable identifiers for private read receipts.
* Drop unused column `application_services_state.last_txn` (#13627)Shay2022-08-303-0/+58
|
* Bg update to populate new `events` table columns (#13215)Richard van der Hoff2022-07-151-0/+47
| | | | | These columns were added back in Synapse 1.52, and have been populated for new events since then. It's now (beyond) time to back-populate them for existing events.
* Drop unused tables from groups/communities. (#12967)Patrick Cloke2022-07-132-3/+35
| | | | These tables have been unused since Synapse v1.61.0, although schema version 72 was added in Synapse v1.62.0.
* Drop unused table `event_reference_hashes` (#13218)Richard van der Hoff2022-07-121-0/+17
| | | This is unused since Synapse 1.60.0 (#12679). It's time for it to go.
* Merge remote-tracking branch 'origin/release-v1.62' into developPatrick Cloke2022-06-301-0/+19
|\
| * Add index to help delete old push actions (#13141)Erik Johnston2022-06-301-0/+19
| |
* | Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)Šimon Brandner2022-06-291-0/+19
|/ | | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Fix serialization errors when rotating notifications (#13118)Erik Johnston2022-06-281-0/+35
|
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-152-3/+22
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Clean up schema for `event_edges` (#12893)Richard van der Hoff2022-06-153-5/+97
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove redundant references to `event_edges.room_id` We don't need to care about the room_id here, because we are already checking the event id. * Clean up the event_edges table We make a number of changes to `event_edges`: * We give the `room_id` and `is_state` columns defaults (null and false respectively) so that we can stop populating them. * We drop any rows that have `is_state` set true - they should no longer exist. * We drop any rows that do not exist in `events` - these should not exist either. * We drop the old unique constraint on all the colums, which wasn't much use. * We create a new unique index on `(event_id, prev_event_id)`. * We add a foreign key constraint to `events`. These happen rather differently depending on whether we are on Postgres or SQLite. For SQLite, we just rebuild the whole table, copying only the rows we want to keep. For Postgres, we try to do things in the background as much as possible. * Stop populating `event_edges.room_id` and `is_state` We can just rely on the defaults.
* Rename delta to apply in the proper schema version. (#13050)Patrick Cloke2022-06-141-0/+0
|
* Replace noop background updates with DELETE. (#12954)Patrick Cloke2022-06-131-0/+61
| | | | Removes the `register_noop_background_update` and deletes the background updates directly in a delta file.
* Remove remaining pieces of groups code. (#12966)Patrick Cloke2022-06-061-0/+1
| | | | | * Remove an unused stream ID generator. * Remove the now unused remote profile cache.
* Stop reading from `event_edges.room_id`. (#12914)Richard van der Hoff2022-05-311-1/+4
| | | event_edges.room_id is implied by the event id, so there is no need to join on the room id.
* Add a migration step to cleanup potential leftovers of bug 11833 (#12784)Mathieu Velten2022-05-301-0/+19
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add a unique index to `state_group_edges` to prevent duplicates being ↵reivilibre2022-05-191-0/+17
| | | | accidentally introduced and the consequential impact to performance. (#12687)
* Remove code which updates `application_services_state.last_txn` (#12680)Richard van der Hoff2022-05-171-1/+4
| | | | This column is unused as of #12209, so let's stop writing to it.
* Add index to cache invalidations (#12747)Erik Johnston2022-05-171-0/+18
| | | | | For workers that rarely write to the cache the `get_all_updated_caches` query can become expensive if the worker falls behind when reading the cache.
* Stop writing to `event_reference_hashes` (#12679)Richard van der Hoff2022-05-101-1/+4
| | | | | This table is never read, since #11794. We stop writing to it; in future we can drop it altogether.
* Process device list updates asynchronously (#12365)Erik Johnston2022-04-121-3/+3
|
* Track device list updates per room. (#12321)Erik Johnston2022-04-042-0/+39
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Use a sequence to generate AS transaction IDs, drop `last_txn` AS state (#12209)Nick Mills-Barrett2022-04-012-1/+48
| | | | | | | | Switching to a sequence means there's no need to track `last_txn` on the AS state table to generate new TXN IDs. This also means that there is no longer contention between the AS scheduler and AS handler on updates to the `application_services_state` table, which will prevent serialization errors during the complete AS txn transaction.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-0/+23
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-1/+0
|
* Faster joins: persist to database (#12012)Richard van der Hoff2022-03-012-0/+113
| | | | | | | | | | | | When we get a partial_state response from send_join, store information in the database about it: * store a record about the room as a whole having partial state, and stash the list of member servers too. * flag the join event itself as having partial state * also, for any new events whose prev-events are partial-stated, note that they will *also* be partial-stated. We don't yet make any attempt to interpret this data, so API calls (and a bunch of other things) are just going to get incorrect data.
* Fix non-strings in the `event_search` table (#12037)Sean Quah2022-02-241-0/+22
| | | | | | | Don't attempt to add non-string `value`s to `event_search` and add a background update to clear out bad rows from `event_search` when using sqlite. Signed-off-by: Sean Quah <seanq@element.io>
* Fix slow performance of `/logout` in some cases where refresh tokens are in ↵reivilibre2022-02-221-0/+28
| | | | use. The slowness existed since the initial implementation of refresh tokens. (#12056)
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+20
| | | | users. (#11655)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-0/+21
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-212-3/+31
| | | | ... and start populating them for new events
* Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff2022-01-212-1/+21
| | | This is a follow-up to #10565.
* Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff2022-01-211-1/+4
| | | | Preparation for dropping this table altogether. Part of #6574.
* Stop populating `state_events.prev_state` (#11558)Richard van der Hoff2021-12-101-1/+4
| | | this field is never read, so we may as well stop populating it.
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-061-0/+27
| | | As a step towards allowing back-channel logout for OIDC.
* Disambiguate queries on `state_key` (#11497)Richard van der Hoff2021-12-021-1/+5
| | | | | We're going to add a `state_key` column to the `events` table, so we need to add some disambiguation to queries which use it.
* Support expiry of refresh tokens and expiry of the overall session when ↵reivilibre2021-11-261-0/+28
| | | | refresh tokens are in use. (#11425)
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-0/+18
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Store arbitrary relations from events. (#11391)Patrick Cloke2021-11-221-1/+1
| | | | | Instead of only known relation types. This also reworks the background update for thread relations to crawl events and search for any relation type, not just threaded relations.
* Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+13
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Rename `remove_deleted_devices_from_device_inbox` to ensure it is always run ↵Andrew Morgan2021-11-161-1/+13
| | | | | | | | | | (#11353) Co-authored-by: reivilibre <oliverw@matrix.org>
| * Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-151-1/+1
| |
* | Remove unused tables `room_stats_historical` and `user_stats_historical` ↵Shay2021-11-122-1/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#11280) * remove unused tables room_stats_historical and user_stats_historical * update changelog number * Bump schema compat version comment * make linter happy * Update comment to give more info Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-111-1/+1
|/
* Add index to `local_group_updates.stream_id` (#11231)Erik Johnston2021-11-021-0/+18
| | | | This should speed up startup times and generally increase performance of groups.
* Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+22
|
* Fix comments referencing v1.46.0 from PR #10969. (#11212)Dirk Klimpel2021-10-291-1/+1
| | | | #10969 was merged after 1.46.0rc1 was cut and will be included in v1.47.0rc1 instead.
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+22
| | | Fixes: #9346
* Add a background update for updating MSC3440 relation threads. (#11181)Patrick Cloke2021-10-261-0/+18
|
* Resolve and share `state_groups` for all historical events in batch ↵Eric Eastwood2021-10-132-1/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#10975) Resolve and share `state_groups` for all historical events in batch. This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch. This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room. For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch. --- How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example: **Before** (new `state_group` for every event 😬, very inefficient): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10 create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11 create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12 create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13 create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14 ``` **After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10 create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10 create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10 create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10 create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10 ```
* Drop backwards-compatibility support for "outlier" (#10903)Richard van der Hoff2021-09-281-4/+2
| | | | | Before Synapse 1.31 (#9411), we relied on `outlier` being stored in the `internal_metadata` column. We can now assume nobody will roll back their deployment that far and drop the legacy support.
* Document changes to schema version 61 - 64 (#10917)Eric Eastwood2021-09-241-0/+11
| | | | | As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244 Retroactively summarize `61` - `64`
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-231-1/+1
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Rename MSC2716 things from `chunk` to `batch` to match `/batch_send` ↵Eric Eastwood2021-09-213-1/+61
| | | | | | | | endpoint (#10838) See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497 Dropping support for older MSC2716 room versions so we don't have to worry about supporting both chunk and batch events.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-132-2/+2
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add a partial index to `presence_stream` to speed up startups (#10748)Sean2021-09-031-0/+18
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+17
| | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* Move the sessions delta to the latest schema version. (#10725)Patrick Cloke2021-08-311-0/+0
| | | This was erroneously put under schema version 62 instead of 63.
* Remove pushers when deleting 3pid from account (#10581)Azrenbeth2021-08-261-0/+20
| | | | | When a user deletes an email from their account it will now also remove all pushers for that email and that user (even if these pushers were created by a different client)
* Make a note to leave a summary when one is bumping the schema version (#10621)Andrew Morgan2021-08-251-0/+2
| | | I found this easy to miss (and evidently, it looks like it was missed for schema version 62).
* Persist room hierarchy pagination sessions to the database. (#10613)Patrick Cloke2021-08-241-0/+23
|
* Implement MSC3231: Token authenticated registration (#10142)Callum Brown2021-08-211-0/+23
| | | | | Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
* Remove the unused public_room_list_stream (#10565)Andrew Morgan2021-08-171-1/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* update links to schema doc (#10620)Richard van der Hoff2021-08-172-3/+3
|
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-042-1/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Make historical events discoverable from backfill for servers without any ↵Eric Eastwood2021-07-281-0/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
* Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359)Erik Johnston2021-07-151-0/+34
| | | | | | | | The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.
* Replace `room_depth.min_depth` with a BIGINT (#10289)Richard van der Hoff2021-07-122-0/+88
| | | | | while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.
* Switch `application_services_txns.txn_id` to BIGINT (#10349)Richard van der Hoff2021-07-091-0/+23
|
* Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-081-1/+5
| | | Fixes #9602
* fix ordering of bg update (#10291)Richard van der Hoff2021-07-011-1/+1
| | | | | this was a typo introduced in #10282. We don't want to end up doing the `replace_stream_ordering_column` update after anything that comes up in migration 60/03.
* Change more stream_ordering columns to BIGINT (#10286)Richard van der Hoff2021-06-301-0/+30
|
* Rebuild other indexes using `stream_ordering` (#10282)Richard van der Hoff2021-06-301-3/+8
| | | | We need to rebuild *all* of the indexes that use the current `stream_ordering` column.
* Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-292-0/+69
|\
| * Handle inbound events from federation asynchronously (#10272)Erik Johnston2021-06-291-0/+32
| | | | | | | | | | | | | | | | | | | | | | Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.
| * Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+37
| | | | | | This adds a simple best effort locking mechanism that works cross workers.
* | Migrate stream_ordering to a bigint (#10264)Richard van der Hoff2021-06-292-1/+41
| | | | | | | | | | | | | | | | | | | | * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog
* | MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-241-0/+34
|/ | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* Fix schema delta to not take as long on large servers (#10227)Erik Johnston2021-06-221-2/+5
| | | Introduced in #6739
* Support for database schema version ranges (#9933)Richard van der Hoff2021-06-113-37/+26
| | | This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
* Implement knock feature (#6739)Sorunome2021-06-091-0/+17
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-181-0/+34
| | | | to them, instead of something in-memory (#9823)
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-07280-0/+10293
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Only run one background update at a timeRichard van der Hoff2020-03-311-0/+19
|
* Move schema delta files to the correct data store.Erik Johnston2019-10-242-73/+0
| | | | They were put in the global schema delta directory due to a bad merge.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-222-0/+73
|\ | | | | | | erikj/refactor_stores
| * Merge branch 'develop' into uhoreg/e2e_cross-signing_mergedHubert Chathi2019-10-189-0/+221
| |\
| * \ Merge branch 'develop' into cross-signing_keysHubert Chathi2019-09-041-0/+152
| |\ \
| * \ \ Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keysHubert Chathi2019-08-284-0/+75
| |\ \ \
| | * \ \ Merge branch 'develop' into uhoreg/e2e_cross-signing_mergedHubert Chathi2019-08-284-0/+75
| | |\ \ \
| | * | | | Cross-signing [1/4] -- hidden devices (#5759)Hubert Chathi2019-08-121-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * allow devices to be marked as "hidden" This is a prerequisite for cross-signing, as it allows us to create other things that live within the device namespace, so they can be used for signatures.
| * | | | | use stream ID generator instead of timestampHubert Chathi2019-08-281-3/+3
| | | | | |
| * | | | | make changes from PR reviewHubert Chathi2019-08-012-41/+55
| | | | | |
| * | | | | Merge branch 'cross-signing_hidden' into cross-signing_keysHubert Chathi2019-08-013-3/+42
| |\ \ \ \ \
| | * | | | | Merge branch 'develop' into cross-signing_hiddenHubert Chathi2019-07-303-3/+42
| | |\| | | |
| * | | | | | Merge branch 'cross-signing_hidden' into cross-signing_keysHubert Chathi2019-07-301-1/+1
| |\| | | | |
| | * | | | | apply changes from PR reviewHubert Chathi2019-07-301-1/+1
| | | | | | |
| * | | | | | allow uploading keys for cross-signingHubert Chathi2019-07-251-0/+41
| |/ / / / /
| * | | | | allow devices to be marked as "hidden"Hubert Chathi2019-07-241-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a prerequisite for cross-signing, as it allows us to create other things that live within the device namespace, so they can be used for signatures.
* | | | | | Move storage classes into a main "data store".Erik Johnston2019-10-21189-8109/+10
| |_|_|_|/ |/| | | | | | | | | | | | | | | | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-103-0/+88
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | erikj/disable_sql_bytes
| * | | | | Rewrite the user_filter migration again (#6184)Richard van der Hoff2019-10-101-26/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | you can't plausibly ALTER TABLE in sqlite, so we create the new table with the right schema to start with.
| * | | | | Merge pull request #6185 from matrix-org/erikj/fix_censored_evnetsErik Johnston2019-10-091-0/+26
| |\ \ \ \ \ | | | | | | | | | | | | | | Fix inserting bytes as text in `censor_redactions`
| * | | | | | Fix unique_user_filter_index schema updateBrendan Abolivier2019-10-071-2/+2
| | | | | | |
| * | | | | | Update `user_filters` table to have a unique index, and non-null columns (#1172)Alexander Maznev2019-10-041-0/+46
| | | | | | |
| * | | | | | Land improved room list based on room stats (#6019)Erik Johnston2019-10-021-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Use room_stats and room_state for room directory search
| * | | | | | Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into developErik Johnston2019-10-022-0/+38
| |\ \ \ \ \ \
| * | | | | | | Drop unused tables (#6115)Richard van der Hoff2019-09-301-0/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These tables are unused since #5893 (as amended by #6047), so we can now drop them. Fixes #6048.
* | | | | | | | Do the update as a background indexErik Johnston2019-10-091-9/+8
| |_|/ / / / / |/| | | | | |
* | | | | | | Fix existing hex encoded json values in DBErik Johnston2019-10-081-0/+26
| |/ / / / / |/| | | | |
* | | | | | Fix errors storing large retry intervals.Erik Johnston2019-10-021-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
* | | | | | Add received_ts column to redactions.Erik Johnston2019-10-011-0/+20
|/ / / / / | | | | | | | | | | | | | | | | | | | | This will allow us to efficiently search for uncensored redactions in the DB before a given time.
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-251-0/+24
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | erikj/cleanup_user_ips
| * \ \ \ \ Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_workRichard van der Hoff2019-09-241-20/+0
| |\ \ \ \ \
| * \ \ \ \ \ Merge branch 'develop' into rav/saml_mapping_workRichard van der Hoff2019-09-197-0/+269
| |\ \ \ \ \ \
| * | | | | | | Record mappings from saml users in an external tableRichard van der Hoff2019-09-131-0/+24
| | |_|_|/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We want to assign unique mxids to saml users based on an incrementing suffix. For that to work, we need to record the allocated mxid in a separate table.
* | | | | | | Add BG update to populate devices last seen infoErik Johnston2019-09-231-0/+3
| | | | | | |
* | | | | | | Add last seen info to devices table.Erik Johnston2019-09-231-0/+21
| |_|/ / / / |/| | | | | | | | | | | | | | | | | | | | | | | This allows us to purge old user_ips entries without having to preserve the latest last seen info for active devices.
* | | | | | Undo the deletion of some tables (#6047)Richard van der Hoff2019-09-191-20/+0
| |/ / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a partial revert of #5893. The problem is that if we drop these tables in the same release as removing the code that writes to them, it prevents users users from being able to roll back to a previous release. So let's leave the tables in place for now, and remember to drop them in a subsequent release. (Note that these tables haven't been *read* for *years*, so any missing rows resulting from a temporary upgrade to vNext won't cause a problem.)
* | | | | Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-0/+25
| | | | | | | | | | | | | | | | | | | | Track the time that a server started failing at, for general analysis purposes.
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-051-0/+152
|\ \ \ \ \ | | |_|_|/ | |/| | | | | | | | erikj/censor_redactions
| * | | | Fix and refactor room and user stats (#5971)Erik Johnston2019-09-041-0/+152
| | |_|/ | |/| | | | | | Previously the stats were not being correctly populated.
* / | | Censor redactions in DB after a monthErik Johnston2019-08-301-0/+17
|/ / /
* | | Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-221-0/+20
| | | | | | | | | | | | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | | Add missing index on users_in_public_rooms. (#5894)Richard van der Hoff2019-08-221-0/+17
| | |
* | | Drop some unused tables. (#5893)Richard van der Hoff2019-08-211-0/+20
| | | | | | | | | These tables are never used, so we may as well drop them.
* | | add the version field to the index for e2e_room_keysHubert Chathi2019-08-141-0/+18
|/ /
* | Don't recreate columnErik Johnston2019-07-302-1/+22
| |
* | Fix current_state_events membership background update.Erik Johnston2019-07-301-0/+0
| | | | | | | | | | | | Turns out not all rooms are in `rooms`, so lets fetch the room list from `current_state_events`. We move the delta file to force it to be run again.
* | Remove lost commentErik Johnston2019-07-291-7/+0
| |
* | Add function to get all forgotten rooms for userErik Johnston2019-07-241-0/+25
|/ | | | | | This will allow us to efficiently filter out rooms that have been forgotten in other queries without having to join against the `room_memberships` table.
* Update comment for new columnErik Johnston2019-07-191-0/+3
|
* Add background update for current_state_events.membership columnErik Johnston2019-07-181-0/+3
|
* Add membership column to current_state_events table.Erik Johnston2019-07-181-0/+19
| | | | | It turns out that doing a join is surprisingly expensive for the DB to do when room_membership table is larger than the disk cache.
* Implement access token expiry (#5660)Richard van der Hoff2019-07-121-0/+18
| | | | Record how long an access token is valid for, and raise a soft-logout once it expires.
* Run Black. (#5482)Amber Brown2019-06-207-38/+58
|
* Track deactivated accounts in the database (#5378)Brendan Abolivier2019-06-141-0/+19
|
* Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1Erik Johnston2019-06-064-0/+2338
|\ | | | | Make a full SQL schema
| * fix maybeAmber H. Brown2019-06-071-20/+2
| |
| * just user dir?Amber H. Brown2019-06-071-1/+19
| |
| * remove background updates that arent neededAmber H. Brown2019-06-071-33/+1
| |
| * add more commentsAmber Brown2019-06-061-0/+5
| |
| * add stuff in bg updatesAmber Brown2019-06-041-10/+62
| |
| * WHY IS THIS CALLED A SLIGHTLY DIFFERENT THINGAmber Brown2019-06-031-0/+1
| |
| * more fixAmber Brown2019-06-031-0/+38
| |
| * fix schemasAmber Brown2019-06-033-59/+18
| |
| * full schemaAmber Brown2019-06-033-0/+2315
| |
* | Add ability to perform password reset via email without trusting the ↵Andrew Morgan2019-06-061-0/+31
|/ | | | | | | | | | | | identity server (#5377) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368
* Merge pull request #5294 from matrix-org/erikj/speed_up_room_statsErik Johnston2019-05-311-0/+28
|\ | | | | Speed up room stats background update
| * Add indices. Remove room_ids accidentally addedErik Johnston2019-05-311-0/+28
| | | | | | | | | | | | | | | | | | | | We have to do this by re-inserting a background update and recreating tables, as the tables only get created during a background update and will later be deleted. We also make sure that we remove any entries that should have been removed but weren't due to a race that has been fixed in a previous commit.
* | Add index to temp tableErik Johnston2019-05-301-0/+1
|/
* Fixup comments and loggingErik Johnston2019-05-301-0/+3
|
* Add DB bg update to cleanup extremities.Erik Johnston2019-05-291-0/+19
| | | | | Due to #5269 we may have extremities in our DB that we shouldn't have, so lets add a cleanup task such to remove those.
* Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schemaBrendan Abolivier2019-05-281-0/+3
|\ | | | | Fix schema update for account validity
| * Fix schema update for account validityBrendan Abolivier2019-05-281-0/+3
| |
* | Store key validity time in the storage layerRichard van der Hoff2019-05-231-0/+23
| | | | | | | | | | | | | | | | This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.
* | Room Statistics (#4338)Amber Brown2019-05-211-0/+80
| |
* | Add simple send_relation API and track in DBErik Johnston2019-05-151-0/+27
|/
* Merge pull request #5047 from matrix-org/babolivier/account_expirationBrendan Abolivier2019-04-171-1/+8
|\ | | | | Send out emails with links to extend an account's validity period
| * Send out emails with links to extend an account's validity periodBrendan Abolivier2019-04-171-1/+8
| |
* | Merge pull request #5033 from matrix-org/erikj/fix_schema_deltaErik Johnston2019-04-101-1/+3
|\ \ | | | | | | Fix schema upgrade when dropping tables
| * | Fix schema upgrade when dropping tablesErik Johnston2019-04-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | We need to drop tables in the correct order due to foreign table constraints (on `application_services`), otherwise the DROP TABLE command will fail. Introduced in #4992.
* | | Merge pull request #5027 from matrix-org/babolivier/account_expirationBrendan Abolivier2019-04-091-0/+20
|\ \ \ | |/ / |/| / | |/ Add time-based account expiration
| * Add account expiration featureBrendan Abolivier2019-04-091-0/+20
| |
* | Remove unused server_tls_certificates functions (#5028)Richard van der Hoff2019-04-082-11/+4
| | | | | | | | These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.
* | drop tables listed in #1830 (#4992)Neil Johnson2019-04-0823-740/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities
* | Remove presence lists (#4989)Neil Johnson2019-04-032-10/+18
| | | | | | Remove presence list support as per MSC 1819
* | Fixup docstringsErik Johnston2019-04-021-2/+2
| |
* | Grandfather in existing user threepidsErik Johnston2019-04-011-0/+2
| | | | | | | | | | | | We assume, as we did before, that users bound their threepid to one of the trusted identity servers. So we simply fill the new table with all threepids in `user_threepids` joined with the trusted identity servers.
* | Track IS used to bind 3PIDsErik Johnston2019-04-011-0/+27
|/ | | | | This will then be used to know which IS to default to when unbinding the threepid.
* Fix user directory background update (#4887)Erik Johnston2019-03-191-1/+1
|
* Migrate the user directory initial population to a background task (#4864)Amber Brown2019-03-191-0/+30
|
* fixupAmber Brown2019-03-131-7/+0
|
* fixupAmber Brown2019-03-121-5/+12
|
* Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-searchAmber Brown2019-03-111-0/+2
|\
| * Add comment to schemaErik Johnston2019-03-081-0/+2
| |
* | initialAmber Brown2019-03-112-3/+28
|/
* Rewrite userdir to be faster (#4537)Amber Brown2019-03-071-0/+47
|
* Analyze user_ips before running deduplicationErik Johnston2019-02-121-3/+7
| | | | | | | Due to the table locks taken out by the naive upsert, the table statistics may be out of date. During deduplication it is important that the correct index is used as otherwise a full table scan may be incorrectly used, which can end up thrashing the database badly.
* Add support for persisting event format versionsErik Johnston2019-01-231-0/+16
| | | | | | | | | | | | | | Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.
* Remove duplicates in the user_ips table and add an index (#4370)Amber Brown2019-01-121-0/+26
|
* create support user (#4141)Neil Johnson2018-12-141-0/+19
| | | | | | Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
* Drop sent_transactionsAaron Raimist2018-12-014-72/+2
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Merge remote-tracking branch 'origin/develop' into ↵David Baker2018-11-093-4/+60
|\ | | | | | | dbkr/e2e_backup_versions_are_numbers
| * Add unique indexes to a couple of tablesRichard van der Hoff2018-11-022-4/+41
| | | | | | | | | | The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-191-0/+39
| |\ | | | | | | | | | erikj/purge_state_groups
| * | Add state_group index to event_to_state_groupsErik Johnston2018-10-041-0/+19
| | | | | | | | | | | | | | | This is needed to efficiently check for unreferenced state groups during purge.
* | | Try & make it work on postgresDavid Baker2018-10-301-2/+2
| | |
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-06 13:46:20 +0000