summary refs log tree commit diff
path: root/synapse/storage/databases/state (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Bump ruff from 0.7.3 to 0.11.10 (#18451)dependabot[bot]2025-05-201-2/+1
| | | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Disable Postgres statement timeouts while purging room state (#18133)Jason Little2025-04-021-0/+7
|
* Delete unreferenced state groups in background (#18254)Devon Hudson2025-03-212-10/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR fixes #18154 to avoid de-deltaing state groups which resulted in DB size temporarily increasing until the DB was `VACUUM`'ed. As a result, less state groups will get deleted now. It also attempts to improve performance by not duplicating work when processing state groups it has already processed in previous iterations. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erikj@element.io>
* Revert the background job to clear unreferenced state groups (that was ↵reivilibre2025-03-072-42/+10
| | | | | | | | | | | | | | | | | introduced in v1.126.0rc1), due to a suspected issue that causes increased disk usage. (#18222) Revert "Add background job to clear unreferenced state groups (#18154)" This mechanism is suspected of inserting large numbers of rows into `state_groups_state`, thus unreasonably increasing disk usage. See: https://github.com/element-hq/synapse/issues/18217 This reverts commit 5121f9210c989fcc909e78195133876dff3bc9b9 (#18154). --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* Add background job to clear unreferenced state groups (#18154)Devon Hudson2025-02-252-10/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #18150 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erikj@element.io>
* Cleanup deleted state group references (#18165)Devon Hudson2025-02-181-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Fix performance of `check_state_groups_and_bump_deletion` (#18141)Erik Johnston2025-02-071-5/+21
| | | | | | | | | | Regressed as part of https://github.com/element-hq/synapse/pull/18107 This does two things: 1. Only check if the state groups have been deleted when calculating the event context (as that's when we will insert them). This avoids lots of checks for read operations. 2. Don't lock the `state_groups` rows when doing the check. This adds overhead, and it doesn't prevent any races.
* Fix bug where persisting some events fails after unclean shutdown. (#18137)Erik Johnston2025-02-051-2/+12
| | | | | | Introduced in #18107 `UniqueViolation: duplicate key value violates unique constraint "state_groups_persisting_pkey"`
* Fix bug where purging history could lead to increase in disk space usage ↵Erik Johnston2025-02-031-1/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | (#18131) When purging history, we try and delete any state groups that become unreferenced (i.e. there are no longer any events that directly reference them). When we delete a state group that is referenced by another state group, we "de-delta" that state group so that it no longer refers to the state group that is deleted. There are two bugs with this approach that we fix here: 1. There is a common pattern where we end up storing two state groups when persisting a state event: the state before and after the new state event, where the latter is stored as a delta to the former. When deleting state groups we only deleted the "new" state and left (and potentially de-deltaed) the old state. This was due to a bug/typo when trying to find referenced state groups. 2. There are times where we store unreferenced state groups in the DB, during the purging of history these would not get rechecked and instead always de-deltaed. Instead, we should check for this case and delete any unreferenced state groups rather than de-deltaing them. The effect of the above bugs is that when purging history we'd end up with lots of unreferenced state groups that had been de-deltaed (i.e. stored as the full state). This can lead to dramatic increases in storage space used.
* Add locking to more safely delete state groups: Part 2 (#18130)Erik Johnston2025-02-032-7/+86
| | | | | | This actually makes it so that deleting state groups goes via the new mechanism. c.f. #18107
* Add locking to more safely delete state groups: Part 1 (#18107)Erik Johnston2025-02-032-15/+470
| | | | | | | | | | | | | | | | Currently we don't really have anything that stops us from deleting state groups when an in-flight event references it. This is a fairly rare race currently, but we want to be able to more aggressively delete state groups so it is important to address this to ensure that the database remains valid. This implements the locking, but doesn't actually use it. See the class docstring of the new data store for an explanation for how this works. --------- Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>
* Properly purge state groups tables when purging a room (#18024)Mathieu Velten2025-01-061-39/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently purging a complex room can lead to a lot of orphaned rows left behind in the state groups tables. It seems it is because we are loosing track of state groups sometimes. This change uses the `room_id` indexed column of `state_groups` table to decide what to delete instead of doing an indirection through `event_to_state_groups`. Related to https://github.com/element-hq/synapse/issues/3364. ### Pull Request Checklist * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erikj@jki.re>
* Fix MSC4222 returning full state (#17915)Erik Johnston2024-11-082-3/+4
| | | | | | | | | | | There was a bug that meant we would return the full state of the room on incremental syncs when using lazy loaded members and there were no entries in the timeline. This was due to trying to use `state_filter or state_filter.all()` as a short hand for handling `None` case, however `state_filter` implements `__bool__` so if the state filter was empty it would be set to full. c.f. MSC4222 and #17888
* Remove `Generator` in `_purge_unreferenced_state_groups` twice (#17815)Jason Little2024-10-301-2/+2
| | | | | | Context: https://github.com/matrix-org/synapse/issues/15439 (https://github.com/element-hq/synapse/issues/15439) Also see discussion in https://github.com/element-hq/synapse/pull/17813
* Format files with Ruff (#17643)Quentin Gliech2024-09-021-1/+1
| | | | | | I thought ruff check would also format, but it doesn't. This runs ruff format in CI and dev scripts. The first commit is just a run of `ruff format .` in the root directory.
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-131-5/+5
|
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-233-0/+3
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Update license headersPatrick Cloke2023-11-213-30/+48
|
* Use attempt_to_set_autocommit everywhere. (#16615)Patrick Cloke2023-11-091-2/+2
| | | To avoid asserting the type of the database connection.
* Remove more usages of cursor_to_dict. (#16551)Patrick Cloke2023-10-261-1/+0
| | | Mostly to improve type safety.
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-261-6/+12
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Convert simple_select_many_batch, simple_select_many_txn to tuples. (#16444)Patrick Cloke2023-10-111-19/+35
|
* Improve comments in StateGroupBackgroundUpdateStore. (#16383)Patrick Cloke2023-09-251-2/+16
|
* Fix a performance issue introduced in Synapse v1.83.0 which meant that ↵reivilibre2023-05-311-0/+31
| | | | | | | | | | | | | purging rooms was very slow and database-intensive. (#15693) * Add indices required to efficiently validate new foreign key constraints on stream_ordering * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Instrument `state` and `state_group` storage related things (tracing) (#15610)Eric Eastwood2023-05-192-0/+20
| | | | | Instrument `state` and `state_group` storage related things (tracing) so it's a little more clear where these database transactions are coming from as there is a lot of wires crossing in these functions. Part of `/messages` performance investigation: https://github.com/matrix-org/synapse/issues/13356
* Include the room ID in more purge room log lines. (#15222)Patrick Cloke2023-03-081-0/+2
|
* Batch up storing state groups when creating new room (#14918)Shay2023-02-241-0/+119
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-222-6/+2
|
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-122-2/+2
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Optimize `filter_events_for_client` for faster `/messages` - v2 (#14527)Eric Eastwood2022-11-221-20/+79
| | | Fix #14108
* Cancel the processing of key query requests when they time out. (#13680)reivilibre2022-09-071-0/+3
|
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-211-1/+8
|
* Reduce memory usage of state group cache (#13323)Erik Johnston2022-07-191-1/+2
|
* Don't pull out the full state when storing state (#13274)Erik Johnston2022-07-151-53/+103
|
* Add a unique index to `state_group_edges` to prevent duplicates being ↵reivilibre2022-05-191-0/+16
| | | | accidentally introduced and the consequential impact to performance. (#12687)
* Refactor `resolve_state_groups_for_events` to not pull out full state when ↵Shay2022-05-181-1/+1
| | | | no state resolution happens. (#12775)
* Back out in-flight state caching changes. (#12126)reivilibre2022-03-021-218/+25
|
* Order in-flight state group queries in biggest-first order (#11610)reivilibre2022-03-011-3/+27
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Cap the number of in-flight requests for state from a single group (#11608)reivilibre2022-02-221-0/+16
|
* Track and deduplicate in-flight requests to `_get_state_for_groups`. (#10870)reivilibre2022-02-181-25/+178
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-132-28/+14
| | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* Fix remove_stale_pushers job on SQLite. (#10843)reivilibre2021-09-201-3/+3
|
* Add type hints to state database module. (#10823)Patrick Cloke2021-09-152-69/+127
|
* Add type hints to state handler. (#10482)Patrick Cloke2021-07-261-6/+11
|
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-0710-233/+0
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-143-3/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-3/+7
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-4/+5
|
* Refactor to ensure we call check_consistency (#9470)Erik Johnston2021-02-241-4/+6
| | | The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-3/+3
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Use execute_batch in more places (#9188)Erik Johnston2021-01-211-2/+2
| | | | | * Use execute_batch in more places * Newsfile
* Mypy fixes for `synapse.handlers.federation` (#8422)Richard van der Hoff2020-09-291-2/+2
| | | For some reason, an apparently unrelated PR upset mypy about this module. Here are a number of little fixes.
* Add checks for postgres sequence consistency (#8402)Erik Johnston2020-09-281-0/+3
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Convert state and stream stores and related code to async (#8194)Patrick Cloke2020-08-281-13/+13
|
* Converts event_federation and registration databases to async/await (#8061)Patrick Cloke2020-08-111-11/+7
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-0513-0/+1267
generated by cgit-magenta 1.5.1 (git 2.48.1) at 2025-08-09 12:12:37 +0000