summary refs log tree commit diff
path: root/synapse/state (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Allow `compute_state_after_events` to use partial state (#14676)David Robertson2022-12-141-2/+8
| | | | | | | * Allow `compute_state_after_events` to use partial state if fetching a subset of state that is trusted during a partial join. * Changelog
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-121-1/+1
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Faster joins: do not wait for full state when creating events to send (#14403)Mathieu Velten2022-11-171-1/+7
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Prepatory work for batching events to send (#13487)Shay2022-09-281-0/+63
| | | This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
* Avoid putting rejected events in room state (#13723)Sean Quah2022-09-161-0/+15
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Remove cached wrap on `_get_joined_users_from_context` method (#13569)Nick Mills-Barrett2022-08-311-1/+1
| | | | | | | The method doesn't actually do any data fetching and the method that does, `_get_joined_profile_from_event_id`, has its own cache. Signed off by Nick @ Beeper (@Fizzadar).
* Rename `event_map` to `unpersisted_events` (#13603)David Robertson2022-08-241-32/+37
|
* Cache user IDs instead of profile objects (#13573)Nick Mills-Barrett2022-08-231-7/+6
| | | The profile objects are never used and increase cache size significantly.
* Correct a misnamed argument in state res v2 (#13467)David Robertson2022-08-081-6/+6
| | | | | | | | | | | In state res v2, we apply two passes of iterative auth checks. The first pass replays power events and events in their auth chains, but only those belonging to the full conflicted set. The source code as written suggests that we want only those belonging to the auth difference (which is a smaller set of events). At runtime we were doing the correct thing anyway, because the only callsite of `_reverse_topological_power_sort` passes in the `full_conflicted_set`. So this really is just a rename.
* Refactor `_resolve_state_at_missing_prevs` to return an `EventContext` (#13404)Sean Quah2022-08-011-0/+8
| | | | | | | | Previously, `_resolve_state_at_missing_prevs` returned the resolved state before an event and a partial state flag. These were unwieldy to carry around would only ever be used to build an event context. Build the event context directly instead. Signed-off-by: Sean Quah <seanq@matrix.org>
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-6/+12
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Don't hold onto full state in state cache (#13324)Erik Johnston2022-07-211-15/+53
|
* Don't pull out the full state when creating an event (#13281)Erik Johnston2022-07-181-1/+2
|
* Don't pull out the full state when storing state (#13274)Erik Johnston2022-07-151-15/+21
|
* Use state before join to determine if we `_should_perform_remote_join` (#13270)David Robertson2022-07-151-8/+13
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix a bug which could lead to incorrect state (#13278)Erik Johnston2022-07-151-5/+14
| | | | | There are two fixes here: 1. A long-standing bug where we incorrectly calculated `delta_ids`; and 2. A bug introduced in #13267 where we got current state incorrect.
* Rip out auth-event reconciliation code (#12943)Richard van der Hoff2022-07-141-26/+0
| | | | | | | There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution. This isn't specced, and there's general agreement we shouldn't be doing it. However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
* Don't pull out state in `compute_event_context` for unconflicted state (#13267)Erik Johnston2022-07-141-50/+67
|
* Faster room joins: fix race in recalculation of current room state (#13151)Sean Quah2022-07-071-0/+25
| | | | | | | | | | | Bounce recalculation of current state to the correct event persister and move recalculation of current state into the event persistence queue, to avoid concurrent updates to a room's current state. Also give recalculation of a room's current state a real stream ordering. Signed-off-by: Sean Quah <seanq@matrix.org>
* Skip waiting for full state for incoming events (#13144)Richard van der Hoff2022-07-011-3/+9
| | | | | When we receive an event over federation during a faster join, there is no need to wait for full state, since we have a whole reconciliation process designed to take the partial state into account.
* Move some event auth checks out to a different method (#13065)Richard van der Hoff2022-06-152-3/+3
| | | | | | | | | | | | | | * Add auth events to events used in tests * Move some event auth checks out to a different method Some of the event auth checks apply to an event's auth_events, rather than the state at the event - which means they can play no part in state resolution. Move them out to a separate method. * Rename check_auth_rules_for_event Now it only checks the state-dependent auth rules, it needs a better name.
* Track a histogram of state res durations (#13036)David Robertson2022-06-151-0/+12
|
* Merge branch 'rav/simplify_event_auth_interface' into developRichard van der Hoff2022-06-132-4/+1
|\
| * Remove `room_version` param from `check_auth_rules_for_event`Richard van der Hoff2022-06-122-4/+1
| | | | | | | | | | | | | | Instead, use the `room_version` property of the event we're checking. The `room_version` was originally added as a parameter somewhere around #4482, but really it's been redundant since #6875 added a `room_version` field to `EventBase`.
* | Type annotations for `test_v2` (#12985)David Robertson2022-06-091-15/+42
|/
* Reduce state pulled from DB due to sending typing and receipts over ↵Erik Johnston2022-06-061-4/+0
| | | | | federation (#12964) Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-061-70/+3
|
* Rename storage classes (#12913)Erik Johnston2022-05-311-20/+31
|
* Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-12/+10
|
* Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-7/+7
|
* Refactor `resolve_state_groups_for_events` to not pull out full state when ↵Shay2022-05-181-16/+19
| | | | no state resolution happens. (#12775)
* Add some type hints to datastore (#12717)Dirk Klimpel2022-05-171-2/+2
|
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-4/+5
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* remove constantly lib use and switch to enums. (#12624)andrew do2022-05-041-1/+1
|
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-1/+1
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Fix type of `events` in `StateGroupStorage` and `StateHandler` (#12156)Richard van der Hoff2022-03-041-3/+3
| | | We make multiple passes over this, so a regular iterable won't do.
* Faster joins: persist to database (#12012)Richard van der Hoff2022-03-011-2/+29
| | | | | | | | | | | | When we get a partial_state response from send_join, store information in the database about it: * store a record about the room as a whole having partial state, and stash the list of member servers too. * flag the join event itself as having partial state * also, for any new events whose prev-events are partial-stated, note that they will *also* be partial-stated. We don't yet make any attempt to interpret this data, so API calls (and a bunch of other things) are just going to get incorrect data.
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Remove `log_function` and its uses (#11761)Richard van der Hoff2022-01-181-2/+0
| | | | | | | I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-4/+4
|
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-4/+1
| | | To improve type hints throughout the code.
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-262-2/+3
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Add remaining type hints to `synapse.events`. (#11098)Patrick Cloke2021-11-021-1/+1
|
* Annotate `log_function` decorator (#10943)reivilibre2021-10-271-2/+3
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Check *all* auth events for room id and rejection (#11009)Richard van der Hoff2021-10-182-3/+3
| | | | | | | | | | | This fixes a bug where we would accept an event whose `auth_events` include rejected events, if the rejected event was shadowed by another `auth_event` with same `(type, state_key)`. The approach is to pass a list of auth events into `check_auth_rules_for_event` instead of a dict, which of course means updating the call sites. This is an extension of #10956.
* `disallow-untyped-defs` for `synapse.state` (#11004)David Robertson2021-10-063-4/+4
| | | | | * `disallow-untyped-defs` for `synapse.state` Much smaller than I was expecting!
* Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-292-9/+3
| | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* Factor out a separate `EventContext.for_outlier` (#10883)Richard van der Hoff2021-09-221-30/+4
| | | | | | Constructing an EventContext for an outlier is actually really simple, and there's no sense in going via an `async` method in the `StateHandler`. This also means that we can resolve a bunch of FIXMEs.
* Add type hints to state handler. (#10482)Patrick Cloke2021-07-261-11/+15
|
* Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-263-21/+42
| | | | (#10254)
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-153-19/+21
|
* Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston2021-05-051-3/+7
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-222-2/+4
| | | This is no longer required, since we have dropped support for Python 3.5.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-143-3/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-2/+3
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-163-8/+26
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-3/+8
|
* Fix mainline ordering in state res v2 (#8971)Erik Johnston2020-12-181-1/+1
| | | | | | This had two effects 1) it'd give the wrong answer and b) would iterate *all* power levels in the auth chain of each event. The latter of which can be *very* expensive for certain types of IRC bridge rooms that have large numbers of power level changes.
* Pass room_id to get_auth_chain_difference (#8879)Erik Johnston2020-12-042-4/+9
| | | | | This is so that we can choose which algorithm to use based on the room ID.
* Correctly handle unpersisted events when calculating auth chain difference. ↵Erik Johnston2020-12-021-4/+83
| | | | | | | | | (#8827) We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.
* Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-233-3/+3
|
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-1/+1
|
* Report state res metrics to Prometheus and logRichard van der Hoff2020-09-291-20/+124
|
* Move Measure calls into `resolve_events_with_store`Richard van der Hoff2020-09-291-32/+31
|
* Move `resolve_events_with_store` into StateResolutionHandlerRichard van der Hoff2020-09-291-45/+47
|
* Improve logging of state resolution (#8371)Richard van der Hoff2020-09-231-48/+16
| | | | | | | | | | I'd like to get a better insight into what we are doing with respect to state res. The list of state groups we are resolving across should be short (if it isn't, that's a massive problem in itself), so it should be fine to log it in ite entiretly. I've done some grepping and found approximately zero cases in which the "shortcut" code delivered the result, so I've ripped that out too.
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-4/+4
|
* Define StateMap as immutable and add a MutableStateMap type. (#8183)Patrick Cloke2020-08-283-20/+28
|
* Add type hints for state. (#8140)Patrick Cloke2020-08-243-186/+348
|
* Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Convert state resolution to async/await (#7942)Patrick Cloke2020-07-243-120/+97
|
* Add another yield point to state res v2 (#7746)Erik Johnston2020-06-261-2/+10
|
* Yield during large v2 state res. (#7735)Erik Johnston2020-06-242-11/+51
| | | | | | | | | | | State res v2 across large data sets can be very CPU intensive, and if all the relevant events are in the cache the algorithm will run from start to finish within a single reactor tick. This can result in blocking the reactor tick for several seconds, which can have major repercussions on other requests. To fix this we simply add the occaisonal `sleep(0)` during iterations to yield execution until the next reactor tick. The aim is to only do this for large data sets so that we don't impact otherwise quick resolutions.=
* Speed up state res v2 across large state differences. (#7725)Erik Johnston2020-06-191-1/+2
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-153-30/+24
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-3/+1
| | | | variables (#6391)
* Improve get auth chain difference algorithm. (#7095)Erik Johnston2020-03-182-48/+12
| | | | | | | | | | | It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-212-9/+9
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Reduce auth chains fetched during v2 state res. (#6952)Erik Johnston2020-02-192-6/+11
| | | | | | The state res v2 algorithm only cares about the difference between auth chains, so we can pass in the known common state to the `get_auth_chain` storage function so that it can ignore those events.
* s/get_room_version/get_room_version_id/Richard van der Hoff2020-01-311-1/+1
| | | | | ... to make way for a forthcoming get_room_version which returns a RoomVersion object.
* Pass room version object into event_auth.check and check_redaction (#6788)Richard van der Hoff2020-01-282-3/+5
| | | | | | | These are easier to work with than the strings and we normally have one around. This fixes `FederationHander._persist_auth_tree` which was passing a RoomVersion object into event_auth.check instead of a string.
* Add StateMap type alias (#6715)Erik Johnston2020-01-163-8/+11
|
* Merge branch 'master' into developRichard van der Hoff2019-12-311-15/+37
|\
| * Hacks to work around #6605 (#6608)Richard van der Hoff2019-12-311-15/+37
| | | | | | | | | | When we have an event which refers to non-existent auth_events, ignore said events rather than exploding in a ball of fire. Fixes #6605.
| * sanity-checking for events used in state res (#6531)Richard van der Hoff2019-12-163-43/+123
| | | | | | | | | | When we perform state resolution, check that all of the events involved are in the right room.
* | sanity-checking for events used in state res (#6531)Richard van der Hoff2019-12-133-43/+123
| | | | | | | | | | | | | | When we perform state resolution, check that all of the events involved are in the right room.
* | Prevent redacted events from appearing in message search (#6377)Andrew Morgan2019-12-111-1/+2
|/
* Fix bug which caused rejected events to be stored with the wrong room state ↵Richard van der Hoff2019-11-061-89/+83
| | | | | | | | | | | | (#6320) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.
* Improve documentation for EventContext fields (#6319)Richard van der Hoff2019-11-051-0/+3
|
* Port to use state storageErik Johnston2019-10-301-5/+8
|
* add some metrics on the federation sender (#6160)Richard van der Hoff2019-10-031-6/+18
|
* Remove double return statements (#5962)Andrew Morgan2019-09-031-1/+0
| | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
* Replace returnValue with return (#5736)Amber Brown2019-07-233-38/+34
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-06-203-151/+104
|\ | | | | | | erikj/histogram_extremities
| * Run Black. (#5482)Amber Brown2019-06-203-151/+104
| |
* | Add descriptions and remove redundant set(..)Erik Johnston2019-06-191-1/+2
| |
* | Only count non-cache state resolutionErik Johnston2019-06-181-3/+3
| |
* | Use consistent bucketsErik Johnston2019-06-181-1/+1
| |
* | Add metric fo number of state groups in resolutionErik Johnston2019-06-171-0/+10
|/
* Fix grammar and document get_current_users_in_room (#4998)Andrew Morgan2019-04-031-2/+13
|
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-012-14/+13
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Enable room version v3Erik Johnston2019-01-291-1/+1
|
* Drop vdh supportErik Johnston2019-01-291-1/+1
|
* Pass through room version to event authErik Johnston2019-01-253-9/+21
|
* Merge pull request #4307 from matrix-org/erikj/v2_roomsRichard van der Hoff2018-12-241-1/+3
|\ | | | | Add v2 room version
| * Add v2 room versionErik Johnston2018-12-181-1/+3
| |
* | Workaround for non-ascii event ids (#4241)Richard van der Hoff2018-12-031-1/+3
|/ | | | | | It turns out that we accept events with non-ascii IDs, which would later cause an explosion during state res. Fixes #4226
* Add helpers for getting prev and auth events (#4139)Erik Johnston2018-11-062-9/+9
| | | | | | | * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.
* Fix None exception in state res v2Erik Johnston2018-11-021-0/+4
|
* Add STATE_V2_TEST room versionErik Johnston2018-11-011-1/+1
|
* Rename resolve_events_with_factoryErik Johnston2018-10-243-9/+9
|
* Add v2 state res algorithm.Erik Johnston2018-10-162-18/+613
| | | | We hook this up to the vdh test room version.
* Logging improvementsRichard van der Hoff2018-09-261-2/+12
| | | | Some logging tweaks to help with debugging incoming federation transactions
* Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-9/+0
| | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* Remove unnecessary resolve_events_with_state_mapErik Johnston2018-08-222-56/+6
| | | | | We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-201-3/+3
| | | | erikj/refactor_state_handler
* Remove redundant room_version checksErik Johnston2018-08-201-14/+3
|
* Add fast path in state res for zero prev eventsErik Johnston2018-08-091-1/+6
|
* Choose state algorithm based on room versionErik Johnston2018-08-091-11/+93
|
* Refactor state moduleErik Johnston2018-08-092-0/+895
We split out the actual state resolution algorithm to prepare for having multiple versions.