summary refs log tree commit diff
path: root/synapse/storage/state.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-13/+43
| | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
* Merge pull request #3382 from matrix-org/rav/optimise_state_groupsRichard van der Hoff2018-06-221-5/+1
|\ | | | | Optimise state_group_cache update
| * Optimise state_group_cache updateRichard van der Hoff2018-06-111-5/+1
| | | | | | | | | | (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.
* | Fix sql error in _get_state_groups_from_groupsRichard van der Hoff2018-06-061-1/+1
| | | | | | | | | | If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.
* | Add hacky cache factor override systemErik Johnston2018-06-041-2/+2
|/
* Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-311-22/+25
| | | | they're not meant to be lazy (#3307)
* fix bug #2926 (loading all state for a given type from the DB if the ↵Matthew Hodgson2018-03-131-7/+27
| | | | | state_key is None) (#2990) Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
* Don't serialize current state over replicationErik Johnston2018-02-151-0/+14
|
* Fix state group storage bug in workersErik Johnston2018-02-151-41/+41
| | | | | We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.
* Store state groups separately from events (#2784)Erik Johnston2018-02-061-96/+100
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Pull out bits of StateStore to a mixinRichard van der Hoff2017-11-141-204/+220
| | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.
* Revert "Revert "move _state_group_cache to statestore""Richard van der Hoff2017-11-141-7/+12
| | | | | | | We're going to fix this properly on this branch, so that the _state_group_cache can end up in StateGroupReadStore. This reverts commit ab335edb023d66cd0be439e045b10ca104b73cb5.
* Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-131-2/+2
| | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* Revert "move _state_group_cache to statestore"Erik Johnston2017-11-131-12/+7
| | | | This reverts commit f5cf3638e9c6086e1c33ddad8eda9298cf53a58e.
* move _state_group_cache to statestoreRichard van der Hoff2017-11-071-7/+12
| | | | this is internal to statestore, so let's keep it there.
* Prefill forward extrems and event to state groupsErik Johnston2017-06-291-2/+8
|
* Fix replicationErik Johnston2017-06-091-1/+1
|
* CommentsErik Johnston2017-06-091-0/+3
|
* Cache state deltasErik Johnston2017-06-091-3/+12
|
* Ensure we don't use unpersisted state group as prev groupErik Johnston2017-06-081-0/+13
|
* CommentsErik Johnston2017-06-071-0/+6
|
* Faster cache for get_joined_hostsErik Johnston2017-05-251-0/+33
|
* Make get_state_groups_from_groups faster.Erik Johnston2017-05-171-29/+11
| | | | | | | | | Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.
* Take a copy before prefilling, as it may be a frozendictErik Johnston2017-05-161-1/+1
|
* Update commentErik Johnston2017-05-151-1/+2
|
* CommentsErik Johnston2017-05-151-0/+3
|
* Prefill state cachesErik Johnston2017-05-151-0/+8
|
* Revert "Prefill state caches"Erik Johnston2017-05-041-12/+0
|
* Update commentErik Johnston2017-05-031-1/+2
|
* CommentsErik Johnston2017-05-031-0/+3
|
* Prefill state cachesErik Johnston2017-05-021-0/+8
|
* Make state caches cache in asciiErik Johnston2017-04-251-2/+3
|
* Reduce _get_state_group_for_event cache sizeErik Johnston2017-04-251-1/+1
|
* Remove unused cacheErik Johnston2017-04-241-6/+1
|
* CommentErik Johnston2017-04-071-0/+9
|
* Speed up get_current_state_idsErik Johnston2017-04-071-10/+17
| | | | | | | | | | | | Using _simple_select_list is fairly expensive for functions that return a lot of rows and/or get called a lot. (This is because it carefully constructs a list of dicts). get_current_state_ids gets called a lot on startup and e.g. when the IRC bridge decided to send tonnes of joins/leaves (as it invalidates the cache). We therefore replace it with a custon txn function that builds up the final result dict without building up and intermediate representation.
* Increase cache size for _get_state_group_for_eventErik Johnston2017-03-291-1/+1
|
* Replace some calls to cursor_to_dictErik Johnston2017-03-241-4/+4
| | | | | cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to.
* Use iter(items|values)Erik Johnston2017-03-241-27/+28
|
* User Cursor.__iter__ instead of fetchallErik Johnston2017-03-231-2/+1
| | | | This prevents unnecessary construction of lists
* Avoid resetting state on rejected eventsRichard van der Hoff2017-03-171-0/+10
| | | | | | | When we get a rejected event, give it the same state_group as its prev_event, rather than no state_group at all. This should fix https://github.com/matrix-org/synapse/issues/1935.
* Get current state by using current_state_events tableErik Johnston2017-03-101-1/+13
|
* Fix bugs in the /keys/changes apiRichard van der Hoff2017-02-141-1/+13
| | | | | | | | | | | | | | | * `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were passing a `RoomStreamToken`, which meant that it returned the *current* extremities, rather than those corresponding to the `from_token`. However: * `get_state_ids_for_events` required a second ('types') parameter; this meant that a `TypeError` was thrown and we ended up acting as though there was *no* prev state. * `get_state_ids_for_events` actually returns a map from event_id to state dictionary - just looking up the state keys in it again meant that we acted as though there was no prev state. We now check if each member's state has changed since *any* of the extremities. Also add/fix some comments.
* Add an index to make membership queries fasterErik Johnston2017-01-311-0/+8
|
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-52/+0
|
* Up cache max entries for stateErik Johnston2017-01-161-1/+1
|
* Increase cache size limitErik Johnston2017-01-161-1/+1
|
* Optionally measure size of cache by sum of length of valuesErik Johnston2017-01-131-1/+1
|
* fix annoying typosMatthew Hodgson2017-01-051-1/+1
|
* Correctly intern keys in state cacheErik Johnston2016-11-081-1/+4
|
* Remove duplicationErik Johnston2016-09-271-31/+19
|
* Remove unused variableErik Johnston2016-09-271-3/+0
|
* Fix perf of fetching state in SQLiteErik Johnston2016-09-261-20/+41
|
* Fix reindexErik Johnston2016-09-141-15/+16
|
* Ensure we don't mutate state cache entriesErik Johnston2016-09-141-20/+25
|
* Create new index concurrentlyErik Johnston2016-09-121-1/+1
|
* Change state fetch query for postgres to be fasterErik Johnston2016-09-121-19/+35
| | | | | | It turns out that postgres doesn't like doing a list of OR's and is about 1000x slower, so we just issue a query for each specific type seperately.
* Reindex state_groups_state after pruningErik Johnston2016-09-081-0/+33
|
* Temporarily disable sequential scans for state fetchingErik Johnston2016-09-081-0/+4
|
* CommentErik Johnston2016-09-071-0/+4
|
* Add appopriate framing clauseErik Johnston2016-09-071-0/+1
|
* Use windowing function to make use of indexErik Johnston2016-09-071-5/+5
|
* Scale the batch size so that we're not bitten by the minimumErik Johnston2016-09-051-1/+5
|
* Correctly handle reindexing state groups that already have an edgeErik Johnston2016-09-051-0/+19
|
* Take value in a better wayErik Johnston2016-09-051-2/+2
|
* Comment about sqlite and WITH RECURSIVEErik Johnston2016-09-051-0/+4
|
* Add commentsErik Johnston2016-09-051-41/+38
|
* Add upgrade script that will slowly prune state_groups_state entriesErik Johnston2016-09-051-75/+203
|
* Limit the length of state chainsErik Johnston2016-09-021-22/+78
|
* Move to storing state_groups_state as deltasErik Johnston2016-09-011-59/+102
|
* Use state_groups table to test existenceErik Johnston2016-08-311-1/+1
|
* Correctly handle the difference between prev and current stateErik Johnston2016-08-311-3/+0
|
* Generate state group ids in state layerErik Johnston2016-08-311-7/+17
|
* Make _state_groups_id_gen a normal IdGeneratorErik Johnston2016-08-301-3/+0
|
* Make sync not pull out full stateErik Johnston2016-08-251-0/+33
|
* Replace context.current_state with context.current_state_idsErik Johnston2016-08-251-8/+8
|
* Pull out event ids rather than full events for stateErik Johnston2016-08-251-17/+38
|
* Remove lru optionErik Johnston2016-08-191-2/+2
|
* Add cache to _get_state_groups_from_groupsErik Johnston2016-04-191-3/+16
|
* Make the cache objects be per instance rather than being globalMark Haines2016-04-061-2/+2
|
* Merge pull request #686 from matrix-org/markjh/doc_stringsMark Haines2016-04-011-5/+8
|\ | | | | Use google style doc strings.
| * Use google style doc strings.Mark Haines2016-04-011-5/+8
| | | | | | | | | | | | | | pycharm supports them so there is no need to use the other format. Might as well convert the existing strings to reduce the risk of people accidentally cargo culting the wrong doc string format.
* | Use a stream id generator for backfilled idsMark Haines2016-04-011-1/+1
|/
* Add a replication stream for state groupsMark Haines2016-03-301-0/+30
|
* Use a stream id generator to assign state group idsMark Haines2016-03-301-8/+8
|
* Don't bother interning keys that are already internedErik Johnston2016-03-231-3/+1
|
* Intern all the thingsErik Johnston2016-03-231-5/+5
|
* String internErik Johnston2016-03-231-3/+9
|
* Don't cache events in get_current_state_for_keyErik Johnston2016-03-231-7/+9
|
* Don't cache events in _state_group_cacheErik Johnston2016-03-221-60/+48
| | | | | | | | | Instead, simply cache the event ids, relying on the event cache to cache the actual events. The problem was that while the state groups cache was limited in the number of groups it could hold, each individual group could consist of thousands of events.
* Load the current id in the IdGenerator constructorMark Haines2016-03-011-1/+1
| | | | | | | | | Rather than loading them lazily. This allows us to remove all the yield statements and spurious arguments for the get_next methods. It also allows us to replace all instances of get_next_txn with get_next since get_next no longer needs to access the db.
* Batch fetch _get_state_groups_from_groupsErik Johnston2016-02-101-32/+34
|
* Use _simple_select_many for _get_state_group_for_eventsErik Johnston2016-02-101-16/+10
|
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Fix a few race conditions in the state calculationRichard van der Hoff2015-11-131-0/+14
| | | | | | | | | | | | | | | | | | | Be a bit more careful about how we calculate the state to be returned by /sync. In a few places, it was possible for /sync to return slightly later state than that represented by the next_batch token and the timeline. In particular, the following cases were susceptible: * On a full state sync, for an active room * During a per-room incremental sync with a timeline gap * When the user has just joined a room. (Refactor check_joined_room to make it less magical) Also, use store.get_state_for_events() (and thus the existing stategroups) to calculate the state corresponding to a particular sync position, rather than state_handler.compute_event_context(), which recalculates from first principles (and tends to miss some state). Merged from PR https://github.com/matrix-org/synapse/pull/372
* Remove unused room_id parameterErik Johnston2015-10-121-6/+5
|
* synapse/storage/state.py: _make_group_id was unusedMark Haines2015-09-231-6/+0
|
* Actually check if event_id isn't returned by _get_state_groupsErik Johnston2015-08-211-2/+9
|
* Remove newline because vertical whitespace makes mjark sadErik Johnston2015-08-181-2/+1
|
* Rejig the code to make it nicerErik Johnston2015-08-181-3/+2
|
* Ensure we never return a None event from _get_state_for_groupsErik Johnston2015-08-181-3/+8
|
* Reduce cache size from obscenely large to quite largeErik Johnston2015-08-181-1/+1
|
* Fix bug where we were leaking None into state event listsErik Johnston2015-08-171-2/+3
|
* Remove some vertical spaceErik Johnston2015-08-131-36/+9
|
* Replace list comprehensionErik Johnston2015-08-131-8/+3
|
* CommentErik Johnston2015-08-121-1/+2
|
* Explain why we're prefilling dict with NonesErik Johnston2015-08-121-0/+4
|
* Better variable nameErik Johnston2015-08-121-5/+5
|
* Split _get_state_for_group_from_cache into twoErik Johnston2015-08-121-32/+53
|
* Move all the caches into their own package, synapse.util.cachesErik Johnston2015-08-111-1/+4
|
* CommentsErik Johnston2015-08-111-3/+11
|
* Fix application of ACLsErik Johnston2015-08-111-3/+3
|
* Fix state cacheErik Johnston2015-08-111-36/+49
|
* Clean up StateStoreErik Johnston2015-08-101-62/+11
|
* Remove unused functionErik Johnston2015-08-101-7/+0
|
* Do bounds checkErik Johnston2015-08-101-1/+4
|
* Line lengthErik Johnston2015-08-101-1/+2
|
* Merge branch 'erikj/cache_varargs_interface' of ↵Erik Johnston2015-08-071-1/+0
|\ | | | | | | github.com:matrix-org/synapse into erikj/dictionary_cache
| * Remove unnecessary cacheErik Johnston2015-08-071-2/+1
| |
* | Batch up various DB requests for event -> stateErik Johnston2015-08-071-77/+142
| |
* | Remove unncessary cacheErik Johnston2015-08-071-1/+0
| |
* | Store absence of state in cacheErik Johnston2015-08-071-6/+16
| |
* | Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse ↵Erik Johnston2015-08-051-3/+2
|\| | | | | | | into erikj/dictionary_cache
| * Add support for using keyword arguments with cached functionsErik Johnston2015-07-271-3/+2
| |
* | Use dictionary cache to do group -> state fetchingErik Johnston2015-08-051-65/+126
| |
* | Add commentErik Johnston2015-08-041-0/+15
| |
* | Speed up event filtering (for ACL) logicErik Johnston2015-08-041-38/+79
|/
* Merge pull request #193 from matrix-org/erikj/bulk_persist_eventErik Johnston2015-07-141-12/+24
|\ | | | | Add bulk insert events API
| * Add bulk insert events APIErik Johnston2015-06-251-12/+24
| |
* | Add basic impl for room history ACL on GET /messages client APIErik Johnston2015-07-021-2/+61
|/
* Add cache to get_state_groupsErik Johnston2015-06-031-8/+12
|
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2015-05-221-1/+24
|\
| * Add a cache for get_current_state with state_keyMark Haines2015-05-211-1/+24
| |
* | Don't fetch prev_content for current_stateErik Johnston2015-05-151-1/+1
| |
* | Make store.get_current_state fetch events asynclyErik Johnston2015-05-151-3/+4
| |
* | Move fetching of events into their own transactionsErik Johnston2015-05-141-2/+0
| |
* | Refactor _get_eventsErik Johnston2015-05-141-1/+1
| |
* | Fetch events from events_id in their own transactionsErik Johnston2015-05-131-5/+5
| |
* | Fetch events in bulkErik Johnston2015-05-131-21/+1
| |
* | Don't fetch redaction and rejection stuff for each event, so we can use ↵Erik Johnston2015-05-131-2/+5
| | | | | | | | index only scan
* | Temp turn off checking for rejections and redactionsErik Johnston2015-05-131-5/+2
| |
* | defer.gatherResults loopErik Johnston2015-05-131-1/+7
| |
* | Batch fetching of events for state groupsErik Johnston2015-05-131-1/+24
| |
* | Load events for state group seperatelyErik Johnston2015-05-131-3/+9
|/
* Add support for using executemanyErik Johnston2015-05-051-7/+9
|
* Remove pointless joinErik Johnston2015-04-301-5/+4
|
* Need more yieldErik Johnston2015-04-301-1/+1
|
* Add missing paramErik Johnston2015-04-301-1/+1
|
* Fix up get_current_state and get_room_name_and_aliases queries to parse ↵Erik Johnston2015-04-301-16/+19
| | | | events in transaction
* Don't use sub queries, it makes postgres sadErik Johnston2015-04-301-11/+5
|
* Don't needlessly join on state_eventsErik Johnston2015-04-301-3/+2
|
* For backwards compat, make state_groups.id have a type of int, not varcharErik Johnston2015-04-131-1/+1
|
* PEP8Erik Johnston2015-04-071-1/+0
|
* Don't use AUTOINCREMENT, use an in memory versionErik Johnston2015-04-071-3/+3
|
* Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.Erik Johnston2015-03-231-1/+0
|
* Merge branch 'develop' of github.com:matrix-org/synapse into mysqlErik Johnston2015-03-201-0/+1
|\
| * Rearrange storage modulesErik Johnston2015-03-201-0/+32
| |
* | Rearrange storage modulesErik Johnston2015-03-201-0/+32
| |
* | Convert storage layer to be mysql compatibleErik Johnston2015-03-191-2/+8
|/
* Copy dict of context.current_state before changing it.Erik Johnston2015-03-161-1/+1
|
* Remove unused functionsMark Haines2015-01-131-6/+0
|
* Merge pull request #28 from matrix-org/erikj-perfMark Haines2015-01-061-8/+6
|\ | | | | Database performance improvements.
| * Remove debug linesErik Johnston2015-01-061-7/+0
| |
| * Add some debug loggingErik Johnston2015-01-061-0/+12
| |
| * Test some ideas that might help performance a bitErik Johnston2014-12-171-8/+1
| |
* | Update copyright noticesMark Haines2015-01-061-1/+1
|/
* Various typos and bug fixes.Erik Johnston2014-12-081-3/+3
|
* Start making more things use EventContext rather than event.*Erik Johnston2014-12-051-4/+9
|
* Correctly handle the case where we get an event for an unknown room, which ↵Erik Johnston2014-11-261-1/+1
| | | | turns out we are actually in
* Document StateStore and use transactionsErik Johnston2014-11-121-28/+56
|
* Fix bugs with invites/joins across federatiom.Erik Johnston2014-11-121-3/+6
| | | | | Both in terms of auth and not trying to fetch missing PDUs for invites, joins etc.
* Fix bug where we /always/ created a new state groupErik Johnston2014-11-111-7/+2
|
* Add transaction level logging and timing information. Add a _simple_delete ↵Erik Johnston2014-10-281-0/+1
| | | | method
* Use state groups to get current state. Make join dance actually work.Erik Johnston2014-10-171-0/+3
|
* Begin making auth use event.old_state_eventsErik Johnston2014-10-151-1/+1
|
* Add missing package storate.stateErik Johnston2014-10-151-0/+97
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-25 20:21:40 +0000