summary refs log tree commit diff
path: root/synapse/storage/events.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move storage classes into a main "data store".Erik Johnston2019-10-211-2489/+0
| | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Replace IN usage with helper funcsErik Johnston2019-10-101-17/+27
|
* Fix inserting bytes as textErik Johnston2019-10-081-4/+2
|
* Use `received_ts` to find uncensored redacted eventsErik Johnston2019-10-011-14/+7
| | | | | | | | Joining against `events` and ordering by `stream_ordering` is inefficient as it forced scanning the entirety of the redactions table. This isn't the case if we use `redactions.received_ts` column as we can then use an index.
* Add received_ts column to redactions.Erik Johnston2019-10-011-9/+11
| | | | | This will allow us to efficiently search for uncensored redactions in the DB before a given time.
* Don't repeatedly attempt to censor events we don't have.Erik Johnston2019-10-011-1/+17
| | | | | | | | | Currently we don't set `have_censored` column if we don't have the target event of a redaction, which means we repeatedly attempt to censor the same non-existant event. When we persist non-redacted events we unset the `have_censored` column for any redactions that target said event.
* Fix commentsErik Johnston2019-09-091-2/+3
|
* Use better names in SQLErik Johnston2019-09-091-5/+8
|
* Fixup commentErik Johnston2019-09-091-4/+12
|
* Don't start looping call unless enabledErik Johnston2019-09-091-1/+2
|
* Handle setting retention period to 0Erik Johnston2019-09-091-1/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-051-7/+8
|\ | | | | | | erikj/censor_redactions
| * Fix and refactor room and user stats (#5971)Erik Johnston2019-09-041-2/+3
| | | | | | Previously the stats were not being correctly populated.
| * Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-5/+5
| | | | | | | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* | Make redaction retention period configurableErik Johnston2019-09-051-2/+4
| |
* | Censor redactions in DB after a monthErik Johnston2019-08-301-1/+87
|/
* Servlet to purge old rooms (#5845)Richard van der Hoff2019-08-221-0/+137
|
* Drop some unused tables. (#5893)Richard van der Hoff2019-08-211-12/+2
| | | These tables are never used, so we may as well drop them.
* Don't unnecessarily block notifying of new events.Erik Johnston2019-08-061-128/+142
| | | | | | | | | | | | | | | | When persisting events we calculate new stream orderings up front. Before we notify about an event all events with lower stream orderings must have finished being persisted. This PR moves the assignment of stream ordering till *after* calculated the new current state and split the batch of events into separate chunks for persistence. This means that if it takes a long time to calculate new current state then it will not block events in other rooms being notified about. This should help reduce some global pauses in the events stream which can last for tens of seconds (if not longer), caused by some particularly expensive state resolutions.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-18/+16
|
* Add membership column to current_state_events table.Erik Johnston2019-07-181-12/+14
| | | | | It turns out that doing a join is surprisingly expensive for the DB to do when room_membership table is larger than the disk cache.
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-2/+2
|
* Fix a number of "Starting txn from sentinel context" warnings (#5605)Richard van der Hoff2019-07-031-1/+8
| | | | Fixes #5602, #5603
* BlackErik Johnston2019-06-201-2/+2
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-06-201-1/+1
|\ | | | | | | erikj/histogram_extremities
| * Run Black. (#5482)Amber Brown2019-06-201-1/+1
| |
* | Add descriptions and remove redundant set(..)Erik Johnston2019-06-191-3/+5
| |
* | Add metrics for len of new extremities persisted.Erik Johnston2019-06-171-1/+23
|/ | | | | Of new events being persisted add metrics for total size of forward extremities and number of unchanged, "stale" extremities.
* Prometheus histograms are cumalativeErik Johnston2019-06-141-1/+2
|
* Expose statistics on extrems to prometheus (#5384)Amber Brown2019-06-131-13/+31
|
* Move event background updates to a separate fileErik Johnston2019-05-301-369/+2
|
* Fixup comments and loggingErik Johnston2019-05-301-9/+12
|
* Log actual number of entries deletedErik Johnston2019-05-291-2/+4
|
* Add DB bg update to cleanup extremities.Erik Johnston2019-05-291-0/+186
| | | | | Due to #5269 we may have extremities in our DB that we shouldn't have, so lets add a cleanup task such to remove those.
* Correctly filter out extremities with soft failed prevs (#5274)Erik Johnston2019-05-291-3/+79
| | | | | | | | | | When we receive a soft failed event we, correctly, *do not* update the forward extremity table with the event. However, if we later receive an event that references the soft failed event we then need to remove the soft failed events prev events from the forward extremities table, otherwise we just build up forward extremities. Fixes #5269
* Exclude soft-failed events from fwd-extremity candidates. (#5146)Richard van der Hoff2019-05-211-2/+7
| | | | | | | | When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.
* Correctly update aggregation counts after redactionErik Johnston2019-05-201-0/+3
|
* Fix relations in worker modeErik Johnston2019-05-161-4/+8
|
* Add simple send_relation API and track in DBErik Johnston2019-05-151-0/+2
|
* drop tables listed in #1830 (#4992)Neil Johnson2019-04-081-12/+0
| | | | | | | | | | | | | | | Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities
* Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-021-6/+18
| | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
* Merge pull request #4955 from matrix-org/rav/merge_state_into_eventsRichard van der Hoff2019-03-281-3/+0
|\ | | | | Combine the CurrentStateDeltaStream into the EventStream
| * Combine the CurrentStateDeltaStream into the EventStreamRichard van der Hoff2019-03-271-3/+0
| |
* | Run `black` on some storage modules that the stats branch touches (#4959)Amber Brown2019-03-291-294/+248
|/
* Make `prev_state` field optionalErik Johnston2019-03-081-15/+0
| | | | | | | The `prev_state` field on events is not specced and so synapse shouldn't explode if an event is missing the field. Fixes #4787
* Implement soft failErik Johnston2019-03-061-0/+1
|
* Batch cache invalidation over replicationErik Johnston2019-02-181-24/+1
| | | | | | | | | | Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.
* Fix flake8 (#4519)Amber Brown2019-01-301-84/+84
|
* Add support for persisting event format versionsErik Johnston2019-01-231-0/+1
| | | | | | | | | | | | | | Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.
* Fix race when persisting create event (#4404)Erik Johnston2019-01-181-1/+12
| | | | | | | | | * Fix race when persisting create event When persisting a chunk of DAG it is sometimes requried to do a state resolution, which requires knowledge of the room version. If this happens while we're persisting the create event then we need to use that event rather than attempting to look it up in the database.
* Add helpers for getting prev and auth events (#4139)Erik Johnston2018-11-061-4/+4
| | | | | | | * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.
* Move _find_unreferenced_groupsErik Johnston2018-10-291-2/+83
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-261-22/+25
|\ | | | | | | erikj/purge_state_groups
| * Refactor state group lookup to reduce DB hits (#4011)Erik Johnston2018-10-251-1/+1
| | | | | | | | | | | | | | | | Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.
| * Update event_auth table for rejected eventsErik Johnston2018-10-161-15/+21
| |
| * Add v2 state res algorithm.Erik Johnston2018-10-161-6/+3
| | | | | | | | We hook this up to the vdh test room version.
* | pep8Erik Johnston2018-10-191-1/+1
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-191-33/+69
|\| | | | | | | erikj/purge_state_groups
| * actually exclude outliersRichard van der Hoff2018-10-031-1/+2
| |
| * remove debuggingRichard van der Hoff2018-10-021-10/+0
| |
| * Fix bug in forward_extremity update logicRichard van der Hoff2018-10-021-33/+78
| | | | | | | | | | An event does not stop being a forward_extremity just because an outlier or rejected event refers to it.
* | Batch process handling state groupsErik Johnston2018-10-191-60/+21
| |
* | Delete unreferened state groups during purgeErik Johnston2018-10-041-6/+27
|/
* commentErik Johnston2018-09-131-0/+2
|
* pep8Erik Johnston2018-09-131-1/+0
|
* Create indices after insertionErik Johnston2018-09-131-14/+18
|
* Make purge history slightly fasterErik Johnston2018-09-131-2/+9
| | | | | Don't pull out events that are outliers and won't be deleted, as nothing should happen to them.
* Speed up lazy loading (#3827)Matthew Hodgson2018-09-121-0/+4
| | | | | * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)
* Port storage/ to Python 3 (#3725)Amber Brown2018-08-311-5/+5
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-201-84/+2
|\ | | | | | | erikj/refactor_state_handler
| * speed up /members and add at= and membership params (#3568)Matthew Hodgson2018-08-151-1/+1
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-151-1/+1
| |\ | | | | | | | | | erikj/split_federation
| | * Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-3/+8
| |\| | | | | | | | | | erikj/split_federation
| * | Move necessary storage functions to worker classesErik Johnston2018-08-061-82/+0
| | |
* | | Choose state algorithm based on room versionErik Johnston2018-08-091-1/+3
| |/ |/|
* | Fix occasional glitches in the synapse_event_persisted_position metricRichard van der Hoff2018-08-071-3/+8
|/ | | | | Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events.
* Merge pull request #3621 from matrix-org/erikj/split_fed_storeErik Johnston2018-08-021-2/+12
|\ | | | | Split out DB writes in federation handler
| * update docsErik Johnston2018-08-011-1/+1
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-301-2/+0
| |\ | | | | | | | | | erikj/split_fed_store
| * | Split out DB writes in federation handlerErik Johnston2018-07-251-2/+12
| | | | | | | | | | | | | | | | | | This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.
* | | Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-4/+10
| | |
* | | Merge pull request #3612 from matrix-org/rav/store_heirarchyRichard van der Hoff2018-07-311-1/+5
|\ \ \ | |_|/ |/| | Make EventStore inherit from EventFederationStore
| * | reinstate explicit include of EventsWorkerStoreRichard van der Hoff2018-07-311-1/+3
| | |
| * | Make EventStore inherit from EventFederationStoreRichard van der Hoff2018-07-261-2/+4
| |/ | | | | | | | | | | | | | | (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.
* | Merge pull request #3614 from matrix-org/rav/stop_populating_event_contentRichard van der Hoff2018-07-261-1/+0
|\ \ | | | | | | Stop populating events.content
| * | Stop populating events.contentRichard van der Hoff2018-07-261-1/+0
| |/ | | | | | | | | | | This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column.
* / Remove some redundant joins on event_edges.room_idRichard van der Hoff2018-07-261-1/+0
|/ | | | | | | | | | We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it.
* Merge pull request #3603 from matrix-org/erikj/handle_outliersErik Johnston2018-07-251-6/+8
|\ | | | | Correctly handle outliers during persist events
| * Actually fix it by adding continueErik Johnston2018-07-251-0/+1
| |
| * Fix typo in conditionalErik Johnston2018-07-251-1/+1
| |
| * Correctly handle outliers during persist eventsErik Johnston2018-07-251-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601
* | Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_moreRichard van der Hoff2018-07-251-5/+4
|\ \ | | | | | | Fix another logcontext leak in _persist_events
| * | Fix another logcontext leak in _persist_eventsRichard van der Hoff2018-07-251-5/+4
| |/ | | | | | | | | | | | | | | We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from
* / Fix occasional 'tuple index out of range' errorRichard van der Hoff2018-07-251-1/+1
|/ | | | | | | This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.
* Expand on docstring comment about return valueErik Johnston2018-07-241-0/+5
|
* Remove unnecessary iteritemsErik Johnston2018-07-241-1/+1
|
* Fixup comment (and indent)Erik Johnston2018-07-241-16/+20
|
* Don't fetch state from the database unless neededErik Johnston2018-07-241-12/+18
|
* Have _get_new_state_after_events return deltaErik Johnston2018-07-241-17/+48
| | | | | | If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.
* Don't require to_delete to have event_idsErik Johnston2018-07-241-36/+55
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-241-12/+23
|\ | | | | | | erikj/speed_up_calculate_state_delta
| * Merge branch 'develop' into rav/logcontext_fixesRichard van der Hoff2018-07-241-11/+21
| |\
| | * fix idiocyRichard van der Hoff2018-07-241-2/+8
| | |
| | * Add some measure blocks to persist_eventsRichard van der Hoff2018-07-231-11/+15
| | | | | | | | | | | | ... to help us figure out where 40% of CPU is going
| * | Logcontext fixesRichard van der Hoff2018-07-241-2/+3
| |/ | | | | | | Fix some random logcontext leaks.
* | Remove unnecessary ifErik Johnston2018-07-241-2/+1
| |
* | Speed up _calculate_state_deltaErik Johnston2018-07-241-10/+3
|/
* CommentsErik Johnston2018-07-231-0/+3
|
* Only get cached state from context in persist_eventErik Johnston2018-07-231-1/+3
| | | | | | We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.
* Use new gettersErik Johnston2018-07-231-1/+1
|
* Run things as background processesRichard van der Hoff2018-07-181-6/+4
| | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
* run isortAmber Brown2018-07-091-19/+15
|
* Invalidate cache on correct threadErik Johnston2018-07-021-1/+2
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+2
|
* Deleting from event_push_actions needs to use an indexMark Haines2018-06-221-1/+8
|
* Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-311-22/+25
| | | | they're not meant to be lazy (#3307)
* cleanupAmber Brown2018-05-221-3/+6
|
* Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-221-24/+53
|\
| * Clarify commentErik Johnston2018-05-161-1/+2
| |
| * CommentsErik Johnston2018-05-161-0/+8
| |
| * Move and rename variableErik Johnston2018-05-161-3/+3
| |
| * Make purge_history operate on tokensErik Johnston2018-05-151-8/+9
| | | | | | | | As we're soon going to change how topological_ordering works
| * Use events_to_purge table rather than tokenErik Johnston2018-05-151-15/+34
| |
* | rest of the changesAmber Brown2018-05-211-17/+11
|/
* Fix up grammarErik Johnston2018-05-031-3/+3
|
* Refactor event storage to not require stateErik Johnston2018-04-271-37/+67
| | | | | | | | | This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.
* Refactor store.have_eventsRichard van der Hoff2018-04-201-7/+42
| | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
* Track where event stream processing have gotten up toErik Johnston2018-04-111-0/+3
|
* Use static JSONEncodersRichard van der Hoff2018-03-291-15/+8
| | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.
* CommentErik Johnston2018-03-271-4/+5
|
* Fix indentErik Johnston2018-03-271-1/+1
|
* CommentErik Johnston2018-03-271-0/+7
|
* Add counter metrics for calculating state deltaErik Johnston2018-03-271-1/+30
| | | | | | This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+0
|\
| * Remove wrong commentErik Johnston2018-03-161-1/+0
| |
| * Replace ujson with simplejsonErik Johnston2018-03-151-2/+2
| |
* | Replace some ujson with simplejson to make it workErik Johnston2018-03-161-1/+1
| |
* | Add Measure block for persist_eventsRichard van der Hoff2018-03-131-4/+5
| | | | | | | | This seems like a useful thing to measure.
* | Fix race in sync when joining roomErik Johnston2018-03-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.
* | Add some caches to help read marker APIErik Johnston2018-03-011-2/+2
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-261-359/+7
|\ \ | | | | | | | | | erikj/handle_unpersisted_events_push
| * | Split EventsWorkerStore into separate fileErik Johnston2018-02-231-360/+5
| | |
| * | Update copyrightErik Johnston2018-02-231-0/+1
| | |
| * | Remove redundant clockErik Johnston2018-02-231-3/+0
| | |
| * | _event_persist_queue shouldn't be in worker storeErik Johnston2018-02-231-4/+4
| | |
| * | Split out get_events and co into a worker storeErik Johnston2018-02-211-345/+350
| | |
* | | Ensure all push actions are deleted from stagingErik Johnston2018-02-201-1/+10
| | |
* | | Refactor _set_push_actions_for_event_and_users_txn to use events_and_contextsErik Johnston2018-02-201-5/+6
|/ /
* | (Really) fix tablescan of event_push_actions on purgeRichard van der Hoff2018-02-161-1/+0
| | | | | | | | | | commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/
* | Remove context.push_actionsErik Johnston2018-02-151-4/+3
| |
* | Update event_push_actions table from staging tableErik Johnston2018-02-151-1/+1
| |
* | purge_history: fix sqlite syntax errorRichard van der Hoff2018-02-141-1/+4
| | | | | | | | apparently sqlite insists on indexes being named
* | purge_history: handle sqlite asshatteryRichard van der Hoff2018-02-141-19/+27
| | | | | | | | | | apparently creating a temporary table commits the transaction. because that's a useful thing.
* | purge_history: fix index useRichard van der Hoff2018-02-141-0/+14
| | | | | | | | | | event_push_actions doesn't have an index on event_id, so we need to specify room_id.
* | Rework event purge to use a temporary tableRichard van der Hoff2018-02-141-35/+58
| | | | | | | | | | ... which should speed things up by reducing the amount of data being shuffled across the connection
* | Fix log message in purge_historyRichard van der Hoff2018-02-131-2/+1
| | | | | | | | (we don't just remove remote events)
* | Merge pull request #2864 from matrix-org/rav/persist_event_cachingRichard van der Hoff2018-02-131-56/+40
|\ \ | | | | | | Use StateResolutionHandler to resolve state in persist_events
| * | style nitRichard van der Hoff2018-02-131-1/+1
| | |
| * | Use StateResolutionHandler to resolve state in persist eventsRichard van der Hoff2018-02-051-48/+24
| | | | | | | | | | | | ... and thus benefit (hopefully) from its cache.
| * | Flatten _get_new_state_after_eventsRichard van der Hoff2018-02-051-44/+46
| | | | | | | | | | | | rejig the if statements to simplify the logic and reduce indentation
| * | Check that events being persisted have state_groupRichard van der Hoff2018-02-051-4/+9
| | |
| * | Add event_map param to resolve_state_groupsRichard van der Hoff2018-02-051-0/+1
| | |
* | | Merge pull request #2858 from matrix-org/rav/purge_updatesRichard van der Hoff2018-02-091-33/+61
|\ \ \ | | | | | | | | delete_local_events for purge_room_history
| * | | purge: move room_depth update to endRichard van der Hoff2018-02-091-6/+12
| | | | | | | | | | | | | | | | ... to avoid locking the table for too long
| * | | delete_local_events for purge_historyRichard van der Hoff2018-02-091-7/+28
| | | | | | | | | | | | | | | | Add a flag which makes the purger delete local events
| * | | purge: Move cache invalidation to more appropriate placeRichard van der Hoff2018-02-091-4/+4
| | | | | | | | | | | | | | | | it was a bit of a non-sequitur there
| * | | bump purge logging to infoRichard van der Hoff2018-02-091-13/+14
| | | | | | | | | | | | | | | | | | | | this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.
| * | | rename delete_old_state -> purge_historyRichard van der Hoff2018-02-091-7/+7
| | | | | | | | | | | | | | | | (beacause it deletes more than state)
* | | | Store state groups separately from events (#2784)Erik Johnston2018-02-061-6/+4
| |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* | | Remove redundant return value from _calculate_state_deltaRichard van der Hoff2018-02-051-10/+23
| | | | | | | | | | | | | | | we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.
* | | factor _get_new_state_after_events out of _calculate_state_deltaRichard van der Hoff2018-01-311-18/+39
| | | | | | | | | | | | This reduces the scope of a bunch of variables
* | | Improve exception handling in persist_eventRichard van der Hoff2018-01-291-3/+3
|/ / | | | | | | | | | | | | | | | | | | | | 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs
* | Merge pull request #2805 from matrix-org/rav/log_state_resRichard van der Hoff2018-01-171-0/+6
|\ \ | | | | | | Log room when doing state resolution
| * | Log room when doing state resolutionRichard van der Hoff2018-01-171-0/+6
| |/ | | | | | | Mostly because it helps figure out what is prompting the resolution
* | Split resolve_events into two functionsRichard van der Hoff2018-01-171-2/+2
| | | | | | | | ... so that the return type doesn't depend on the arg types
* | Fix a logcontext leak in persist_eventsRichard van der Hoff2018-01-161-1/+8
|/ | | | | | | | | | | | | ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)
* Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-131-2/+2
| | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* Fix logcontext handling for persist_eventsRichard van der Hoff2017-10-171-7/+17
| | | | | | | | * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments
* Invalidate cacheErik Johnston2017-10-031-0/+3
|
* Make into listErik Johnston2017-06-291-1/+1
|
* Prefill forward extrems and event to state groupsErik Johnston2017-06-291-0/+5
|
* Merge pull request #2286 from matrix-org/erikj/split_out_user_dirErik Johnston2017-06-161-0/+18
|\ | | | | Split out user directory to a separate process
| * Initial worker implErik Johnston2017-06-161-0/+18
| |
* | Merge pull request #2281 from matrix-org/erikj/phone_home_statsErik Johnston2017-06-151-58/+43
|\ \ | |/ |/| Fix phone home stats
| * Add some more statsErik Johnston2017-06-151-0/+15
| |
| * Fix phone home statsErik Johnston2017-06-141-59/+29
| |
* | Add cache for is_host_joinedErik Johnston2017-06-131-0/+5
| |
* | Merge pull request #2259 from matrix-org/erikj/fix_state_woesErik Johnston2017-06-071-1/+1
|\ \ | | | | | | Fix bug where state_group tables got corrupted
| * | Fix bug where state_group tables got corruptedErik Johnston2017-06-071-1/+1
| |/ | | | | | | | | | | | | | | This is due to the fact that we prefilled caches using txn.call_after, which always gets called including on error. We fix this by making txn.call_after only fire when a transaction completes successfully, which is what we want most of the time anyway.
* | Fix typoErik Johnston2017-05-311-1/+1
| |
* | Add stream change cacheErik Johnston2017-05-311-0/+4
| |
* | Add clobbered event_idErik Johnston2017-05-301-0/+1
| |
* | Add current_state_delta_stream tableErik Johnston2017-05-301-8/+23
| |
* | Only store event_auth for state eventsErik Johnston2017-05-241-0/+1
|/
* Merge pull request #2224 from matrix-org/erikj/prefill_stateErik Johnston2017-05-161-5/+10
|\ | | | | Prefill state caches
| * Remove spurious merge artifactsErik Johnston2017-05-161-13/+0
| |
| * CommentsErik Johnston2017-05-151-3/+3
| |
| * Add more granular event send metricsErik Johnston2017-05-151-0/+12
| |
| * Prefill state cachesErik Johnston2017-05-151-2/+8
| |
* | Don't create event_search index on sqliteRichard van der Hoff2017-05-111-0/+1
| | | | | | | | ... because the table is virtual
* | Add more logging for purgingRichard van der Hoff2017-05-111-7/+14
| | | | | | | | Log the number of events we will be deleting at info.
* | Add an index to event_searchRichard van der Hoff2017-05-111-0/+11
|/ | | | - to make the purge API quicker
* Tidy purge code and add some commentsRichard van der Hoff2017-05-111-12/+19
| | | | Try to make this clearer with more comments and some variable renames
* Don't de-delta state groups we're about to deleteRichard van der Hoff2017-05-101-11/+9
|
* add some logging to purge_historyRichard van der Hoff2017-05-101-4/+21
|
* Expand docstring a bitErik Johnston2017-05-081-1/+7
|
* Don't update event cache hit ratio from get_joined_usersErik Johnston2017-05-081-2/+11
| | | | | Otherwise the hit ration of plain get_events gets completely skewed by calls to get_joined_users* functions.
* Revert "Prefill state caches"Erik Johnston2017-05-041-11/+5
|
* CommentsErik Johnston2017-05-031-3/+3
|
* Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_stateErik Johnston2017-05-031-0/+18
|\
| * Add more granular event send metricsErik Johnston2017-05-021-0/+16
| |
* | Prefill state cachesErik Johnston2017-05-021-2/+6
|/
* Correct logic in is_event_afterLuke Barnard2017-04-131-1/+1
|
* Simplify is_event_after logicLuke Barnard2017-04-121-10/+3
|
* travis flake8..Luke Barnard2017-04-111-0/+1
|
* Refactor event ordering check to events storeLuke Barnard2017-04-111-0/+28
|
* Add new storage functions for new replicationErik Johnston2017-03-301-0/+88
| | | | | The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one.
* TypoErik Johnston2017-03-271-1/+1
|
* Short circuit if all new events have same state groupErik Johnston2017-03-271-36/+51
|
* Use iter(items|values)Erik Johnston2017-03-241-18/+14
|
* User Cursor.__iter__ instead of fetchallErik Johnston2017-03-231-1/+1
| | | | This prevents unnecessary construction of lists
* Merge pull request #2033 from matrix-org/erikj/repl_speedErik Johnston2017-03-211-17/+14
|\ | | | | Don't send the full event json over replication
| * Don't send the full event json over replicationErik Johnston2017-03-171-17/+14
| |
* | Avoid resetting state on rejected eventsRichard van der Hoff2017-03-171-4/+4
| | | | | | | | | | | | | | When we get a rejected event, give it the same state_group as its prev_event, rather than no state_group at all. This should fix https://github.com/matrix-org/synapse/issues/1935.
* | Refactoring and cleanupsRichard van der Hoff2017-03-171-60/+213
|/ | | | | | | | | | | | A few non-functional changes: * A bunch of docstrings to document types * Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more readable. * Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid mind-bending conditional. * Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid mind-bending conditional.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2017-03-141-3/+28
|\ | | | | | | erikj/public_list_speed
| * Fix current_state_events table to not lieErik Johnston2017-03-141-3/+28
| | | | | | | | | | If we try and persist two state events that have the same ancestor we calculate the wrong current state when persisting those events.
* | Get current state by using current_state_events tableErik Johnston2017-03-101-10/+8
|/
* Remove needless checkErik Johnston2017-02-271-2/+0
|
* It should be allErik Johnston2017-02-271-1/+1
|
* Don't fetch current state in common caseErik Johnston2017-02-271-0/+17
| | | | | | Currently we fetch the list of current state events whenever we send something in a room. This is overkill for the common case of persisting a simple chain of non-state events, so lets handle that case specially.
* Ignore new rejected events when working out forward extremeties.Erik Johnston2017-02-081-9/+6
|
* Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_eventsErik Johnston2017-02-011-0/+2
|\ | | | | Add a small cache get_all_new_events
| * Add a small cache get_all_new_eventsErik Johnston2017-02-011-0/+2
| |
* | Only invalidate membership caches based on the cache streamErik Johnston2017-01-311-20/+0
|/ | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive.
* Stream cache invalidations for room membership storage functionsErik Johnston2017-01-301-2/+6
|
* TypoErik Johnston2017-01-231-1/+1
|
* Refactor to calculate state delta outside transactionErik Johnston2017-01-231-87/+118
|
* CommentsErik Johnston2017-01-201-8/+19
|
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-26/+52
|
* SpellingErik Johnston2017-01-201-1/+1
|
* Update all call sites after renameErik Johnston2017-01-201-1/+1
|
* CommentsErik Johnston2017-01-201-0/+11
|
* Calculate the forward extremeties onceErik Johnston2017-01-201-56/+86
|
* Derive current_state_events from state groupsErik Johnston2017-01-201-70/+118
|
* Lower loading events log to DEBUGErik Johnston2017-01-171-2/+2
|
* More logging for the linearizer and for get_eventsMark Haines2017-01-051-0/+2
|
* New Flake8 fixesErik Johnston2016-11-151-0/+1
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-11 01:02:04 +0000