Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Add helpers for getting prev and auth events (#4139) | Erik Johnston | 2018-11-06 | 1 | -4/+4 |
| | | | | | | | * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions. | ||||
* | Move _find_unreferenced_groups | Erik Johnston | 2018-10-29 | 1 | -2/+83 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-10-26 | 1 | -22/+25 |
|\ | | | | | | | erikj/purge_state_groups | ||||
| * | Refactor state group lookup to reduce DB hits (#4011) | Erik Johnston | 2018-10-25 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit. | ||||
| * | Update event_auth table for rejected events | Erik Johnston | 2018-10-16 | 1 | -15/+21 |
| | | |||||
| * | Add v2 state res algorithm. | Erik Johnston | 2018-10-16 | 1 | -6/+3 |
| | | | | | | | | We hook this up to the vdh test room version. | ||||
* | | pep8 | Erik Johnston | 2018-10-19 | 1 | -1/+1 |
| | | |||||
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-10-19 | 1 | -33/+69 |
|\| | | | | | | | erikj/purge_state_groups | ||||
| * | actually exclude outliers | Richard van der Hoff | 2018-10-03 | 1 | -1/+2 |
| | | |||||
| * | remove debugging | Richard van der Hoff | 2018-10-02 | 1 | -10/+0 |
| | | |||||
| * | Fix bug in forward_extremity update logic | Richard van der Hoff | 2018-10-02 | 1 | -33/+78 |
| | | | | | | | | | | An event does not stop being a forward_extremity just because an outlier or rejected event refers to it. | ||||
* | | Batch process handling state groups | Erik Johnston | 2018-10-19 | 1 | -60/+21 |
| | | |||||
* | | Delete unreferened state groups during purge | Erik Johnston | 2018-10-04 | 1 | -6/+27 |
|/ | |||||
* | comment | Erik Johnston | 2018-09-13 | 1 | -0/+2 |
| | |||||
* | pep8 | Erik Johnston | 2018-09-13 | 1 | -1/+0 |
| | |||||
* | Create indices after insertion | Erik Johnston | 2018-09-13 | 1 | -14/+18 |
| | |||||
* | Make purge history slightly faster | Erik Johnston | 2018-09-13 | 1 | -2/+9 |
| | | | | | Don't pull out events that are outliers and won't be deleted, as nothing should happen to them. | ||||
* | Speed up lazy loading (#3827) | Matthew Hodgson | 2018-09-12 | 1 | -0/+4 |
| | | | | | * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840) | ||||
* | Port storage/ to Python 3 (#3725) | Amber Brown | 2018-08-31 | 1 | -5/+5 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-20 | 1 | -84/+2 |
|\ | | | | | | | erikj/refactor_state_handler | ||||
| * | speed up /members and add at= and membership params (#3568) | Matthew Hodgson | 2018-08-15 | 1 | -1/+1 |
| | | |||||
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-15 | 1 | -1/+1 |
| |\ | | | | | | | | | | erikj/split_federation | ||||
| | * | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) | Amber Brown | 2018-08-10 | 1 | -1/+1 |
| | | | |||||
| * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-09 | 1 | -3/+8 |
| |\| | | | | | | | | | | erikj/split_federation | ||||
| * | | Move necessary storage functions to worker classes | Erik Johnston | 2018-08-06 | 1 | -82/+0 |
| | | | |||||
* | | | Choose state algorithm based on room version | Erik Johnston | 2018-08-09 | 1 | -1/+3 |
| |/ |/| | |||||
* | | Fix occasional glitches in the synapse_event_persisted_position metric | Richard van der Hoff | 2018-08-07 | 1 | -3/+8 |
|/ | | | | | Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events. | ||||
* | Merge pull request #3621 from matrix-org/erikj/split_fed_store | Erik Johnston | 2018-08-02 | 1 | -2/+12 |
|\ | | | | | Split out DB writes in federation handler | ||||
| * | update docs | Erik Johnston | 2018-08-01 | 1 | -1/+1 |
| | | |||||
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-07-30 | 1 | -2/+0 |
| |\ | | | | | | | | | | erikj/split_fed_store | ||||
| * | | Split out DB writes in federation handler | Erik Johnston | 2018-07-25 | 1 | -2/+12 |
| | | | | | | | | | | | | | | | | | | This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers. | ||||
* | | | Python 3: Convert some unicode/bytes uses (#3569) | Amber Brown | 2018-08-02 | 1 | -4/+10 |
| | | | |||||
* | | | Merge pull request #3612 from matrix-org/rav/store_heirarchy | Richard van der Hoff | 2018-07-31 | 1 | -1/+5 |
|\ \ \ | |_|/ |/| | | Make EventStore inherit from EventFederationStore | ||||
| * | | reinstate explicit include of EventsWorkerStore | Richard van der Hoff | 2018-07-31 | 1 | -1/+3 |
| | | | |||||
| * | | Make EventStore inherit from EventFederationStore | Richard van der Hoff | 2018-07-26 | 1 | -2/+4 |
| |/ | | | | | | | | | | | | | | | (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py. | ||||
* | | Merge pull request #3614 from matrix-org/rav/stop_populating_event_content | Richard van der Hoff | 2018-07-26 | 1 | -1/+0 |
|\ \ | | | | | | | Stop populating events.content | ||||
| * | | Stop populating events.content | Richard van der Hoff | 2018-07-26 | 1 | -1/+0 |
| |/ | | | | | | | | | | | This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column. | ||||
* / | Remove some redundant joins on event_edges.room_id | Richard van der Hoff | 2018-07-26 | 1 | -1/+0 |
|/ | | | | | | | | | | We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it. | ||||
* | Merge pull request #3603 from matrix-org/erikj/handle_outliers | Erik Johnston | 2018-07-25 | 1 | -6/+8 |
|\ | | | | | Correctly handle outliers during persist events | ||||
| * | Actually fix it by adding continue | Erik Johnston | 2018-07-25 | 1 | -0/+1 |
| | | |||||
| * | Fix typo in conditional | Erik Johnston | 2018-07-25 | 1 | -1/+1 |
| | | |||||
| * | Correctly handle outliers during persist events | Erik Johnston | 2018-07-25 | 1 | -6/+7 |
| | | | | | | | | | | | | | | | | | | | | | | | | We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601 | ||||
* | | Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more | Richard van der Hoff | 2018-07-25 | 1 | -5/+4 |
|\ \ | | | | | | | Fix another logcontext leak in _persist_events | ||||
| * | | Fix another logcontext leak in _persist_events | Richard van der Hoff | 2018-07-25 | 1 | -5/+4 |
| |/ | | | | | | | | | | | | | | | We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from | ||||
* / | Fix occasional 'tuple index out of range' error | Richard van der Hoff | 2018-07-25 | 1 | -1/+1 |
|/ | | | | | | | This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying. | ||||
* | Expand on docstring comment about return value | Erik Johnston | 2018-07-24 | 1 | -0/+5 |
| | |||||
* | Remove unnecessary iteritems | Erik Johnston | 2018-07-24 | 1 | -1/+1 |
| | |||||
* | Fixup comment (and indent) | Erik Johnston | 2018-07-24 | 1 | -16/+20 |
| | |||||
* | Don't fetch state from the database unless needed | Erik Johnston | 2018-07-24 | 1 | -12/+18 |
| | |||||
* | Have _get_new_state_after_events return delta | Erik Johnston | 2018-07-24 | 1 | -17/+48 |
| | | | | | | If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state. | ||||
* | Don't require to_delete to have event_ids | Erik Johnston | 2018-07-24 | 1 | -36/+55 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-07-24 | 1 | -12/+23 |
|\ | | | | | | | erikj/speed_up_calculate_state_delta | ||||
| * | Merge branch 'develop' into rav/logcontext_fixes | Richard van der Hoff | 2018-07-24 | 1 | -11/+21 |
| |\ | |||||
| | * | fix idiocy | Richard van der Hoff | 2018-07-24 | 1 | -2/+8 |
| | | | |||||
| | * | Add some measure blocks to persist_events | Richard van der Hoff | 2018-07-23 | 1 | -11/+15 |
| | | | | | | | | | | | | ... to help us figure out where 40% of CPU is going | ||||
| * | | Logcontext fixes | Richard van der Hoff | 2018-07-24 | 1 | -2/+3 |
| |/ | | | | | | | Fix some random logcontext leaks. | ||||
* | | Remove unnecessary if | Erik Johnston | 2018-07-24 | 1 | -2/+1 |
| | | |||||
* | | Speed up _calculate_state_delta | Erik Johnston | 2018-07-24 | 1 | -10/+3 |
|/ | |||||
* | Comments | Erik Johnston | 2018-07-23 | 1 | -0/+3 |
| | |||||
* | Only get cached state from context in persist_event | Erik Johnston | 2018-07-23 | 1 | -1/+3 |
| | | | | | | We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation. | ||||
* | Use new getters | Erik Johnston | 2018-07-23 | 1 | -1/+1 |
| | |||||
* | Run things as background processes | Richard van der Hoff | 2018-07-18 | 1 | -6/+4 |
| | | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally). | ||||
* | run isort | Amber Brown | 2018-07-09 | 1 | -19/+15 |
| | |||||
* | Invalidate cache on correct thread | Erik Johnston | 2018-07-02 | 1 | -1/+2 |
| | |||||
* | Attempt to be more performant on PyPy (#3462) | Amber Brown | 2018-06-28 | 1 | -1/+2 |
| | |||||
* | Deleting from event_push_actions needs to use an index | Mark Haines | 2018-06-22 | 1 | -1/+8 |
| | |||||
* | Consistently use six's iteritems and wrap lazy keys/values in list() if ↵ | Amber Brown | 2018-05-31 | 1 | -22/+25 |
| | | | | they're not meant to be lazy (#3307) | ||||
* | cleanup | Amber Brown | 2018-05-22 | 1 | -3/+6 |
| | |||||
* | Merge remote-tracking branch 'origin/develop' into 3218-official-prom | Amber Brown | 2018-05-22 | 1 | -24/+53 |
|\ | |||||
| * | Clarify comment | Erik Johnston | 2018-05-16 | 1 | -1/+2 |
| | | |||||
| * | Comments | Erik Johnston | 2018-05-16 | 1 | -0/+8 |
| | | |||||
| * | Move and rename variable | Erik Johnston | 2018-05-16 | 1 | -3/+3 |
| | | |||||
| * | Make purge_history operate on tokens | Erik Johnston | 2018-05-15 | 1 | -8/+9 |
| | | | | | | | | As we're soon going to change how topological_ordering works | ||||
| * | Use events_to_purge table rather than token | Erik Johnston | 2018-05-15 | 1 | -15/+34 |
| | | |||||
* | | rest of the changes | Amber Brown | 2018-05-21 | 1 | -17/+11 |
|/ | |||||
* | Fix up grammar | Erik Johnston | 2018-05-03 | 1 | -3/+3 |
| | |||||
* | Refactor event storage to not require state | Erik Johnston | 2018-04-27 | 1 | -37/+67 |
| | | | | | | | | | This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones. | ||||
* | Refactor store.have_events | Richard van der Hoff | 2018-04-20 | 1 | -7/+42 |
| | | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly. | ||||
* | Track where event stream processing have gotten up to | Erik Johnston | 2018-04-11 | 1 | -0/+3 |
| | |||||
* | Use static JSONEncoders | Richard van der Hoff | 2018-03-29 | 1 | -15/+8 |
| | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it. | ||||
* | Comment | Erik Johnston | 2018-03-27 | 1 | -4/+5 |
| | |||||
* | Fix indent | Erik Johnston | 2018-03-27 | 1 | -1/+1 |
| | |||||
* | Comment | Erik Johnston | 2018-03-27 | 1 | -0/+7 |
| | |||||
* | Add counter metrics for calculating state delta | Erik Johnston | 2018-03-27 | 1 | -1/+30 |
| | | | | | | This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event. | ||||
* | Merge branch 'master' of github.com:matrix-org/synapse into develop | Erik Johnston | 2018-03-19 | 1 | -1/+0 |
|\ | |||||
| * | Remove wrong comment | Erik Johnston | 2018-03-16 | 1 | -1/+0 |
| | | |||||
| * | Replace ujson with simplejson | Erik Johnston | 2018-03-15 | 1 | -2/+2 |
| | | |||||
* | | Replace some ujson with simplejson to make it work | Erik Johnston | 2018-03-16 | 1 | -1/+1 |
| | | |||||
* | | Add Measure block for persist_events | Richard van der Hoff | 2018-03-13 | 1 | -4/+5 |
| | | | | | | | | This seems like a useful thing to measure. | ||||
* | | Fix race in sync when joining room | Erik Johnston | 2018-03-07 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | | | | | | | The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token. | ||||
* | | Add some caches to help read marker API | Erik Johnston | 2018-03-01 | 1 | -2/+2 |
| | | |||||
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-02-26 | 1 | -359/+7 |
|\ \ | | | | | | | | | | erikj/handle_unpersisted_events_push | ||||
| * | | Split EventsWorkerStore into separate file | Erik Johnston | 2018-02-23 | 1 | -360/+5 |
| | | | |||||
| * | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | |||||
| * | | Remove redundant clock | Erik Johnston | 2018-02-23 | 1 | -3/+0 |
| | | | |||||
| * | | _event_persist_queue shouldn't be in worker store | Erik Johnston | 2018-02-23 | 1 | -4/+4 |
| | | | |||||
| * | | Split out get_events and co into a worker store | Erik Johnston | 2018-02-21 | 1 | -345/+350 |
| | | | |||||
* | | | Ensure all push actions are deleted from staging | Erik Johnston | 2018-02-20 | 1 | -1/+10 |
| | | | |||||
* | | | Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts | Erik Johnston | 2018-02-20 | 1 | -5/+6 |
|/ / | |||||
* | | (Really) fix tablescan of event_push_actions on purge | Richard van der Hoff | 2018-02-16 | 1 | -1/+0 |
| | | | | | | | | | | commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/ | ||||
* | | Remove context.push_actions | Erik Johnston | 2018-02-15 | 1 | -4/+3 |
| | | |||||
* | | Update event_push_actions table from staging table | Erik Johnston | 2018-02-15 | 1 | -1/+1 |
| | | |||||
* | | purge_history: fix sqlite syntax error | Richard van der Hoff | 2018-02-14 | 1 | -1/+4 |
| | | | | | | | | apparently sqlite insists on indexes being named | ||||
* | | purge_history: handle sqlite asshattery | Richard van der Hoff | 2018-02-14 | 1 | -19/+27 |
| | | | | | | | | | | apparently creating a temporary table commits the transaction. because that's a useful thing. | ||||
* | | purge_history: fix index use | Richard van der Hoff | 2018-02-14 | 1 | -0/+14 |
| | | | | | | | | | | event_push_actions doesn't have an index on event_id, so we need to specify room_id. | ||||
* | | Rework event purge to use a temporary table | Richard van der Hoff | 2018-02-14 | 1 | -35/+58 |
| | | | | | | | | | | ... which should speed things up by reducing the amount of data being shuffled across the connection | ||||
* | | Fix log message in purge_history | Richard van der Hoff | 2018-02-13 | 1 | -2/+1 |
| | | | | | | | | (we don't just remove remote events) | ||||
* | | Merge pull request #2864 from matrix-org/rav/persist_event_caching | Richard van der Hoff | 2018-02-13 | 1 | -56/+40 |
|\ \ | | | | | | | Use StateResolutionHandler to resolve state in persist_events | ||||
| * | | style nit | Richard van der Hoff | 2018-02-13 | 1 | -1/+1 |
| | | | |||||
| * | | Use StateResolutionHandler to resolve state in persist events | Richard van der Hoff | 2018-02-05 | 1 | -48/+24 |
| | | | | | | | | | | | | ... and thus benefit (hopefully) from its cache. | ||||
| * | | Flatten _get_new_state_after_events | Richard van der Hoff | 2018-02-05 | 1 | -44/+46 |
| | | | | | | | | | | | | rejig the if statements to simplify the logic and reduce indentation | ||||
| * | | Check that events being persisted have state_group | Richard van der Hoff | 2018-02-05 | 1 | -4/+9 |
| | | | |||||
| * | | Add event_map param to resolve_state_groups | Richard van der Hoff | 2018-02-05 | 1 | -0/+1 |
| | | | |||||
* | | | Merge pull request #2858 from matrix-org/rav/purge_updates | Richard van der Hoff | 2018-02-09 | 1 | -33/+61 |
|\ \ \ | | | | | | | | | delete_local_events for purge_room_history | ||||
| * | | | purge: move room_depth update to end | Richard van der Hoff | 2018-02-09 | 1 | -6/+12 |
| | | | | | | | | | | | | | | | | ... to avoid locking the table for too long | ||||
| * | | | delete_local_events for purge_history | Richard van der Hoff | 2018-02-09 | 1 | -7/+28 |
| | | | | | | | | | | | | | | | | Add a flag which makes the purger delete local events | ||||
| * | | | purge: Move cache invalidation to more appropriate place | Richard van der Hoff | 2018-02-09 | 1 | -4/+4 |
| | | | | | | | | | | | | | | | | it was a bit of a non-sequitur there | ||||
| * | | | bump purge logging to info | Richard van der Hoff | 2018-02-09 | 1 | -13/+14 |
| | | | | | | | | | | | | | | | | | | | | this thing takes ages and the only sign of any progress is the logs, so having some logs is useful. | ||||
| * | | | rename delete_old_state -> purge_history | Richard van der Hoff | 2018-02-09 | 1 | -7/+7 |
| | | | | | | | | | | | | | | | | (beacause it deletes more than state) | ||||
* | | | | Store state groups separately from events (#2784) | Erik Johnston | 2018-02-06 | 1 | -6/+4 |
| |/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints | ||||
* | | | Remove redundant return value from _calculate_state_delta | Richard van der Hoff | 2018-02-05 | 1 | -10/+23 |
| | | | | | | | | | | | | | | | we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing. | ||||
* | | | factor _get_new_state_after_events out of _calculate_state_delta | Richard van der Hoff | 2018-01-31 | 1 | -18/+39 |
| | | | | | | | | | | | | This reduces the scope of a bunch of variables | ||||
* | | | Improve exception handling in persist_event | Richard van der Hoff | 2018-01-29 | 1 | -3/+3 |
|/ / | | | | | | | | | | | | | | | | | | | | | 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs | ||||
* | | Merge pull request #2805 from matrix-org/rav/log_state_res | Richard van der Hoff | 2018-01-17 | 1 | -0/+6 |
|\ \ | | | | | | | Log room when doing state resolution | ||||
| * | | Log room when doing state resolution | Richard van der Hoff | 2018-01-17 | 1 | -0/+6 |
| |/ | | | | | | | Mostly because it helps figure out what is prompting the resolution | ||||
* | | Split resolve_events into two functions | Richard van der Hoff | 2018-01-17 | 1 | -2/+2 |
| | | | | | | | | ... so that the return type doesn't depend on the arg types | ||||
* | | Fix a logcontext leak in persist_events | Richard van der Hoff | 2018-01-16 | 1 | -1/+8 |
|/ | | | | | | | | | | | | | ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway) | ||||
* | Make __init__ consitstent across Store heirarchy | Richard van der Hoff | 2017-11-13 | 1 | -2/+2 |
| | | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores) | ||||
* | replace 'except:' with 'except Exception:' | Richard van der Hoff | 2017-10-23 | 1 | -1/+1 |
| | | | | what could possibly go wrong | ||||
* | Fix logcontext handling for persist_events | Richard van der Hoff | 2017-10-17 | 1 | -7/+17 |
| | | | | | | | | * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments | ||||
* | Invalidate cache | Erik Johnston | 2017-10-03 | 1 | -0/+3 |
| | |||||
* | Make into list | Erik Johnston | 2017-06-29 | 1 | -1/+1 |
| | |||||
* | Prefill forward extrems and event to state groups | Erik Johnston | 2017-06-29 | 1 | -0/+5 |
| | |||||
* | Merge pull request #2286 from matrix-org/erikj/split_out_user_dir | Erik Johnston | 2017-06-16 | 1 | -0/+18 |
|\ | | | | | Split out user directory to a separate process | ||||
| * | Initial worker impl | Erik Johnston | 2017-06-16 | 1 | -0/+18 |
| | | |||||
* | | Merge pull request #2281 from matrix-org/erikj/phone_home_stats | Erik Johnston | 2017-06-15 | 1 | -58/+43 |
|\ \ | |/ |/| | Fix phone home stats | ||||
| * | Add some more stats | Erik Johnston | 2017-06-15 | 1 | -0/+15 |
| | | |||||
| * | Fix phone home stats | Erik Johnston | 2017-06-14 | 1 | -59/+29 |
| | | |||||
* | | Add cache for is_host_joined | Erik Johnston | 2017-06-13 | 1 | -0/+5 |
| | | |||||
* | | Merge pull request #2259 from matrix-org/erikj/fix_state_woes | Erik Johnston | 2017-06-07 | 1 | -1/+1 |
|\ \ | | | | | | | Fix bug where state_group tables got corrupted | ||||
| * | | Fix bug where state_group tables got corrupted | Erik Johnston | 2017-06-07 | 1 | -1/+1 |
| |/ | | | | | | | | | | | | | | | This is due to the fact that we prefilled caches using txn.call_after, which always gets called including on error. We fix this by making txn.call_after only fire when a transaction completes successfully, which is what we want most of the time anyway. | ||||
* | | Fix typo | Erik Johnston | 2017-05-31 | 1 | -1/+1 |
| | | |||||
* | | Add stream change cache | Erik Johnston | 2017-05-31 | 1 | -0/+4 |
| | | |||||
* | | Add clobbered event_id | Erik Johnston | 2017-05-30 | 1 | -0/+1 |
| | | |||||
* | | Add current_state_delta_stream table | Erik Johnston | 2017-05-30 | 1 | -8/+23 |
| | | |||||
* | | Only store event_auth for state events | Erik Johnston | 2017-05-24 | 1 | -0/+1 |
|/ | |||||
* | Merge pull request #2224 from matrix-org/erikj/prefill_state | Erik Johnston | 2017-05-16 | 1 | -5/+10 |
|\ | | | | | Prefill state caches | ||||
| * | Remove spurious merge artifacts | Erik Johnston | 2017-05-16 | 1 | -13/+0 |
| | | |||||
| * | Comments | Erik Johnston | 2017-05-15 | 1 | -3/+3 |
| | | |||||
| * | Add more granular event send metrics | Erik Johnston | 2017-05-15 | 1 | -0/+12 |
| | | |||||
| * | Prefill state caches | Erik Johnston | 2017-05-15 | 1 | -2/+8 |
| | | |||||
* | | Don't create event_search index on sqlite | Richard van der Hoff | 2017-05-11 | 1 | -0/+1 |
| | | | | | | | | ... because the table is virtual | ||||
* | | Add more logging for purging | Richard van der Hoff | 2017-05-11 | 1 | -7/+14 |
| | | | | | | | | Log the number of events we will be deleting at info. | ||||
* | | Add an index to event_search | Richard van der Hoff | 2017-05-11 | 1 | -0/+11 |
|/ | | | | - to make the purge API quicker | ||||
* | Tidy purge code and add some comments | Richard van der Hoff | 2017-05-11 | 1 | -12/+19 |
| | | | | Try to make this clearer with more comments and some variable renames | ||||
* | Don't de-delta state groups we're about to delete | Richard van der Hoff | 2017-05-10 | 1 | -11/+9 |
| | |||||
* | add some logging to purge_history | Richard van der Hoff | 2017-05-10 | 1 | -4/+21 |
| | |||||
* | Expand docstring a bit | Erik Johnston | 2017-05-08 | 1 | -1/+7 |
| | |||||
* | Don't update event cache hit ratio from get_joined_users | Erik Johnston | 2017-05-08 | 1 | -2/+11 |
| | | | | | Otherwise the hit ration of plain get_events gets completely skewed by calls to get_joined_users* functions. | ||||
* | Revert "Prefill state caches" | Erik Johnston | 2017-05-04 | 1 | -11/+5 |
| | |||||
* | Comments | Erik Johnston | 2017-05-03 | 1 | -3/+3 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state | Erik Johnston | 2017-05-03 | 1 | -0/+18 |
|\ | |||||
| * | Add more granular event send metrics | Erik Johnston | 2017-05-02 | 1 | -0/+16 |
| | | |||||
* | | Prefill state caches | Erik Johnston | 2017-05-02 | 1 | -2/+6 |
|/ | |||||
* | Correct logic in is_event_after | Luke Barnard | 2017-04-13 | 1 | -1/+1 |
| | |||||
* | Simplify is_event_after logic | Luke Barnard | 2017-04-12 | 1 | -10/+3 |
| | |||||
* | travis flake8.. | Luke Barnard | 2017-04-11 | 1 | -0/+1 |
| | |||||
* | Refactor event ordering check to events store | Luke Barnard | 2017-04-11 | 1 | -0/+28 |
| | |||||
* | Add new storage functions for new replication | Erik Johnston | 2017-03-30 | 1 | -0/+88 |
| | | | | | The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one. | ||||
* | Typo | Erik Johnston | 2017-03-27 | 1 | -1/+1 |
| | |||||
* | Short circuit if all new events have same state group | Erik Johnston | 2017-03-27 | 1 | -36/+51 |
| | |||||
* | Use iter(items|values) | Erik Johnston | 2017-03-24 | 1 | -18/+14 |
| | |||||
* | User Cursor.__iter__ instead of fetchall | Erik Johnston | 2017-03-23 | 1 | -1/+1 |
| | | | | This prevents unnecessary construction of lists | ||||
* | Merge pull request #2033 from matrix-org/erikj/repl_speed | Erik Johnston | 2017-03-21 | 1 | -17/+14 |
|\ | | | | | Don't send the full event json over replication | ||||
| * | Don't send the full event json over replication | Erik Johnston | 2017-03-17 | 1 | -17/+14 |
| | | |||||
* | | Avoid resetting state on rejected events | Richard van der Hoff | 2017-03-17 | 1 | -4/+4 |
| | | | | | | | | | | | | | | When we get a rejected event, give it the same state_group as its prev_event, rather than no state_group at all. This should fix https://github.com/matrix-org/synapse/issues/1935. | ||||
* | | Refactoring and cleanups | Richard van der Hoff | 2017-03-17 | 1 | -60/+213 |
|/ | | | | | | | | | | | | A few non-functional changes: * A bunch of docstrings to document types * Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more readable. * Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid mind-bending conditional. * Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid mind-bending conditional. | ||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2017-03-14 | 1 | -3/+28 |
|\ | | | | | | | erikj/public_list_speed | ||||
| * | Fix current_state_events table to not lie | Erik Johnston | 2017-03-14 | 1 | -3/+28 |
| | | | | | | | | | | If we try and persist two state events that have the same ancestor we calculate the wrong current state when persisting those events. | ||||
* | | Get current state by using current_state_events table | Erik Johnston | 2017-03-10 | 1 | -10/+8 |
|/ | |||||
* | Remove needless check | Erik Johnston | 2017-02-27 | 1 | -2/+0 |
| | |||||
* | It should be all | Erik Johnston | 2017-02-27 | 1 | -1/+1 |
| | |||||
* | Don't fetch current state in common case | Erik Johnston | 2017-02-27 | 1 | -0/+17 |
| | | | | | | Currently we fetch the list of current state events whenever we send something in a room. This is overkill for the common case of persisting a simple chain of non-state events, so lets handle that case specially. | ||||
* | Ignore new rejected events when working out forward extremeties. | Erik Johnston | 2017-02-08 | 1 | -9/+6 |
| | |||||
* | Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events | Erik Johnston | 2017-02-01 | 1 | -0/+2 |
|\ | | | | | Add a small cache get_all_new_events | ||||
| * | Add a small cache get_all_new_events | Erik Johnston | 2017-02-01 | 1 | -0/+2 |
| | | |||||
* | | Only invalidate membership caches based on the cache stream | Erik Johnston | 2017-01-31 | 1 | -20/+0 |
|/ | | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive. | ||||
* | Stream cache invalidations for room membership storage functions | Erik Johnston | 2017-01-30 | 1 | -2/+6 |
| | |||||
* | Typo | Erik Johnston | 2017-01-23 | 1 | -1/+1 |
| | |||||
* | Refactor to calculate state delta outside transaction | Erik Johnston | 2017-01-23 | 1 | -87/+118 |
| | |||||
* | Comments | Erik Johnston | 2017-01-20 | 1 | -8/+19 |
| | |||||
* | Insert delta of current_state_events to be more efficient | Erik Johnston | 2017-01-20 | 1 | -26/+52 |
| | |||||
* | Spelling | Erik Johnston | 2017-01-20 | 1 | -1/+1 |
| | |||||
* | Update all call sites after rename | Erik Johnston | 2017-01-20 | 1 | -1/+1 |
| | |||||
* | Comments | Erik Johnston | 2017-01-20 | 1 | -0/+11 |
| | |||||
* | Calculate the forward extremeties once | Erik Johnston | 2017-01-20 | 1 | -56/+86 |
| | |||||
* | Derive current_state_events from state groups | Erik Johnston | 2017-01-20 | 1 | -70/+118 |
| | |||||
* | Lower loading events log to DEBUG | Erik Johnston | 2017-01-17 | 1 | -2/+2 |
| | |||||
* | More logging for the linearizer and for get_events | Mark Haines | 2017-01-05 | 1 | -0/+2 |
| | |||||
* | New Flake8 fixes | Erik Johnston | 2016-11-15 | 1 | -0/+1 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth | Erik Johnston | 2016-10-17 | 1 | -15/+29 |
|\ | |||||
| * | Fix background reindex of origin_server_ts | Erik Johnston | 2016-09-27 | 1 | -15/+29 |
| | | | | | | | | | | | | | | | | The storage function `_get_events_txn` was removed everywhere except from this background reindex. The function was removed due to it being (almost) completely unused while also being large and complex. Therefore, instead of resurrecting `_get_events_txn` we manually reimplement the bits that are needed directly. | ||||
* | | Drop some unused indices | Erik Johnston | 2016-10-17 | 1 | -1/+0 |
|/ | |||||
* | Index contains_url for file search queries | Erik Johnston | 2016-09-12 | 1 | -0/+8 |
| | |||||
* | Take value in a better way | Erik Johnston | 2016-09-05 | 1 | -1/+1 |
| | |||||
* | Correctly delete old state groups in purge history API | Erik Johnston | 2016-09-05 | 1 | -20/+79 |
| | |||||
* | Limit the length of state chains | Erik Johnston | 2016-09-02 | 1 | -21/+28 |
| | |||||
* | Generate state group ids in state layer | Erik Johnston | 2016-08-31 | 1 | -9/+1 |
| | |||||
* | Make _state_groups_id_gen a normal IdGenerator | Erik Johnston | 2016-08-30 | 1 | -44/+39 |
| | |||||
* | Use state handler instead of get_users_in_room/get_joined_hosts | Erik Johnston | 2016-08-26 | 1 | -1/+0 |
| | |||||
* | Preserve some logcontexts | Erik Johnston | 2016-08-24 | 1 | -6/+10 |
| | |||||
* | Move defer.returnValue out of Measure | Erik Johnston | 2016-08-19 | 1 | -1/+1 |
| | |||||
* | Measure _get_event_from_row | Erik Johnston | 2016-08-19 | 1 | -41/+43 |
| | |||||
* | Add missing database corruption recovery case | Benjamin Saunders | 2016-08-14 | 1 | -1/+2 |
| | | | | Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com> | ||||
* | Raise 404 when couldn't find event | Erik Johnston | 2016-08-05 | 1 | -1/+1 |
| | |||||
* | Delete more tables | Erik Johnston | 2016-08-05 | 1 | -0/+5 |
| | |||||
* | Tweak integrity error recovery to work as intended | Benjamin Saunders | 2016-08-04 | 1 | -0/+3 |
| | |||||
* | Fix typo | Erik Johnston | 2016-08-04 | 1 | -3/+4 |
| | |||||
* | Retry event persistence on IntegrityError | Erik Johnston | 2016-08-04 | 1 | -4/+65 |
| | | | | | | | | | | Due to a bug in the porting script some backfilled events were not correctly persisted, causing irrecoverable IntegrityErrors on future attempts to persist those events. This commit adds a retry mechanism invoked upon IntegrityError, where when retried the tables are purged for all references to the events being persisted. | ||||
* | Remove other bit of deduplication | Erik Johnston | 2016-08-03 | 1 | -19/+0 |
| | |||||
* | Ensure we only persist an event once at a time | Erik Johnston | 2016-08-03 | 1 | -1/+18 |
| | |||||
* | Fix how push_actions are redacted. | Mark Haines | 2016-07-26 | 1 | -8/+10 |
| | |||||
* | Don't add rejected events if we've seen them befrore. Add some comments to ↵ | Mark Haines | 2016-07-26 | 1 | -5/+48 |
| | | | | explain what the code is doing mechanically | ||||
* | Only update the events and event_json tables for rejected events | Mark Haines | 2016-07-25 | 1 | -51/+62 |
| | |||||
* | Don't add the events to forward extremities if the event is rejected | Mark Haines | 2016-07-25 | 1 | -3/+6 |
| | |||||
* | Don't add rejections to the state_group, persist all rejections | Mark Haines | 2016-07-25 | 1 | -4/+5 |
| | |||||
* | Merge pull request #922 from matrix-org/erikj/file_api2 | Erik Johnston | 2016-07-20 | 1 | -0/+82 |
|\ | | | | | Feature: Add filter to /messages. Add 'contains_url' to filter. | ||||
| * | Add sender and contains_url field to events table | Erik Johnston | 2016-07-14 | 1 | -0/+82 |
| | | |||||
* | | Fix /purge_history bug | Erik Johnston | 2016-07-15 | 1 | -19/+15 |
| | | | | | | | | This was caused by trying to insert duplicate backward extremeties | ||||
* | | event_backwards_extremeties may not be empty | Erik Johnston | 2016-07-15 | 1 | -1/+11 |
|/ | |||||
* | Comment | Erik Johnston | 2016-07-07 | 1 | -0/+3 |
| | |||||
* | Return 400 rather than 500 | Erik Johnston | 2016-07-07 | 1 | -1/+4 |
| | |||||
* | Add rest servlet. Fix SQL. | Erik Johnston | 2016-07-06 | 1 | -5/+4 |
| | |||||
* | Add purge_history API | Erik Johnston | 2016-07-05 | 1 | -0/+6 |
| | |||||
* | Add storage function to purge history for a room | Erik Johnston | 2016-07-04 | 1 | -0/+140 |
| | |||||
* | Use similar naming we use in email notifs for push | David Baker | 2016-06-24 | 1 | -7/+0 |
| | | | | Fixes https://github.com/vector-im/vector-web/issues/1654 | ||||
* | Merge pull request #841 from matrix-org/erikj/event_counter | Erik Johnston | 2016-06-06 | 1 | -0/+9 |
|\ | | | | | Add metric counter for number of persisted events | ||||
| * | Add metric counter for number of persisted events | Erik Johnston | 2016-06-06 | 1 | -0/+9 |
| | | |||||
* | | Add events to cache when we persist them | Erik Johnston | 2016-06-06 | 1 | -0/+41 |
|/ | |||||
* | Don't rely on options when inserting event into cache | Erik Johnston | 2016-06-03 | 1 | -40/+43 |
| | |||||
* | Change the way we cache events | Erik Johnston | 2016-06-03 | 1 | -39/+41 |
| | |||||
* | Remove event fetching from DB threads | Erik Johnston | 2016-06-03 | 1 | -138/+0 |
| | |||||
* | Use state to calculate get_users_in_room | Erik Johnston | 2016-06-01 | 1 | -3/+0 |
| | |||||
* | Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs | David Baker | 2016-05-10 | 1 | -11/+149 |
|\ | |||||
| * | Remove unused import | Erik Johnston | 2016-05-06 | 1 | -1/+1 |
| | | |||||
| * | Pull loop one level up | Erik Johnston | 2016-05-06 | 1 | -36/+41 |
| | | |||||
| * | Queue events for persistence | Erik Johnston | 2016-05-06 | 1 | -11/+144 |
| | | |||||
* | | Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs | David Baker | 2016-04-29 | 1 | -2/+8 |
|\| | |||||
| * | Check that somethign has happend before running the selects | Mark Haines | 2016-04-27 | 1 | -2/+8 |
| | | |||||
* | | First bits of emailpusher | David Baker | 2016-04-19 | 1 | -0/+2 |
|/ | | | | Mostly logic of when to send an email |