summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptionsRichard van der Hoff2018-04-271-7/+17
|\ | | | | Improve exception handling for background processes
| * Improve exception handling for background processesRichard van der Hoff2018-04-271-7/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* | Add missing consumeErrorsRichard van der Hoff2018-04-271-1/+1
|/ | | | | In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.
* Fix media admin APIsErik Johnston2018-04-261-2/+2
|
* Also fix reindexing of searchErik Johnston2018-04-251-1/+1
|
* Fix quarantine media admin APIErik Johnston2018-04-251-1/+1
|
* Merge pull request #3110 from NotAFile/py3-six-queueRichard van der Hoff2018-04-201-2/+4
|\ | | | | Replace Queue with six.moves.queue
| * Replace Queue with six.moves.queueAdrian Tschira2018-04-161-2/+4
| | | | | | | | | | | | and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3117 from matrix-org/rav/refactor_have_eventsRichard van der Hoff2018-04-201-7/+42
|\ \ | | | | | | Refactor store.have_events
| * | Refactor store.have_eventsRichard van der Hoff2018-04-201-7/+42
| |/ | | | | | | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
* / Avoid creating events with huge numbers of prev_eventsRichard van der Hoff2018-04-161-16/+41
|/ | | | | | In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.
* Format docstringErik Johnston2018-04-121-2/+3
|
* Doc we raise on unknown eventErik Johnston2018-04-121-2/+3
|
* Track last processed event received_tsErik Johnston2018-04-111-0/+18
|
* Track where event stream processing have gotten up toErik Johnston2018-04-111-0/+3
|
* Merge pull request #2760 from Valodim/pypyRichard van der Hoff2018-04-112-1/+6
|\ | | | | Synapse on PyPy
| * Fix pep8 error on psycopg2cffi hackRichard van der Hoff2018-04-101-6/+5
| |
| * Use psycopg2cffi module instead of psycopg2 if running on pypyVincent Breitmoser2018-04-102-2/+8
| | | | | | | | | | | | | | The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.
* | fix typoNeil Johnson2018-04-101-2/+2
| |
* | remove errant print v0.27.3-rc2Neil Johnson2018-04-091-1/+1
| |
* | Fix msec to sec, againNeil Johnson2018-04-091-3/+3
| |
* | Fix msec to secNeil Johnson2018-04-091-1/+1
| |
* | Fix psycopg2 interpolationNeil Johnson2018-04-091-5/+5
|/
* Don't use redundant inlineCallbacksLuke Barnard2018-04-061-4/+1
|
* join_rule -> join_policyLuke Barnard2018-04-061-1/+1
|
* is_joinable -> join_ruleLuke Barnard2018-04-061-4/+1
|
* pep8David Baker2018-04-061-1/+2
|
* Implement group join APIDavid Baker2018-04-061-3/+9
|
* Merge pull request #3045 from matrix-org/dbkr/group_joinableLuke Barnard2018-04-053-0/+42
|\ | | | | Add joinability for groups
| * NON NULL -> NOT NULLLuke Barnard2018-04-051-1/+1
| |
| * Use DEFAULT join_policy of "invite" in dbLuke Barnard2018-04-051-1/+1
| |
| * Document set_group_join_policyLuke Barnard2018-04-051-0/+6
| |
| * Use join_policy API instead of joinableLuke Barnard2018-04-032-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".
| * OK, smallint it is thenDavid Baker2018-03-281-1/+1
| |
| * Grr. Copy the definition from is_adminDavid Baker2018-03-281-1/+1
| |
| * Make column definition that works on both dbsDavid Baker2018-03-281-1/+1
| |
| * Add schema delta fileDavid Baker2018-03-281-0/+16
| |
| * Add joinability for groupsDavid Baker2018-03-282-1/+15
| | | | | | | | | | Adds API to set the 'joinable' flag, and corresponding flag in the table.
* | Merge pull request #3041 from matrix-org/r30_statsNeil Johnson2018-04-054-7/+111
|\ \ | | | | | | R30 stats
| * | Review commentsNeil Johnson2018-04-051-4/+5
| | | | | | | | | | | | | | | Use iteritems over item to loop over dict formatting
| * | pep8Neil Johnson2018-03-291-2/+4
| | |
| * | Remove need for sqlite specific queryNeil Johnson2018-03-291-30/+57
| | |
| * | fix pep8 errorsNeil Johnson2018-03-281-3/+0
| | |
| * | remove twisted deferral cruftNeil Johnson2018-03-281-6/+3
| | |
| * | bump schema versionNeil Johnson2018-03-281-1/+1
| | |
| * | Support multi client R30 for psqlNeil Johnson2018-03-281-8/+28
| | |
| * | Add user_ips last seen indexNeil Johnson2018-03-282-0/+24
| | |
| * | No need to cast in count_daily_usersNeil Johnson2018-03-281-2/+2
| | |
| * | query and call for r30 statsNeil Johnson2018-03-281-0/+36
| | |
| * | count_daily_users failed if db was sqlite due to type failure - presumably ↵Neil Johnson2018-03-281-1/+1
| |/ | | | | | | this prevcented all sqlite homeservers reporting home
* | Merge pull request #3060 from matrix-org/rav/kill_event_contentRichard van der Hoff2018-04-053-7/+12
|\ \ | | | | | | Remove uses of events.content
| * | Remove uses of events.contentRichard van der Hoff2018-03-293-7/+12
| | |
* | | Merge pull request #3049 from matrix-org/rav/use_staticjsonRichard van der Hoff2018-04-031-15/+8
|\ \ \ | | | | | | | | Use static JSONEncoders
| * | | Use static JSONEncodersRichard van der Hoff2018-03-291-15/+8
| |/ / | | | | | | | | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.
* / / Use simplejson throughoutRichard van der Hoff2018-03-291-1/+2
|/ / | | | | | | Let's use simplejson rather than json, for consistency.
* | Fix search_user_dir multiple sqlite versions do different thingsNeil Johnson2018-03-281-2/+2
| |
* | Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_idErik Johnston2018-03-281-3/+1
|\ \ | | | | | | Linearize calls to _generate_user_id
| * | Don't use _cursor_to_dict in find_next_generated_user_id_localpartErik Johnston2018-03-261-3/+1
| |/
* | Merge pull request #3030 from matrix-org/erikj/no_ujsonErik Johnston2018-03-281-1/+1
|\ \ | | | | | | Remove last usage of ujson
| * | Remove last usage of ujsonErik Johnston2018-03-261-1/+1
| |/
* | CommentErik Johnston2018-03-271-4/+5
| |
* | Fix indentErik Johnston2018-03-271-1/+1
| |
* | CommentErik Johnston2018-03-271-0/+7
| |
* | Add counter metrics for calculating state deltaErik Johnston2018-03-271-1/+30
|/ | | | | | This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-1917-26/+25
|\
| * Remove wrong commentErik Johnston2018-03-161-1/+0
| |
| * Replace ujson with simplejsonErik Johnston2018-03-1517-26/+26
| |
* | Replace some ujson with simplejson to make it workErik Johnston2018-03-162-2/+2
| |
* | Merge pull request #2988 from matrix-org/erikj/split_profile_storeErik Johnston2018-03-141-24/+26
|\ \ | | | | | | Split up ProfileStore
| * | Split up ProfileStoreErik Johnston2018-03-131-24/+26
| | |
* | | Merge pull request #2993 from matrix-org/erikj/is_blockedErik Johnston2018-03-141-13/+17
|\ \ \ | | | | | | | | Add is_blocked to worker store
| * | | Add is_blocked to worker storeErik Johnston2018-03-131-13/+17
| |/ /
* / / fix bug #2926 (loading all state for a given type from the DB if the ↵Matthew Hodgson2018-03-131-7/+27
|/ / | | | | | | | | state_key is None) (#2990) Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
* | Add Measure block for persist_eventsRichard van der Hoff2018-03-131-4/+5
| | | | | | | | This seems like a useful thing to measure.
* | Fix race in sync when joining roomErik Johnston2018-03-072-2/+27
| | | | | | | | | | | | | | | | | | | | | | The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.
* | Merge pull request #2946 from matrix-org/rav/timestamp_to_purgeRichard van der Hoff2018-03-061-0/+27
|\ \ | | | | | | Implement purge_history by timestamp
| * | Provide a means to pass a timestamp to purge_historyRichard van der Hoff2018-03-051-0/+27
| | |
* | | Merge pull request #2948 from matrix-org/erikj/kill_as_syncErik Johnston2018-03-062-153/+5
|\ \ \ | | | | | | | | Remove ability for AS users to call /events and /sync
| * | | Remove ability for AS users to call /events and /syncErik Johnston2018-03-052-153/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144
* | | | Merge pull request #2947 from matrix-org/erikj/split_directory_storeErik Johnston2018-03-051-23/+27
|\ \ \ \ | |_|/ / |/| | | Split Directory store
| * | | Fix cache invalidation on deletionErik Johnston2018-03-051-1/+4
| | | |
| * | | Split Directory storeErik Johnston2018-03-051-22/+23
| |/ /
* | | Merge pull request #2943 from ↵Richard van der Hoff2018-03-051-12/+71
|\ \ \ | |/ / |/| | | | | | | | matrix-org/rav/fix_find_first_stream_ordering_after_ts Test and fix find_first_stream_ordering_after_ts
| * | Test and fix find_first_stream_ordering_after_tsRichard van der Hoff2018-03-051-15/+53
| | | | | | | | | | | | It seemed to suffer from a bunch of off-by-one errors.
| * | Add find_first_stream_ordering_after_tsRichard van der Hoff2018-03-051-0/+21
| | | | | | | | | | | | Expose this as a public function which can be called outside a txn
* | | Merge pull request #2934 from matrix-org/erikj/cache_fixErik Johnston2018-03-052-13/+15
|\ \ \ | | | | | | | | Fix bug with delayed cache invalidation stream
| * | | Fix bug with delayed cache invalidation streamErik Johnston2018-03-022-13/+15
| | | | | | | | | | | | | | | | | | | | | | | | We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.
* | | | Split registration storeErik Johnston2018-03-021-57/+61
|/ / /
* | | Merge pull request #2925 from matrix-org/erikj/split_sig_fedErik Johnston2018-03-012-133/+143
|\ \ \ | |/ / |/| | Split out SignatureStore and EventFederationStore
| * | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fedErik Johnston2018-03-014-371/+406
| |\ \
| * | | Stub out broken function only used for cacheErik Johnston2018-03-011-1/+3
| | | |
| * | | Split out SignatureStore and EventFederationStoreErik Johnston2018-03-012-132/+140
| | | |
* | | | Merge pull request #2927 from matrix-org/erikj/read_marker_cachesErik Johnston2018-03-012-2/+39
|\ \ \ \ | |_|/ / |/| | | Improve caching for read_marker API
| * | | Fewer lies are betterErik Johnston2018-03-011-2/+2
| | | |
| * | | Improve caching for read_marker APIErik Johnston2018-03-011-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.
| * | | Add some caches to help read marker APIErik Johnston2018-03-012-2/+4
| |/ /
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-03-013-197/+206
|\ \ \ | | | | | | | | | | | | erikj/split_stream_store
| * \ \ Merge pull request #2923 from matrix-org/erikj/stream_ago_workerErik Johnston2018-03-012-79/+85
| |\ \ \ | | | | | | | | | | Calculate stream_ordering_month_ago correctly on workers
| | * | | Default stream_ordering_*_ago to NoneErik Johnston2018-03-011-2/+2
| | | | |
| | * | | Fix comment typoErik Johnston2018-03-011-1/+1
| | | | |
| | * | | Calculate stream_ordering_month_ago correctly on workersErik Johnston2018-03-012-79/+85
| | |/ /
| * / / Split up RoomStoreErik Johnston2018-03-011-118/+121
| |/ /
* | | Remove unused variablesErik Johnston2018-03-011-8/+1
| | |
* | | Document abstract class and method betterErik Johnston2018-03-011-8/+13
| | |
* | | Split out stream storeErik Johnston2018-03-012-165/+193
|/ /
* | Move storage functions for push calculationsErik Johnston2018-02-274-87/+99
| | | | | | | | This will allow push actions for an event to be calculated on workers.
* | Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidationErik Johnston2018-02-271-14/+14
|\ \ | | | | | | Fix missing invalidations for receipt storage
| * | Fix missing invalidations for receipt storageErik Johnston2018-02-211-14/+14
| | |
* | | Merge pull request #2903 from matrix-org/erikj/split_roommember_storeErik Johnston2018-02-271-179/+182
|\ \ \ | | | | | | | | Split out RoomMemberStore
| * \ \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-238-501/+607
| |\ \ \ | | | | | | | | | | | | | | | erikj/split_roommember_store
| * | | | Update copyrightErik Johnston2018-02-232-0/+2
| | | | |
| * | | | Split out RoomMemberStoreErik Johnston2018-02-211-179/+181
| | | | |
* | | | | Merge pull request #2901 from matrix-org/erikj/split_as_storesErik Johnston2018-02-271-9/+22
|\ \ \ \ \ | | | | | | | | | | | | Split AS stores
| * | | | | Add commentErik Johnston2018-02-271-0/+4
| | | | | |
| * | | | | Update copyrightErik Johnston2018-02-231-0/+1
| | | | | |
| * | | | | Split AS storesErik Johnston2018-02-211-9/+17
| |/ / / /
* | | | | Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actionsErik Johnston2018-02-261-17/+36
|\ \ \ \ \ | | | | | | | | | | | | Batch inserts into event_push_actions_staging
| * | | | | Batch inserts into event_push_actions_stagingErik Johnston2018-02-201-17/+36
| | | | | |
* | | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-269-560/+700
|\ \ \ \ \ \ | | |_|/ / / | |/| | | | | | | | | | erikj/handle_unpersisted_events_push
| * | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-237-424/+532
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | erikj/split_event_push_actions
| | * \ \ \ \ Merge pull request #2902 from matrix-org/erikj/split_events_storeErik Johnston2018-02-232-359/+402
| | |\ \ \ \ \ | | | | | | | | | | | | | | | | Split out get_events and co into a worker store
| | | * | | | | Split EventsWorkerStore into separate fileErik Johnston2018-02-232-360/+400
| | | | | | | |
| | | * | | | | Update copyrightErik Johnston2018-02-231-0/+1
| | | | | | | |
| | | * | | | | Remove redundant clockErik Johnston2018-02-231-3/+0
| | | | | | | |
| | | * | | | | _event_persist_queue shouldn't be in worker storeErik Johnston2018-02-231-4/+4
| | | | |/ / / | | | |/| | |
| | | * | | | Split out get_events and co into a worker storeErik Johnston2018-02-211-345/+350
| | | | |/ / | | | |/| |
| | * | | | Merge pull request #2899 from matrix-org/erikj/split_pushersErik Johnston2018-02-231-4/+7
| | |\ \ \ \ | | | | | | | | | | | | | | Split PusherStore
| | | * | | | Update copyrightErik Johnston2018-02-231-0/+1
| | | | | | |
| | | * | | | Split PusherStoreErik Johnston2018-02-211-4/+6
| | | |/ / /
| | * | | | Merge pull request #2898 from matrix-org/erikj/split_push_rules_storeErik Johnston2018-02-232-28/+57
| | |\ \ \ \ | | | | | | | | | | | | | | Split PushRulesStore
| | | * | | | Update copyrightErik Johnston2018-02-232-0/+2
| | | | | | |
| | | * | | | Split PushRulesStoreErik Johnston2018-02-212-28/+55
| | | |/ / /
| | * | | | Update copyrightErik Johnston2018-02-232-0/+2
| | | | | |
| | * | | | Use absolute importsErik Johnston2018-02-232-3/+3
| | | | | |
| | * | | | Split AccountDataStore and TagStoreErik Johnston2018-02-213-33/+65
| | |/ / /
| * | | | Update copyrightErik Johnston2018-02-232-0/+2
| | | | |
| * | | | Split out EventPushActionWorkerStoreErik Johnston2018-02-211-71/+73
| |/ / /
| * | | Raise exception in abstract methodErik Johnston2018-02-201-1/+1
| | | |
| * | | Fix commentErik Johnston2018-02-201-1/+1
| | | |
| * | | Use abstract base class to access stream IDsErik Johnston2018-02-201-15/+27
| | | |
| * | | Split ReceiptsStoreErik Johnston2018-02-202-48/+64
| |/ /
* | | Actually use new paramErik Johnston2018-02-211-1/+3
| | |
* | | Ensure all push actions are deleted from stagingErik Johnston2018-02-202-2/+19
| | |
* | | Refactor _set_push_actions_for_event_and_users_txn to use events_and_contextsErik Johnston2018-02-202-33/+41
|/ /
* | (Really) fix tablescan of event_push_actions on purgeRichard van der Hoff2018-02-161-1/+0
| | | | | | | | | | commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/
* | Fix typo of double is_highlightErik Johnston2018-02-161-1/+1
| |
* | CommentsErik Johnston2018-02-162-1/+6
| |
* | Ensure that we delete staging push actions on errorsErik Johnston2018-02-151-0/+16
| |
* | Remove context.push_actionsErik Johnston2018-02-151-4/+3
| |
* | Update event_push_actions table from staging tableErik Johnston2018-02-152-22/+39
| |
* | Store push actions in staging areaErik Johnston2018-02-152-0/+51
| |
* | Don't serialize current state over replicationErik Johnston2018-02-151-0/+14
| |
* | Fix state group storage bug in workersErik Johnston2018-02-151-41/+41
| | | | | | | | | | We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.
* | Merge pull request #2867 from matrix-org/rav/rework_purgeRichard van der Hoff2018-02-151-35/+83
|\ \ | | | | | | purge_history cleanups
| * | purge_history: fix sqlite syntax errorRichard van der Hoff2018-02-141-1/+4
| | | | | | | | | | | | apparently sqlite insists on indexes being named
| * | purge_history: handle sqlite asshatteryRichard van der Hoff2018-02-141-19/+27
| | | | | | | | | | | | | | | apparently creating a temporary table commits the transaction. because that's a useful thing.
| * | purge_history: fix index useRichard van der Hoff2018-02-141-0/+14
| | | | | | | | | | | | | | | event_push_actions doesn't have an index on event_id, so we need to specify room_id.
| * | Rework event purge to use a temporary tableRichard van der Hoff2018-02-141-35/+58
| | | | | | | | | | | | | | | ... which should speed things up by reducing the amount of data being shuffled across the connection
* | | Merge pull request #2769 from matrix-org/matthew/hit_the_ginRichard van der Hoff2018-02-145-20/+110
|\ \ \ | | | | | | | | switch back from GIST to GIN indexes
| * | | remove overzealous exception handlingRichard van der Hoff2018-02-141-18/+10
| | | |
| * | | Merge branch 'matthew/gin_work_mem' into matthew/hit_the_ginRichard van der Hoff2018-02-1313-312/+605
| |\ \ \
| | * \ \ Merge branch 'develop' into matthew/gin_work_memRichard van der Hoff2018-02-138-223/+390
| | |\ \ \
| | * | | | Factor out common code for search insertRichard van der Hoff2018-02-041-33/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | we can reuse the same code as is used for event insert, for doing the background index population.
| | * | | | Clean up work_mem handlingRichard van der Hoff2018-02-031-11/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add some comments and improve exception handling when twiddling work_mem for the search update
| | * | | | Move store_event_search_txn to SearchStoreRichard van der Hoff2018-02-032-37/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ... as a precursor to making event storing and doing the bg update share some code.
| | * | | | Merge branch 'develop' into matthew/gin_work_memRichard van der Hoff2018-02-037-39/+100
| | |\ \ \ \
| | * | | | | oopshera2018-01-092-2/+2
| | | | | | |
| | * | | | | oops, tweak work_mem when actually storingMatthew Hodgson2018-01-091-0/+2
| | | | | | |
| | * | | | | avoid 80s GIN inserts by tweaking work_memMatthew Hodgson2018-01-091-0/+4
| | | |_|_|/ | | |/| | | | | | | | | | | | | | | see https://github.com/matrix-org/synapse/issues/2753 for details
| * | | | | move search reindex to schema 47Richard van der Hoff2018-02-132-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.
| * | | | | GIN reindex: Fix syntax errors, improve exception handlingRichard van der Hoff2018-02-131-13/+27
| | | | | |
| * | | | | Reinstate event_search_postgres_gist handlerRichard van der Hoff2018-02-023-6/+31
| | | | | | | | | | | | | | | | | | | | | | | | People may have queued updates for this, so we can't just delete it.
| * | | | | fix GIST->GIN switchMatthew Hodgson2018-01-093-14/+37
| | | | | |
| * | | | | switch back from GIST to GIN indexesMatthew Hodgson2018-01-091-8/+13
| |/ / / /
* | | | | Merge pull request #2854 from matrix-org/erikj/event_create_workerErik Johnston2018-02-131-0/+13
|\ \ \ \ \ | |_|_|_|/ |/| | | | Create a worker for event creation
| * | | | Add replication http endpoint for event sendingErik Johnston2018-02-071-0/+13
| | | | |
* | | | | Fix log message in purge_historyRichard van der Hoff2018-02-131-2/+1
| | | | | | | | | | | | | | | | | | | | (we don't just remove remote events)
* | | | | Merge pull request #2864 from matrix-org/rav/persist_event_cachingRichard van der Hoff2018-02-131-56/+40
|\ \ \ \ \ | |_|_|_|/ |/| | | | Use StateResolutionHandler to resolve state in persist_events
| * | | | style nitRichard van der Hoff2018-02-131-1/+1
| | | | |
| * | | | Use StateResolutionHandler to resolve state in persist eventsRichard van der Hoff2018-02-051-48/+24
| | | | | | | | | | | | | | | | | | | | ... and thus benefit (hopefully) from its cache.
| * | | | Flatten _get_new_state_after_eventsRichard van der Hoff2018-02-051-44/+46
| | | | | | | | | | | | | | | | | | | | rejig the if statements to simplify the logic and reduce indentation
| * | | | Check that events being persisted have state_groupRichard van der Hoff2018-02-051-4/+9
| | | | |
| * | | | Add event_map param to resolve_state_groupsRichard van der Hoff2018-02-051-0/+1
| | | | |
* | | | | Merge pull request #2858 from matrix-org/rav/purge_updatesRichard van der Hoff2018-02-091-33/+61
|\ \ \ \ \ | |_|/ / / |/| | | | delete_local_events for purge_room_history
| * | | | purge: move room_depth update to endRichard van der Hoff2018-02-091-6/+12
| | | | | | | | | | | | | | | | | | | | ... to avoid locking the table for too long
| * | | | delete_local_events for purge_historyRichard van der Hoff2018-02-091-7/+28
| | | | | | | | | | | | | | | | | | | | Add a flag which makes the purger delete local events
| * | | | purge: Move cache invalidation to more appropriate placeRichard van der Hoff2018-02-091-4/+4
| | | | | | | | | | | | | | | | | | | | it was a bit of a non-sequitur there
| * | | | bump purge logging to infoRichard van der Hoff2018-02-091-13/+14
| | | | | | | | | | | | | | | | | | | | | | | | | this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.
| * | | | rename delete_old_state -> purge_historyRichard van der Hoff2018-02-091-7/+7
| | | | | | | | | | | | | | | | | | | | (beacause it deletes more than state)
* | | | | Store state groups separately from events (#2784)Erik Johnston2018-02-066-103/+166
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* | | | | Merge pull request #2849 from matrix-org/rav/clean_up_state_deltaRichard van der Hoff2018-02-051-10/+23
|\ \ \ \ \ | | |/ / / | |/| | | Remove redundant return value from _calculate_state_delta
| * | | | Remove redundant return value from _calculate_state_deltaRichard van der Hoff2018-02-051-10/+23
| | | | | | | | | | | | | | | | | | | | | | | | | we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.
* | | | | Factor out common code for search insertRichard van der Hoff2018-02-051-33/+62
| | | | | | | | | | | | | | | | | | | | | | | | | we can reuse the same code as is used for event insert, for doing the background index population.
* | | | | Move store_event_search_txn to SearchStoreRichard van der Hoff2018-02-052-35/+41
|/ / / / | | | | | | | | | | | | | | | | ... as a precursor to making event storing and doing the bg update share some code.
* | | | Merge pull request #2841 from matrix-org/rav/refactor_calc_state_deltaRichard van der Hoff2018-02-021-18/+39
|\ \ \ \ | | | | | | | | | | factor _get_new_state_after_events out of _calculate_state_delta
| * | | | factor _get_new_state_after_events out of _calculate_state_deltaRichard van der Hoff2018-01-311-18/+39
| | | | | | | | | | | | | | | | | | | | This reduces the scope of a bunch of variables
* | | | | Merge branch 'develop' into travis/admin-list-mediaTravis Ralston2018-02-013-11/+11
|\ \ \ \ \ | | |_|_|/ | |/| | |
| * | | | Merge pull request #2837 from matrix-org/rav/fix_quarantine_mediaRichard van der Hoff2018-02-011-1/+1
| |\ \ \ \ | | |/ / / | |/| | | Fix sql error in quarantine_media
| | * | | Fix sql error in quarantine_mediaRichard van der Hoff2018-01-301-1/+1
| | | | |
| * | | | Improve exception handling in persist_eventRichard van der Hoff2018-01-291-3/+3
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs
| * / / Fix SQL for user searchRichard van der Hoff2018-01-271-7/+7
| |/ / | | | | | | | | | | | | | | | fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully
* | | pep8Travis Ralston2018-01-311-9/+12
| | | | | | | | | Signed-off-by: Travis Ralston <travpc@gmail.com>
* | | Documentation and namingTravis Ralston2018-01-311-7/+26
| | | | | | | | | Signed-off-by: Travis Ralston <travpc@gmail.com>
* | | Appease the linterTravis Ralston2018-01-201-5/+5
| | | | | | | | | | | | | | | These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>
* | | Add an admin route to get all the media in a roomTravis Ralston2018-01-201-56/+75
|/ / | | | | | | | | This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>
* | Merge pull request #2805 from matrix-org/rav/log_state_resRichard van der Hoff2018-01-171-0/+6
|\ \ | | | | | | Log room when doing state resolution
| * | Log room when doing state resolutionRichard van der Hoff2018-01-171-0/+6
| | | | | | | | | | | | Mostly because it helps figure out what is prompting the resolution
* | | Merge pull request #2783 from matrix-org/erikj/media_last_accessedErik Johnston2018-01-173-4/+37
|\ \ \ | | | | | | | | Keep track of last access time for local media
| * | | Remove lost commentErik Johnston2018-01-171-3/+0
| | | |
| * | | Keep track of last access time for local mediaErik Johnston2018-01-173-4/+40
| | | |
* | | | Merge pull request #2803 from matrix-org/matthew/fix-userdir-sqlMatthew Hodgson2018-01-171-2/+7
|\ \ \ \ | | | | | | | | | | fix SQL when searching all users
| * | | | fix SQL when searching all usersMatthew Hodgson2018-01-171-2/+7
| | | | |
* | | | | Split resolve_events into two functionsRichard van der Hoff2018-01-171-2/+2
|/ / / / | | | | | | | | | | | | ... so that the return type doesn't depend on the arg types
* | | | Track DB scheduling delay per-requestRichard van der Hoff2018-01-161-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.
* | | | rework runInteraction in terms of runConnectionRichard van der Hoff2018-01-161-20/+31
|/ / / | | | | | | | | | ... so that we can share the code
* / / Fix a logcontext leak in persist_eventsRichard van der Hoff2018-01-161-1/+8
|/ / | | | | | | | | | | | | | | | | | | | | | | | | ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)
* / Remove dead code related to default thumbnailsErik Johnston2018-01-091-3/+0
|/
* fix StoreError syntaxMatthew Hodgson2017-12-051-1/+1
|
* fix pep8 and testsMatthew Hodgson2017-12-041-4/+3
|
* switch to a simpler 'search_all_users' button as per review feedbackMatthew Hodgson2017-12-041-24/+16
|
* Merge branch 'develop' into matthew/search-all-local-usersMatthew Hodgson2017-11-307-55/+143
|\
| * Remove pushers when deleting access tokensRichard van der Hoff2017-11-291-5/+5
| | | | | | | | | | Whenever an access token is invalidated, we should remove the associated pushers.
| * Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_errorRichard van der Hoff2017-11-275-14/+82
| |\ | | | | | | Fix error on sqlite 3.7
| | * fix sql failsRichard van der Hoff2017-11-221-1/+1
| | |
| | * Check database in has_completed_background_updatesRichard van der Hoff2017-11-222-11/+32
| | | | | | | | | | | | so that the right thing happens on workers.
| | * Fix error on sqlite 3.7Richard van der Hoff2017-11-214-5/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.
| * | Avoid retrying forever on IntegrityErrorRichard van der Hoff2017-11-271-0/+7
| | |
| * | Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsertRichard van der Hoff2017-11-211-36/+49
| |\ \ | | |/ | |/| Avoid locking account_data tables for upserts
| | * Avoid locking account_data tables for upsertsRichard van der Hoff2017-11-161-36/+49
| | |
* | | remove null constraint on user_dir.room_idMatthew Hodgson2017-11-301-0/+35
| | |
* | | fix alternation operator for FTS4 - how did this ever work!?Matthew Hodgson2017-11-301-1/+1
| | |
* | | fix thinkos galoreMatthew Hodgson2017-11-302-16/+35
| | |
* | | untested WIP but might actually workMatthew Hodgson2017-11-293-13/+38
| | |
* | | Add user_directory_include_pattern config param to expand search results to ↵Matthew Hodgson2017-11-291-4/+12
|/ / | | | | | | | | | | | | | | | | | | additional users Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive. It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users... ...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too. Committing it here as a WIP
* / Avoid locking for upsert on pushers tablesRichard van der Hoff2017-11-162-5/+51
|/ | | | | * replace the upsert into deleted_pushers with an insert * no need to lock for upsert on pusher_throttle
* Fix broken ref to IntegrityErrorRichard van der Hoff2017-11-161-1/+1
|
* Avoid locking `pushers` table on upsertRichard van der Hoff2017-11-161-27/+28
| | | | | Now that _simple_upsert will retry on IntegrityError, we don't need to lock the table.
* _simple_upsert: retry on IntegrityErrorRichard van der Hoff2017-11-161-6/+29
| | | | | | wrap the call to _simple_upsert_txn in a loop so that we retry on an integrityerror: this means we can avoid locking the table provided there is an unique index.
* Cleanup in _simple_upsert_txnRichard van der Hoff2017-11-161-16/+17
| | | | Bail out early to reduce indentation
* Merge pull request #2661 from matrix-org/rav/statereadstoreRichard van der Hoff2017-11-152-216/+231
|\ | | | | Pull out bits of StateStore to a mixin
| * Pull out bits of StateStore to a mixinRichard van der Hoff2017-11-141-204/+220
| | | | | | | | | | | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.
| * Revert "Revert "move _state_group_cache to statestore""Richard van der Hoff2017-11-142-13/+12
| | | | | | | | | | | | | | We're going to fix this properly on this branch, so that the _state_group_cache can end up in StateGroupReadStore. This reverts commit ab335edb023d66cd0be439e045b10ca104b73cb5.
* | Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcsRichard van der Hoff2017-11-151-2/+2
|\ \ | |/ |/| Remove preserve_context_over_{fn, deferred}
| * Remove preserve_context_over_{fn, deferred}Richard van der Hoff2017-11-141-2/+2
| | | | | | | | | | Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.
* | Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-1316-32/+32
|/ | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* Revert "move _state_group_cache to statestore"Erik Johnston2017-11-132-12/+13
| | | | This reverts commit f5cf3638e9c6086e1c33ddad8eda9298cf53a58e.
* Up cache size of get_global_account_data_by_type_for_userErik Johnston2017-11-131-1/+1
|
* Fix typoErik Johnston2017-11-091-1/+1
|
* Merge pull request #2656 from matrix-org/rav/fix_deactivateRichard van der Hoff2017-11-091-2/+1
|\ | | | | Fix 'NoneType' not iterable in /deactivate
| * Fix 'NoneType' not iterable in /deactivateRichard van der Hoff2017-11-091-2/+1
| | | | | | | | make sure we actually return a value from user_delete_access_tokens
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-09 02:11:32 +0000