summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-254-10/+26
| | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* Pull out did_forget to worker storeErik Johnston2018-07-241-25/+27
|
* Merge pull request #3595 from matrix-org/erikj/use_deltasErik Johnston2018-07-241-53/+118
|\ | | | | Use deltas to calculate current state deltas
| * Expand on docstring comment about return valueErik Johnston2018-07-241-0/+5
| |
| * Remove unnecessary iteritemsErik Johnston2018-07-241-1/+1
| |
| * Fixup comment (and indent)Erik Johnston2018-07-241-16/+20
| |
| * Don't fetch state from the database unless neededErik Johnston2018-07-241-12/+18
| |
| * Have _get_new_state_after_events return deltaErik Johnston2018-07-241-17/+48
| | | | | | | | | | | | If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.
| * Don't require to_delete to have event_idsErik Johnston2018-07-241-36/+55
| |
* | Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-242-28/+0
|\ \ | |/ |/| Remove redundant checks on room forgottenness
| * Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-242-15/+31
| |\
| * \ Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-235-26/+70
| |\ \
| * | | Remove redundant checks on room forgottennessRichard van der Hoff2018-07-232-28/+0
| | | | | | | | | | | | | | | | Fixes #3550
* | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-242-13/+24
|\ \ \ \ | | |_|/ | |/| | | | | | erikj/speed_up_calculate_state_delta
| * | | Merge branch 'develop' into rav/logcontext_fixesRichard van der Hoff2018-07-241-11/+21
| |\ \ \
| | * | | fix idiocyRichard van der Hoff2018-07-241-2/+8
| | | | |
| | * | | Add some measure blocks to persist_eventsRichard van der Hoff2018-07-231-11/+15
| | | | | | | | | | | | | | | | | | | | ... to help us figure out where 40% of CPU is going
| * | | | Logcontext fixesRichard van der Hoff2018-07-242-3/+4
| |/ / / | | | | | | | | | | | | Fix some random logcontext leaks.
* | | | Remove unnecessary ifErik Johnston2018-07-241-2/+1
| | | |
* | | | Speed up _calculate_state_deltaErik Johnston2018-07-241-10/+3
|/ / /
* | | CommentsErik Johnston2018-07-231-0/+3
| | |
* | | Only get cached state from context in persist_eventErik Johnston2018-07-231-1/+3
| |/ |/| | | | | | | | | We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.
* | Use new gettersErik Johnston2018-07-233-5/+11
| |
* | Merge branch 'develop' into matthew/sync_deleted_devicesMatthew Hodgson2018-07-236-23/+49
|\|
| * revert 00bc979Richard van der Hoff2018-07-191-4/+3
| | | | | | | | | | ... we've fixed the things that caused the warnings, so we should reinstate the warning.
| * Merge remote-tracking branch 'origin/release-v0.33.0' into developRichard van der Hoff2018-07-191-3/+4
| |\
| | * Disable logcontext warningRichard van der Hoff2018-07-191-3/+4
| | | | | | | | | | | | Temporary workaround to #3518 while we release 0.33.0.
| * | Run things as background processesRichard van der Hoff2018-07-184-16/+29
| |/ | | | | | | | | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
| * Merge pull request #3505 from matrix-org/erikj/receipts_cahceErik Johnston2018-07-121-6/+19
| |\ | | | | | | Use stream cache in get_linearized_receipts_for_room
| | * Update return value docstringErik Johnston2018-07-121-1/+1
| | |
| | * Use 'is not None' and add commentsErik Johnston2018-07-101-2/+6
| | |
| | * Use stream cache in get_linearized_receipts_for_roomErik Johnston2018-07-101-4/+13
| | | | | | | | | | | | | | | This avoids us from uncessarily hitting the database when there has been no change for the room
* | | add trailing commaMatthew Hodgson2018-07-231-1/+1
| | |
* | | yes, we do need to invalidate the device_id_exists_cache when deleting a ↵Matthew Hodgson2018-07-191-1/+0
| | | | | | | | | | | | remote device
* | | spell out that include_deleted_devices requires include_all_devicesMatthew Hodgson2018-07-191-1/+5
| | |
* | | shift to using an explicit deleted flag on m.device_list_update EDUsMatthew Hodgson2018-07-122-20/+25
| | | | | | | | | | | | and generally make it work.
* | | WIP to announce deleted devices over federationMatthew Hodgson2018-07-122-13/+43
|/ / | | | | | | | | | | Previously we queued up the poke correctly when the device was deleted, but then the actual EDU wouldn't get sent, as the device was no longer known. Instead, we now send EDUs for deleted devices too if there's a poke for them.
* / s/becuase/because/gMatthew Hodgson2018-07-101-2/+2
|/
* Attempt to include db threads in cpu usage stats (#3496)Richard van der Hoff2018-07-102-18/+17
| | | | | Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.
* Add CPU metrics for _fetch_event_listRichard van der Hoff2018-07-091-19/+32
| | | | | add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.
* run isortAmber Brown2018-07-0949-246/+244
|
* Invalidate cache on correct threadErik Johnston2018-07-021-1/+2
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-2819-28/+35
|
* Fix error on deleting users pending deactivationDavid Baker2018-06-261-1/+3
| | | | Use simple_delete instead of simple_delete_one as commented
* Revert "Revert "Merge pull request #3431 from ↵Erik Johnston2018-06-253-0/+126
| | | | | | matrix-org/rav/erasure_visibility"" This reverts commit 1d009013b3c3e814177afc59f066e02a202b21cd.
* Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"Richard van der Hoff2018-06-223-126/+0
| | | | | This reverts commit ce0d911156b355c5bf452120bfb08653dad96497, reversing changes made to b4a5d767a94f1680d07edfd583aae54ce422573e.
* Deleting from event_push_actions needs to use an indexMark Haines2018-06-221-1/+8
|
* Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterableRichard van der Hoff2018-06-221-1/+1
|\ | | | | Make _get_joined_hosts_cache cache non-iterable
| * Make _get_joined_hosts_cache cache non-iterableErik Johnston2018-06-221-1/+1
| |
* | Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotationRichard van der Hoff2018-06-221-3/+5
|\ \ | |/ |/| Make push actions rotation configurable
| * Make push actions rotation configurableErik Johnston2018-06-221-3/+5
| |
* | Merge pull request #3431 from matrix-org/rav/erasure_visibilityErik Johnston2018-06-223-0/+126
|\ \ | | | | | | Support hiding events from deleted users
| * | UserErasureStoreRichard van der Hoff2018-06-123-0/+126
| | | | | | | | | | | | to store which users have been erased
* | | Merge pull request #3427 from matrix-org/erikj/remove_filtersErik Johnston2018-06-221-20/+1
|\| | | |/ |/| remove dead filter_events_for_clients
| * remove dead filter_events_for_clientsRichard van der Hoff2018-06-121-20/+1
| | | | | | | | | | | | This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.
* | Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-13/+43
| | | | | | | | | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
* | Merge pull request #3382 from matrix-org/rav/optimise_state_groupsRichard van der Hoff2018-06-221-5/+1
|\ \ | | | | | | Optimise state_group_cache update
| * | Optimise state_group_cache updateRichard van der Hoff2018-06-111-5/+1
| | | | | | | | | | | | | | | (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.
* | | Merge pull request #3419 from matrix-org/rav/events_per_requestRichard van der Hoff2018-06-221-0/+4
|\ \ \ | | | | | | | | Log number of events fetched from DB
| * | | Indirect evt_count updates via method callRichard van der Hoff2018-06-221-1/+1
| | | | | | | | | | | | | | | | so that we can stub it for the sentinel and not have a billion failing UTs
| * | | Log number of events fetched from DBRichard van der Hoff2018-06-211-0/+4
| | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]
* / | Pass around the reactor explicitly (#3385)Amber Brown2018-06-224-9/+9
|/ /
* | Merge pull request #3276 from matrix-org/dbkr/unbindDavid Baker2018-06-111-9/+0
|\ \ | | | | | | Remove email addresses / phone numbers from ID servers when they're removed from synapse
| * \ Merge remote-tracking branch 'origin/develop' into dbkr/unbindDavid Baker2018-05-241-9/+6
| |\ \
| * | | Hit the 3pid unbind endpoint on deactivationDavid Baker2018-05-231-9/+0
| | | |
* | | | Fix sql error in _get_state_groups_from_groupsRichard van der Hoff2018-06-061-1/+1
| | | | | | | | | | | | | | | | | | | | If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.
* | | | Merge pull request #3324 from matrix-org/rav/remove_dead_methodRichard van der Hoff2018-06-041-26/+0
|\ \ \ \ | | | | | | | | | | Remove was_forgotten_at
| * | | | Remove was_forgotten_atRichard van der Hoff2018-06-011-26/+0
| | |_|/ | |/| | | | | | | | | | | | | | | | | | This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
* / | | Add hacky cache factor override systemErik Johnston2018-06-041-2/+2
|/ / /
* | | Merge pull request #3264 from matrix-org/neil/sign-up-statsNeil Johnson2018-06-013-1/+56
|\ \ \ | | | | | | | | daily user type phone home stats
| * | | remove unnecessary INSERTNeil Johnson2018-06-011-1/+0
| | | |
| * | | create users index on creation_tsNeil Johnson2018-05-291-0/+20
| | | |
| * | | create index in backgroundNeil Johnson2018-05-291-0/+7
| | | |
| * | | bump versionNeil Johnson2018-05-291-1/+1
| | | |
| * | | daily user type phone home statsNeil Johnson2018-05-221-0/+29
| |/ /
* | | Handle RRs which arrive before their eventsRichard van der Hoff2018-06-011-25/+19
| | |
* | | Ignore depth when updating read-receiptsRichard van der Hoff2018-06-011-30/+37
| | | | | | | | | | | | Order read receipts by stream ordering instead of depth
* | | Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-315-54/+59
| | | | | | | | | | | | they're not meant to be lazy (#3307)
* | | Merge pull request #3303 from NotAFile/py3-memoryviewAmber Brown2018-05-303-6/+30
|\ \ \ | | | | | | | | use memoryview in py3
| * | | add remaining memoryview changesAdrian Tschira2018-05-292-4/+12
| | | |
| * | | use memoryview in py3Adrian Tschira2018-05-292-2/+18
| | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | | Merge pull request #3281 from NotAFile/py3-six-isinstanceAmber Brown2018-05-301-2/+3
|\ \ \ \ | | | | | | | | | | remaining isintance fixes
| * | | | remaining isintance fixesAdrian Tschira2018-05-241-2/+3
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | | | Exempt AS-registered users from doing gdprRichard van der Hoff2018-05-291-0/+1
| |/ / / |/| | |
* | | | update metrics to be in secondsAmber Brown2018-05-281-15/+15
| | | |
* | | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-2810-43/+60
|\ \ \ \
| * \ \ \ Merge pull request #3279 from NotAFile/py3-more-iteritemsAmber Brown2018-05-241-4/+6
| |\ \ \ \ | | | | | | | | | | | | more six iteritems
| | * | | | more iteritemsAdrian Tschira2018-05-241-4/+6
| | |/ / /
| * / / / Misc. py3 fixesAdrian Tschira2018-05-243-3/+3
| |/ / / | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | Merge pull request #3278 from NotAFile/py3-storage-baseAmber Brown2018-05-241-17/+22
| |\ \ \ | | | | | | | | | | Py3 storage/_base.py
| | * | | Py3 storage/_base.pyAdrian Tschira2018-05-241-17/+22
| | | |/ | | |/| | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-244-9/+16
| |\ \ \ | | | | | | | | | | replace some iteritems with six
| | * | | replace some iteritems with sixAdrian Tschira2018-05-194-9/+16
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | | Merge pull request #3247 from NotAFile/py3-miscAmber Brown2018-05-241-1/+7
| |\ \ \ \ | | |_|/ / | |/| | | Misc Python3 fixes
| | * | | use stand-in value if maxint is not availableAdrian Tschira2018-05-191-1/+7
| | |/ / | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | Remove unused string formatting paramErik Johnston2018-05-221-1/+0
| | | |
| * | | Don't support limitless paginationErik Johnston2018-05-221-8/+6
| | |/ | |/| | | | | | | | | | | | | | | | The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.
* | | cleanupAmber Brown2018-05-221-3/+6
| | |
* | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-2211-75/+286
|\| |
| * | Send users a server notice about consentRichard van der Hoff2018-05-222-7/+59
| | | | | | | | | | | | | | | When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.
| * | Tighter filtering for user_daily_visitsNeil Johnson2018-05-181-1/+5
| |/
| * fix psql compatability bugNeil Johnson2018-05-181-1/+1
| |
| * Merge pull request #3235 from matrix-org/rav/fix_receipts_deferredRichard van der Hoff2018-05-181-7/+11
| |\ | | | | | | Fix error in handling receipts
| | * Fix error in handling receiptsRichard van der Hoff2018-05-181-7/+11
| | | | | | | | | | | | | | | | | | | | | | | | Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234
| * | Merge pull request #3221 from matrix-org/erikj/purge_tokenErik Johnston2018-05-181-24/+53
| |\ \ | | |/ | |/| Make purge_history operate on tokens
| | * Clarify commentErik Johnston2018-05-161-1/+2
| | |
| | * CommentsErik Johnston2018-05-161-0/+8
| | |
| | * Move and rename variableErik Johnston2018-05-161-3/+3
| | |
| | * Make purge_history operate on tokensErik Johnston2018-05-151-8/+9
| | | | | | | | | | | | As we're soon going to change how topological_ordering works
| | * Use events_to_purge table rather than tokenErik Johnston2018-05-151-15/+34
| | |
| * | Merge pull request #3212 from matrix-org/erikj/epa_streamErik Johnston2018-05-172-38/+16
| |\ \ | | | | | | | | Use stream rather depth ordering for push actions
| | * | Use stream rather depth ordering for push actionsErik Johnston2018-05-112-38/+16
| | | | | | | | | | | | | | | | | | | | | | | | This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.
| * | | Merge pull request #3163 from matrix-org/cohort_analyticsNeil Johnson2018-05-165-5/+113
| |\ \ \ | | | | | | | | | | user visit data
| | * | | remove empty lineNeil Johnson2018-05-161-1/+0
| | | | |
| | * | | style and further contraining queryNeil Johnson2018-05-161-8/+12
| | | | |
| | * | | Limit query load of generate_user_daily_visitsNeil Johnson2018-05-151-13/+47
| | | | | | | | | | | | | | | | | | | | The aim is to keep track of when it was last called and only query from that point in time
| | * | | instead of inserting user daily visit data at the end of the day, instead ↵Neil Johnson2018-05-141-40/+14
| | | | | | | | | | | | | | | | | | | | insert incrementally through the day
| | * | | Merge branch 'develop' of https://github.com/matrix-org/synapse into ↵Neil Johnson2018-05-1410-290/+380
| | |\| | | | | | | | | | | | | | | | | cohort_analytics
| | * | | remove user agent from data model, will just join on user_ipsNeil Johnson2018-05-011-12/+11
| | | | |
| | * | | add inidexes based on usageNeil Johnson2018-05-011-6/+2
| | | | |
| | * | | pep8Neil Johnson2018-04-251-3/+4
| | | | |
| | * | | Generate user daily statsNeil Johnson2018-04-255-5/+106
| | | | |
| * | | | Merge pull request #3213 from matrix-org/rav/consent_handlerRichard van der Hoff2018-05-162-0/+36
| |\ \ \ \ | | |_|_|/ | |/| | | ConsentResource to gather policy consent from users
| | * | | ConsentResource to gather policy consent from usersRichard van der Hoff2018-05-152-0/+36
| | | |/ | | |/| | | | | | | | | | | | | Hopefully there are enough comments and docs in this that it makes sense on its own.
* | / | rest of the changesAmber Brown2018-05-212-27/+18
|/ / /
* | | Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivateDavid Baker2018-05-142-0/+64
|\ \ \ | |/ / |/| | Part user from rooms on account deactivate
| * | Many docstringsDavid Baker2018-05-101-0/+12
| | |
| * | Add the schema fileDavid Baker2018-05-091-0/+25
| | |
| * | Part deactivated users in the backgroundDavid Baker2018-05-091-0/+27
| | | | | | | | | | | | | | | One room at a time so we don't take out the whole server with leave events, and restart at server restart.
* | | Merge pull request #3199 from matrix-org/erikj/pagination_syncErik Johnston2018-05-091-39/+36
|\ \ \ | | | | | | | | Refactor sync APIs to reuse pagination API
| * | | Refactor sync APIs to reuse pagination APIErik Johnston2018-05-091-38/+35
| | | | | | | | | | | | | | | | | | | | | | | | The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.
| * | | Don't unnecessarily require token to be stream tokenErik Johnston2018-05-091-1/+1
| | | | | | | | | | | | | | | | | | | | This allows calling the `get_recent_event_ids_for_room` function in more situations.
* | | | Merge pull request #3198 from matrix-org/erikj/fixup_return_paginationErik Johnston2018-05-091-1/+15
|\ \ \ \ | | | | | | | | | | Refactor get_recent_events_for_room return type
| * | | | Fix up commentErik Johnston2018-05-091-1/+1
| | | | |
| * | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
| |\ \ \ \ | | |/ / / | |/| | | | | | | | erikj/fixup_return_pagination
| * | | | Refactor get_recent_events_for_room return typeErik Johnston2018-05-091-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
|\ \ \ \ \ | | |/ / / | |/| | | | | | | | erikj/remove_membership_change
| * | | | Update commentsErik Johnston2018-05-091-5/+6
| |/ / /
* / / / Remove unused code path from member change DB funcErik Johnston2018-05-091-24/+10
|/ / / | | | | | | | | | | | | The function is never called without a from_key, so we can remove all the handling for that scenario.
* | | Refactor pagination DB API to return concrete typeErik Johnston2018-05-091-28/+48
| | | | | | | | | | | | | | | This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.
* | | Remove unused from_token paramErik Johnston2018-05-091-7/+4
| | |
* | | Refactor recent events func to use pagination funcErik Johnston2018-05-091-48/+27
| | | | | | | | | | | | This also removes a cache that is unlikely to ever get hit.
* | | Fix up comments and make function privateErik Johnston2018-05-091-7/+7
| | |
* | | Reuse existing pagination code for context APIErik Johnston2018-05-081-75/+15
| | |
* | | Parse tokens before calling DB functionErik Johnston2018-05-081-14/+18
| | |
* | | Only fetch required fields from databaseErik Johnston2018-05-081-1/+2
| | |
* | | Split paginate_room_events storage functionErik Johnston2018-05-081-28/+72
|/ /
* | Merge pull request #3141 from matrix-org/erikj/fixup_stateErik Johnston2018-05-031-37/+67
|\ \ | | | | | | Refactor event storage to prepare for changes in state calculations
| * | Fix up grammarErik Johnston2018-05-031-3/+3
| | |
| * | Refactor event storage to not require stateErik Johnston2018-04-271-37/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.
* | | Merge pull request #3129 from matrix-org/matthew/fix_group_dupsMatthew Hodgson2018-04-301-0/+57
|\ \ \ | | | | | | | | remove duplicates from groups tables
| * | | fix missing importMatthew Hodgson2018-04-281-0/+1
| | | |
| * | | pep8Matthew Hodgson2018-04-281-3/+5
| | | |
| * | | make it work with sqliteMatthew Hodgson2018-04-282-34/+54
| | | |
| * | | remove duplicates from groups tablesMatthew Hodgson2018-04-251-0/+34
| | |/ | |/| | | | | | | | | | and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
* | | Merge branch 'develop' into py3-xrange-1Richard van der Hoff2018-04-302-4/+6
|\ \ \
| * \ \ Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-272-8/+18
| |\ \ \
| * | | | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-272-4/+6
| | |_|/ | |/| | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | | Move more xrange to sixAdrian Tschira2018-04-284-4/+12
| |/ / |/| | | | | | | | | | | | | | plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptionsRichard van der Hoff2018-04-271-7/+17
|\ \ \ | | | | | | | | Improve exception handling for background processes
| * | | Improve exception handling for background processesRichard van der Hoff2018-04-271-7/+17
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* / / Add missing consumeErrorsRichard van der Hoff2018-04-271-1/+1
|/ / | | | | | | | | In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.
* | Fix media admin APIsErik Johnston2018-04-261-2/+2
| |
* | Also fix reindexing of searchErik Johnston2018-04-251-1/+1
| |
* | Fix quarantine media admin APIErik Johnston2018-04-251-1/+1
|/
* Merge pull request #3110 from NotAFile/py3-six-queueRichard van der Hoff2018-04-201-2/+4
|\ | | | | Replace Queue with six.moves.queue
| * Replace Queue with six.moves.queueAdrian Tschira2018-04-161-2/+4
| | | | | | | | | | | | and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3117 from matrix-org/rav/refactor_have_eventsRichard van der Hoff2018-04-201-7/+42
|\ \ | | | | | | Refactor store.have_events
| * | Refactor store.have_eventsRichard van der Hoff2018-04-201-7/+42
| |/ | | | | | | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
* / Avoid creating events with huge numbers of prev_eventsRichard van der Hoff2018-04-161-16/+41
|/ | | | | | In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.
* Format docstringErik Johnston2018-04-121-2/+3
|
* Doc we raise on unknown eventErik Johnston2018-04-121-2/+3
|
* Track last processed event received_tsErik Johnston2018-04-111-0/+18
|
* Track where event stream processing have gotten up toErik Johnston2018-04-111-0/+3
|
* Merge pull request #2760 from Valodim/pypyRichard van der Hoff2018-04-112-1/+6
|\ | | | | Synapse on PyPy
| * Fix pep8 error on psycopg2cffi hackRichard van der Hoff2018-04-101-6/+5
| |
| * Use psycopg2cffi module instead of psycopg2 if running on pypyVincent Breitmoser2018-04-102-2/+8
| | | | | | | | | | | | | | The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.
* | fix typoNeil Johnson2018-04-101-2/+2
| |
* | remove errant print v0.27.3-rc2Neil Johnson2018-04-091-1/+1
| |
* | Fix msec to sec, againNeil Johnson2018-04-091-3/+3
| |
* | Fix msec to secNeil Johnson2018-04-091-1/+1
| |
* | Fix psycopg2 interpolationNeil Johnson2018-04-091-5/+5
|/
* Don't use redundant inlineCallbacksLuke Barnard2018-04-061-4/+1
|
* join_rule -> join_policyLuke Barnard2018-04-061-1/+1
|
* is_joinable -> join_ruleLuke Barnard2018-04-061-4/+1
|
* pep8David Baker2018-04-061-1/+2
|
* Implement group join APIDavid Baker2018-04-061-3/+9
|
* Merge pull request #3045 from matrix-org/dbkr/group_joinableLuke Barnard2018-04-053-0/+42
|\ | | | | Add joinability for groups
| * NON NULL -> NOT NULLLuke Barnard2018-04-051-1/+1
| |
| * Use DEFAULT join_policy of "invite" in dbLuke Barnard2018-04-051-1/+1
| |
| * Document set_group_join_policyLuke Barnard2018-04-051-0/+6
| |
| * Use join_policy API instead of joinableLuke Barnard2018-04-032-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".
| * OK, smallint it is thenDavid Baker2018-03-281-1/+1
| |
| * Grr. Copy the definition from is_adminDavid Baker2018-03-281-1/+1
| |
| * Make column definition that works on both dbsDavid Baker2018-03-281-1/+1
| |
| * Add schema delta fileDavid Baker2018-03-281-0/+16
| |
| * Add joinability for groupsDavid Baker2018-03-282-1/+15
| | | | | | | | | | Adds API to set the 'joinable' flag, and corresponding flag in the table.
* | Merge pull request #3041 from matrix-org/r30_statsNeil Johnson2018-04-054-7/+111
|\ \ | | | | | | R30 stats
| * | Review commentsNeil Johnson2018-04-051-4/+5
| | | | | | | | | | | | | | | Use iteritems over item to loop over dict formatting
| * | pep8Neil Johnson2018-03-291-2/+4
| | |
| * | Remove need for sqlite specific queryNeil Johnson2018-03-291-30/+57
| | |
| * | fix pep8 errorsNeil Johnson2018-03-281-3/+0
| | |
| * | remove twisted deferral cruftNeil Johnson2018-03-281-6/+3
| | |
| * | bump schema versionNeil Johnson2018-03-281-1/+1
| | |
| * | Support multi client R30 for psqlNeil Johnson2018-03-281-8/+28
| | |
| * | Add user_ips last seen indexNeil Johnson2018-03-282-0/+24
| | |
| * | No need to cast in count_daily_usersNeil Johnson2018-03-281-2/+2
| | |
| * | query and call for r30 statsNeil Johnson2018-03-281-0/+36
| | |
| * | count_daily_users failed if db was sqlite due to type failure - presumably ↵Neil Johnson2018-03-281-1/+1
| |/ | | | | | | this prevcented all sqlite homeservers reporting home
* | Merge pull request #3060 from matrix-org/rav/kill_event_contentRichard van der Hoff2018-04-053-7/+12
|\ \ | | | | | | Remove uses of events.content
| * | Remove uses of events.contentRichard van der Hoff2018-03-293-7/+12
| | |
* | | Merge pull request #3049 from matrix-org/rav/use_staticjsonRichard van der Hoff2018-04-031-15/+8
|\ \ \ | | | | | | | | Use static JSONEncoders
| * | | Use static JSONEncodersRichard van der Hoff2018-03-291-15/+8
| |/ / | | | | | | | | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.
* / / Use simplejson throughoutRichard van der Hoff2018-03-291-1/+2
|/ / | | | | | | Let's use simplejson rather than json, for consistency.
* | Fix search_user_dir multiple sqlite versions do different thingsNeil Johnson2018-03-281-2/+2
| |
* | Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_idErik Johnston2018-03-281-3/+1
|\ \ | | | | | | Linearize calls to _generate_user_id
| * | Don't use _cursor_to_dict in find_next_generated_user_id_localpartErik Johnston2018-03-261-3/+1
| |/
* | Merge pull request #3030 from matrix-org/erikj/no_ujsonErik Johnston2018-03-281-1/+1
|\ \ | | | | | | Remove last usage of ujson
| * | Remove last usage of ujsonErik Johnston2018-03-261-1/+1
| |/
* | CommentErik Johnston2018-03-271-4/+5
| |
* | Fix indentErik Johnston2018-03-271-1/+1
| |
* | CommentErik Johnston2018-03-271-0/+7
| |
* | Add counter metrics for calculating state deltaErik Johnston2018-03-271-1/+30
|/ | | | | | This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-1917-26/+25
|\
| * Remove wrong commentErik Johnston2018-03-161-1/+0
| |
| * Replace ujson with simplejsonErik Johnston2018-03-1517-26/+26
| |
* | Replace some ujson with simplejson to make it workErik Johnston2018-03-162-2/+2
| |
* | Merge pull request #2988 from matrix-org/erikj/split_profile_storeErik Johnston2018-03-141-24/+26
|\ \ | | | | | | Split up ProfileStore
| * | Split up ProfileStoreErik Johnston2018-03-131-24/+26
| | |
* | | Merge pull request #2993 from matrix-org/erikj/is_blockedErik Johnston2018-03-141-13/+17
|\ \ \ | | | | | | | | Add is_blocked to worker store
| * | | Add is_blocked to worker storeErik Johnston2018-03-131-13/+17
| |/ /
* / / fix bug #2926 (loading all state for a given type from the DB if the ↵Matthew Hodgson2018-03-131-7/+27
|/ / | | | | | | | | state_key is None) (#2990) Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
* | Add Measure block for persist_eventsRichard van der Hoff2018-03-131-4/+5
| | | | | | | | This seems like a useful thing to measure.
* | Fix race in sync when joining roomErik Johnston2018-03-072-2/+27
| | | | | | | | | | | | | | | | | | | | | | The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.
* | Merge pull request #2946 from matrix-org/rav/timestamp_to_purgeRichard van der Hoff2018-03-061-0/+27
|\ \ | | | | | | Implement purge_history by timestamp
| * | Provide a means to pass a timestamp to purge_historyRichard van der Hoff2018-03-051-0/+27
| | |
* | | Merge pull request #2948 from matrix-org/erikj/kill_as_syncErik Johnston2018-03-062-153/+5
|\ \ \ | | | | | | | | Remove ability for AS users to call /events and /sync
| * | | Remove ability for AS users to call /events and /syncErik Johnston2018-03-052-153/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144
* | | | Merge pull request #2947 from matrix-org/erikj/split_directory_storeErik Johnston2018-03-051-23/+27
|\ \ \ \ | |_|/ / |/| | | Split Directory store
| * | | Fix cache invalidation on deletionErik Johnston2018-03-051-1/+4
| | | |
| * | | Split Directory storeErik Johnston2018-03-051-22/+23
| |/ /
* | | Merge pull request #2943 from ↵Richard van der Hoff2018-03-051-12/+71
|\ \ \ | |/ / |/| | | | | | | | matrix-org/rav/fix_find_first_stream_ordering_after_ts Test and fix find_first_stream_ordering_after_ts
| * | Test and fix find_first_stream_ordering_after_tsRichard van der Hoff2018-03-051-15/+53
| | | | | | | | | | | | It seemed to suffer from a bunch of off-by-one errors.
| * | Add find_first_stream_ordering_after_tsRichard van der Hoff2018-03-051-0/+21
| | | | | | | | | | | | Expose this as a public function which can be called outside a txn
* | | Merge pull request #2934 from matrix-org/erikj/cache_fixErik Johnston2018-03-052-13/+15
|\ \ \ | | | | | | | | Fix bug with delayed cache invalidation stream
| * | | Fix bug with delayed cache invalidation streamErik Johnston2018-03-022-13/+15
| | | | | | | | | | | | | | | | | | | | | | | | We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.
* | | | Split registration storeErik Johnston2018-03-021-57/+61
|/ / /
* | | Merge pull request #2925 from matrix-org/erikj/split_sig_fedErik Johnston2018-03-012-133/+143
|\ \ \ | |/ / |/| | Split out SignatureStore and EventFederationStore
| * | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fedErik Johnston2018-03-014-371/+406
| |\ \
| * | | Stub out broken function only used for cacheErik Johnston2018-03-011-1/+3
| | | |
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-09 02:50:58 +0000