summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #3610 from matrix-org/rav/fix_looping_callsRichard van der Hoff2018-07-265-6/+8
|\ | | | | Fix some looping_call calls which were broken in #3604
| * Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-265-6/+8
| | | | | | | | | | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* | comment on event_edgesRichard van der Hoff2018-07-261-1/+2
|/
* switch missing_types to be a boolMatthew Hodgson2018-07-251-4/+4
|
* Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-07-258-125/+187
|\
| * Merge pull request #3603 from matrix-org/erikj/handle_outliersErik Johnston2018-07-251-6/+8
| |\ | | | | | | Correctly handle outliers during persist events
| | * Actually fix it by adding continueErik Johnston2018-07-251-0/+1
| | |
| | * Fix typo in conditionalErik Johnston2018-07-251-1/+1
| | |
| | * Correctly handle outliers during persist eventsErik Johnston2018-07-251-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601
| * | Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_moreRichard van der Hoff2018-07-252-5/+10
| |\ \ | | | | | | | | Fix another logcontext leak in _persist_events
| | * | Fix another logcontext leak in _persist_eventsRichard van der Hoff2018-07-252-5/+10
| | |/ | | | | | | | | | | | | | | | | | | | | | We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from
| * | Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_errorRichard van der Hoff2018-07-251-1/+1
| |\ \ | | | | | | | | Fix occasional 'tuple index out of range' error
| | * | Fix occasional 'tuple index out of range' errorRichard van der Hoff2018-07-251-1/+1
| | |/ | | | | | | | | | | | | | | | | | | This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.
| * / Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-254-10/+26
| |/ | | | | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
| * Pull out did_forget to worker storeErik Johnston2018-07-241-25/+27
| |
| * Merge pull request #3595 from matrix-org/erikj/use_deltasErik Johnston2018-07-241-53/+118
| |\ | | | | | | Use deltas to calculate current state deltas
| | * Expand on docstring comment about return valueErik Johnston2018-07-241-0/+5
| | |
| | * Remove unnecessary iteritemsErik Johnston2018-07-241-1/+1
| | |
| | * Fixup comment (and indent)Erik Johnston2018-07-241-16/+20
| | |
| | * Don't fetch state from the database unless neededErik Johnston2018-07-241-12/+18
| | |
| | * Have _get_new_state_after_events return deltaErik Johnston2018-07-241-17/+48
| | | | | | | | | | | | | | | | | | If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.
| | * Don't require to_delete to have event_idsErik Johnston2018-07-241-36/+55
| | |
| * | Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-242-28/+0
| |\ \ | | |/ | |/| Remove redundant checks on room forgottenness
| | * Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-242-15/+31
| | |\
| | * \ Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_roomRichard van der Hoff2018-07-235-26/+70
| | |\ \
| | * | | Remove redundant checks on room forgottennessRichard van der Hoff2018-07-232-28/+0
| | | | | | | | | | | | | | | | | | | | Fixes #3550
* | | | | incorporate more review.Matthew Hodgson2018-07-251-16/+14
| | | | |
* | | | | handle the edge case for _get_some_state_from_cache where types is []Matthew Hodgson2018-07-241-1/+7
| | | | |
* | | | | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-07-241-12/+4
|\| | | |
| * | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-242-13/+24
| |\ \ \ \ | | | |_|/ | | |/| | | | | | | erikj/speed_up_calculate_state_delta
| * | | | Remove unnecessary ifErik Johnston2018-07-241-2/+1
| | | | |
| * | | | Speed up _calculate_state_deltaErik Johnston2018-07-241-10/+3
| | | | |
* | | | | consider non-filter_type types as wildcards, thus missing from the ↵Matthew Hodgson2018-07-241-1/+4
| | | | | | | | | | | | | | | | | | | | state-group-cache
* | | | | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-07-242-14/+25
|\ \ \ \ \ | | |/ / / | |/| | |
| * | | | Merge branch 'develop' into rav/logcontext_fixesRichard van der Hoff2018-07-241-11/+21
| |\ \ \ \
| | * | | | fix idiocyRichard van der Hoff2018-07-241-2/+8
| | | | | |
| | * | | | Add some measure blocks to persist_eventsRichard van der Hoff2018-07-231-11/+15
| | |/ / / | | | | | | | | | | | | | | | ... to help us figure out where 40% of CPU is going
| * / / / Logcontext fixesRichard van der Hoff2018-07-242-3/+4
| |/ / / | | | | | | | | | | | | Fix some random logcontext leaks.
* | | | incorporate more reviewMatthew Hodgson2018-07-241-27/+9
| | | |
* | | | handle case where types is [] on postgres correctlyMatthew Hodgson2018-07-231-1/+1
| | | |
* | | | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-07-235-26/+75
|\| | |
| * | | CommentsErik Johnston2018-07-231-0/+3
| | | |
| * | | Only get cached state from context in persist_eventErik Johnston2018-07-231-1/+3
| | |/ | |/| | | | | | | | | | | | | We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.
| * | Use new gettersErik Johnston2018-07-233-5/+11
| | |
| * | Merge branch 'develop' into matthew/sync_deleted_devicesMatthew Hodgson2018-07-236-23/+49
| |\|
| * | add trailing commaMatthew Hodgson2018-07-231-1/+1
| | |
| * | yes, we do need to invalidate the device_id_exists_cache when deleting a ↵Matthew Hodgson2018-07-191-1/+0
| | | | | | | | | | | | remote device
| * | spell out that include_deleted_devices requires include_all_devicesMatthew Hodgson2018-07-191-1/+5
| | |
| * | shift to using an explicit deleted flag on m.device_list_update EDUsMatthew Hodgson2018-07-122-20/+25
| | | | | | | | | | | | and generally make it work.
| * | WIP to announce deleted devices over federationMatthew Hodgson2018-07-122-13/+43
| | | | | | | | | | | | | | | | | | Previously we queued up the poke correctly when the device was deleted, but then the actual EDU wouldn't get sent, as the device was no longer known. Instead, we now send EDUs for deleted devices too if there's a poke for them.
* | | incorporate reviewMatthew Hodgson2018-07-231-12/+8
| | |
* | | fix thinkos; unbreak testsMatthew Hodgson2018-07-191-2/+2
| | |
* | | incorporate reviewMatthew Hodgson2018-07-191-3/+4
| | |
* | | add a filtered_types param to limit filtering to specific typesMatthew Hodgson2018-07-191-53/+60
| | |
* | | merge develop pydoc for _get_state_for_groupsMatthew Hodgson2018-07-1950-375/+550
|\ \ \ | | |/ | |/|
| * | revert 00bc979Richard van der Hoff2018-07-191-4/+3
| | | | | | | | | | | | | | | ... we've fixed the things that caused the warnings, so we should reinstate the warning.
| * | Merge remote-tracking branch 'origin/release-v0.33.0' into developRichard van der Hoff2018-07-191-3/+4
| |\ \
| | * | Disable logcontext warningRichard van der Hoff2018-07-191-3/+4
| | | | | | | | | | | | | | | | Temporary workaround to #3518 while we release 0.33.0.
| * | | Run things as background processesRichard van der Hoff2018-07-184-16/+29
| |/ / | | | | | | | | | | | | | | | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
| * | Merge pull request #3505 from matrix-org/erikj/receipts_cahceErik Johnston2018-07-121-6/+19
| |\ \ | | |/ | |/| Use stream cache in get_linearized_receipts_for_room
| | * Update return value docstringErik Johnston2018-07-121-1/+1
| | |
| | * Use 'is not None' and add commentsErik Johnston2018-07-101-2/+6
| | |
| | * Use stream cache in get_linearized_receipts_for_roomErik Johnston2018-07-101-4/+13
| | | | | | | | | | | | | | | This avoids us from uncessarily hitting the database when there has been no change for the room
| * | s/becuase/because/gMatthew Hodgson2018-07-101-2/+2
| |/
| * Attempt to include db threads in cpu usage stats (#3496)Richard van der Hoff2018-07-102-18/+17
| | | | | | | | | | Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.
| * Add CPU metrics for _fetch_event_listRichard van der Hoff2018-07-091-19/+32
| | | | | | | | | | add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.
| * run isortAmber Brown2018-07-0949-246/+244
| |
| * Invalidate cache on correct threadErik Johnston2018-07-021-1/+2
| |
| * Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-2819-28/+35
| |
| * Fix error on deleting users pending deactivationDavid Baker2018-06-261-1/+3
| | | | | | | | Use simple_delete instead of simple_delete_one as commented
| * Revert "Revert "Merge pull request #3431 from ↵Erik Johnston2018-06-253-0/+126
| | | | | | | | | | | | matrix-org/rav/erasure_visibility"" This reverts commit 1d009013b3c3e814177afc59f066e02a202b21cd.
| * Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"Richard van der Hoff2018-06-223-126/+0
| | | | | | | | | | This reverts commit ce0d911156b355c5bf452120bfb08653dad96497, reversing changes made to b4a5d767a94f1680d07edfd583aae54ce422573e.
| * Deleting from event_push_actions needs to use an indexMark Haines2018-06-221-1/+8
| |
| * Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterableRichard van der Hoff2018-06-221-1/+1
| |\ | | | | | | Make _get_joined_hosts_cache cache non-iterable
| | * Make _get_joined_hosts_cache cache non-iterableErik Johnston2018-06-221-1/+1
| | |
| * | Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotationRichard van der Hoff2018-06-221-3/+5
| |\ \ | | |/ | |/| Make push actions rotation configurable
| | * Make push actions rotation configurableErik Johnston2018-06-221-3/+5
| | |
| * | Merge pull request #3431 from matrix-org/rav/erasure_visibilityErik Johnston2018-06-223-0/+126
| |\ \ | | | | | | | | Support hiding events from deleted users
| | * | UserErasureStoreRichard van der Hoff2018-06-123-0/+126
| | | | | | | | | | | | | | | | to store which users have been erased
| * | | Merge pull request #3427 from matrix-org/erikj/remove_filtersErik Johnston2018-06-221-20/+1
| |\| | | | |/ | |/| remove dead filter_events_for_clients
| | * remove dead filter_events_for_clientsRichard van der Hoff2018-06-121-20/+1
| | | | | | | | | | | | | | | | | | This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.
| * | Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-13/+43
| | | | | | | | | | | | | | | | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
| * | Merge pull request #3382 from matrix-org/rav/optimise_state_groupsRichard van der Hoff2018-06-221-5/+1
| |\ \ | | | | | | | | Optimise state_group_cache update
| | * | Optimise state_group_cache updateRichard van der Hoff2018-06-111-5/+1
| | | | | | | | | | | | | | | | | | | | (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.
| * | | Merge pull request #3419 from matrix-org/rav/events_per_requestRichard van der Hoff2018-06-221-0/+4
| |\ \ \ | | | | | | | | | | Log number of events fetched from DB
| | * | | Indirect evt_count updates via method callRichard van der Hoff2018-06-221-1/+1
| | | | | | | | | | | | | | | | | | | | so that we can stub it for the sentinel and not have a billion failing UTs
| | * | | Log number of events fetched from DBRichard van der Hoff2018-06-211-0/+4
| | | |/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]
| * / | Pass around the reactor explicitly (#3385)Amber Brown2018-06-224-9/+9
| |/ /
| * | Merge pull request #3276 from matrix-org/dbkr/unbindDavid Baker2018-06-111-9/+0
| |\ \ | | | | | | | | Remove email addresses / phone numbers from ID servers when they're removed from synapse
| | * \ Merge remote-tracking branch 'origin/develop' into dbkr/unbindDavid Baker2018-05-241-9/+6
| | |\ \
| | * | | Hit the 3pid unbind endpoint on deactivationDavid Baker2018-05-231-9/+0
| | | | |
* | | | | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-06-102-29/+3
|\| | | |
| * | | | Fix sql error in _get_state_groups_from_groupsRichard van der Hoff2018-06-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.
| * | | | Merge pull request #3324 from matrix-org/rav/remove_dead_methodRichard van der Hoff2018-06-041-26/+0
| |\ \ \ \ | | | | | | | | | | | | Remove was_forgotten_at
| | * | | | Remove was_forgotten_atRichard van der Hoff2018-06-011-26/+0
| | | |_|/ | | |/| | | | | | | | | | | | | | | | | | | | | | This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
| * / | | Add hacky cache factor override systemErik Johnston2018-06-041-2/+2
| |/ / /
* | | | Merge branch 'develop' into matthew/filter_membersMatthew Hodgson2018-06-0412-92/+179
|\| | |
| * | | Merge pull request #3264 from matrix-org/neil/sign-up-statsNeil Johnson2018-06-013-1/+56
| |\ \ \ | | | | | | | | | | daily user type phone home stats
| | * | | remove unnecessary INSERTNeil Johnson2018-06-011-1/+0
| | | | |
| | * | | create users index on creation_tsNeil Johnson2018-05-291-0/+20
| | | | |
| | * | | create index in backgroundNeil Johnson2018-05-291-0/+7
| | | | |
| | * | | bump versionNeil Johnson2018-05-291-1/+1
| | | | |
| | * | | daily user type phone home statsNeil Johnson2018-05-221-0/+29
| | |/ /
| * | | Handle RRs which arrive before their eventsRichard van der Hoff2018-06-011-25/+19
| | | |
| * | | Ignore depth when updating read-receiptsRichard van der Hoff2018-06-011-30/+37
| | | | | | | | | | | | | | | | Order read receipts by stream ordering instead of depth
| * | | Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-315-54/+59
| | | | | | | | | | | | | | | | they're not meant to be lazy (#3307)
| * | | Merge pull request #3303 from NotAFile/py3-memoryviewAmber Brown2018-05-303-6/+30
| |\ \ \ | | | | | | | | | | use memoryview in py3
| | * | | add remaining memoryview changesAdrian Tschira2018-05-292-4/+12
| | | | |
| | * | | use memoryview in py3Adrian Tschira2018-05-292-2/+18
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | | Merge pull request #3281 from NotAFile/py3-six-isinstanceAmber Brown2018-05-301-2/+3
| |\ \ \ \ | | | | | | | | | | | | remaining isintance fixes
| | * | | | remaining isintance fixesAdrian Tschira2018-05-241-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | | | Exempt AS-registered users from doing gdprRichard van der Hoff2018-05-291-0/+1
| | |/ / / | |/| | |
* | | | | add pydocMatthew Hodgson2018-05-291-22/+54
| | | | |
* | | | | merge developMatthew Hodgson2018-05-2940-519/+1165
|\| | | |
| * | | | update metrics to be in secondsAmber Brown2018-05-281-15/+15
| | | | |
| * | | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-2810-43/+60
| |\ \ \ \
| | * \ \ \ Merge pull request #3279 from NotAFile/py3-more-iteritemsAmber Brown2018-05-241-4/+6
| | |\ \ \ \ | | | | | | | | | | | | | | more six iteritems
| | | * | | | more iteritemsAdrian Tschira2018-05-241-4/+6
| | | |/ / /
| | * / / / Misc. py3 fixesAdrian Tschira2018-05-243-3/+3
| | |/ / / | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | | Merge pull request #3278 from NotAFile/py3-storage-baseAmber Brown2018-05-241-17/+22
| | |\ \ \ | | | | | | | | | | | | Py3 storage/_base.py
| | | * | | Py3 storage/_base.pyAdrian Tschira2018-05-241-17/+22
| | | | |/ | | | |/| | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | | Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-244-9/+16
| | |\ \ \ | | | | | | | | | | | | replace some iteritems with six
| | | * | | replace some iteritems with sixAdrian Tschira2018-05-194-9/+16
| | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | | | Merge pull request #3247 from NotAFile/py3-miscAmber Brown2018-05-241-1/+7
| | |\ \ \ \ | | | |_|/ / | | |/| | | Misc Python3 fixes
| | | * | | use stand-in value if maxint is not availableAdrian Tschira2018-05-191-1/+7
| | | |/ / | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | | Remove unused string formatting paramErik Johnston2018-05-221-1/+0
| | | | |
| | * | | Don't support limitless paginationErik Johnston2018-05-221-8/+6
| | | |/ | | |/| | | | | | | | | | | | | | | | | | | | | The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.
| * | | cleanupAmber Brown2018-05-221-3/+6
| | | |
| * | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-2211-75/+286
| |\| |
| | * | Send users a server notice about consentRichard van der Hoff2018-05-222-7/+59
| | | | | | | | | | | | | | | | | | | | When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.
| | * | Tighter filtering for user_daily_visitsNeil Johnson2018-05-181-1/+5
| | |/
| | * fix psql compatability bugNeil Johnson2018-05-181-1/+1
| | |
| | * Merge pull request #3235 from matrix-org/rav/fix_receipts_deferredRichard van der Hoff2018-05-181-7/+11
| | |\ | | | | | | | | Fix error in handling receipts
| | | * Fix error in handling receiptsRichard van der Hoff2018-05-181-7/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234
| | * | Merge pull request #3221 from matrix-org/erikj/purge_tokenErik Johnston2018-05-181-24/+53
| | |\ \ | | | |/ | | |/| Make purge_history operate on tokens
| | | * Clarify commentErik Johnston2018-05-161-1/+2
| | | |
| | | * CommentsErik Johnston2018-05-161-0/+8
| | | |
| | | * Move and rename variableErik Johnston2018-05-161-3/+3
| | | |
| | | * Make purge_history operate on tokensErik Johnston2018-05-151-8/+9
| | | | | | | | | | | | | | | | As we're soon going to change how topological_ordering works
| | | * Use events_to_purge table rather than tokenErik Johnston2018-05-151-15/+34
| | | |
| | * | Merge pull request #3212 from matrix-org/erikj/epa_streamErik Johnston2018-05-172-38/+16
| | |\ \ | | | | | | | | | | Use stream rather depth ordering for push actions
| | | * | Use stream rather depth ordering for push actionsErik Johnston2018-05-112-38/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.
| | * | | Merge pull request #3163 from matrix-org/cohort_analyticsNeil Johnson2018-05-165-5/+113
| | |\ \ \ | | | | | | | | | | | | user visit data
| | | * | | remove empty lineNeil Johnson2018-05-161-1/+0
| | | | | |
| | | * | | style and further contraining queryNeil Johnson2018-05-161-8/+12
| | | | | |
| | | * | | Limit query load of generate_user_daily_visitsNeil Johnson2018-05-151-13/+47
| | | | | | | | | | | | | | | | | | | | | | | | The aim is to keep track of when it was last called and only query from that point in time
| | | * | | instead of inserting user daily visit data at the end of the day, instead ↵Neil Johnson2018-05-141-40/+14
| | | | | | | | | | | | | | | | | | | | | | | | insert incrementally through the day
| | | * | | Merge branch 'develop' of https://github.com/matrix-org/synapse into ↵Neil Johnson2018-05-1410-290/+380
| | | |\| | | | | | | | | | | | | | | | | | | | cohort_analytics
| | | * | | remove user agent from data model, will just join on user_ipsNeil Johnson2018-05-011-12/+11
| | | | | |
| | | * | | add inidexes based on usageNeil Johnson2018-05-011-6/+2
| | | | | |
| | | * | | pep8Neil Johnson2018-04-251-3/+4
| | | | | |
| | | * | | Generate user daily statsNeil Johnson2018-04-255-5/+106
| | | | | |
| | * | | | Merge pull request #3213 from matrix-org/rav/consent_handlerRichard van der Hoff2018-05-162-0/+36
| | |\ \ \ \ | | | |_|_|/ | | |/| | | ConsentResource to gather policy consent from users
| | | * | | ConsentResource to gather policy consent from usersRichard van der Hoff2018-05-152-0/+36
| | | | |/ | | | |/| | | | | | | | | | | | | | | | Hopefully there are enough comments and docs in this that it makes sense on its own.
| * | / | rest of the changesAmber Brown2018-05-212-27/+18
| |/ / /
| * | | Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivateDavid Baker2018-05-142-0/+64
| |\ \ \ | | |/ / | |/| | Part user from rooms on account deactivate
| | * | Many docstringsDavid Baker2018-05-101-0/+12
| | | |
| | * | Add the schema fileDavid Baker2018-05-091-0/+25
| | | |
| | * | Part deactivated users in the backgroundDavid Baker2018-05-091-0/+27
| | | | | | | | | | | | | | | | | | | | One room at a time so we don't take out the whole server with leave events, and restart at server restart.
| * | | Merge pull request #3199 from matrix-org/erikj/pagination_syncErik Johnston2018-05-091-39/+36
| |\ \ \ | | | | | | | | | | Refactor sync APIs to reuse pagination API
| | * | | Refactor sync APIs to reuse pagination APIErik Johnston2018-05-091-38/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.
| | * | | Don't unnecessarily require token to be stream tokenErik Johnston2018-05-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | This allows calling the `get_recent_event_ids_for_room` function in more situations.
| * | | | Merge pull request #3198 from matrix-org/erikj/fixup_return_paginationErik Johnston2018-05-091-1/+15
| |\ \ \ \ | | | | | | | | | | | | Refactor get_recent_events_for_room return type
| | * | | | Fix up commentErik Johnston2018-05-091-1/+1
| | | | | |
| | * | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
| | |\ \ \ \ | | | |/ / / | | |/| | | | | | | | | erikj/fixup_return_pagination
| | * | | | Refactor get_recent_events_for_room return typeErik Johnston2018-05-091-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs
| * | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
| |\ \ \ \ \ | | | |/ / / | | |/| | | | | | | | | erikj/remove_membership_change
| | * | | | Update commentsErik Johnston2018-05-091-5/+6
| | |/ / /
| * / / / Remove unused code path from member change DB funcErik Johnston2018-05-091-24/+10
| |/ / / | | | | | | | | | | | | | | | | The function is never called without a from_key, so we can remove all the handling for that scenario.
| * | | Refactor pagination DB API to return concrete typeErik Johnston2018-05-091-28/+48
| | | | | | | | | | | | | | | | | | | | This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.
| * | | Remove unused from_token paramErik Johnston2018-05-091-7/+4
| | | |
| * | | Refactor recent events func to use pagination funcErik Johnston2018-05-091-48/+27
| | | | | | | | | | | | | | | | This also removes a cache that is unlikely to ever get hit.
| * | | Fix up comments and make function privateErik Johnston2018-05-091-7/+7
| | | |
| * | | Reuse existing pagination code for context APIErik Johnston2018-05-081-75/+15
| | | |
| * | | Parse tokens before calling DB functionErik Johnston2018-05-081-14/+18
| | | |
| * | | Only fetch required fields from databaseErik Johnston2018-05-081-1/+2
| | | |
| * | | Split paginate_room_events storage functionErik Johnston2018-05-081-28/+72
| |/ /
| * | Merge pull request #3141 from matrix-org/erikj/fixup_stateErik Johnston2018-05-031-37/+67
| |\ \ | | | | | | | | Refactor event storage to prepare for changes in state calculations
| | * | Fix up grammarErik Johnston2018-05-031-3/+3
| | | |
| | * | Refactor event storage to not require stateErik Johnston2018-04-271-37/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.
| * | | Merge pull request #3129 from matrix-org/matthew/fix_group_dupsMatthew Hodgson2018-04-301-0/+57
| |\ \ \ | | | | | | | | | | remove duplicates from groups tables
| | * | | fix missing importMatthew Hodgson2018-04-281-0/+1
| | | | |
| | * | | pep8Matthew Hodgson2018-04-281-3/+5
| | | | |
| | * | | make it work with sqliteMatthew Hodgson2018-04-282-34/+54
| | | | |
| | * | | remove duplicates from groups tablesMatthew Hodgson2018-04-251-0/+34
| | | |/ | | |/| | | | | | | | | | | | | and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
| * | | Merge branch 'develop' into py3-xrange-1Richard van der Hoff2018-04-302-4/+6
| |\ \ \
| | * \ \ Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-272-8/+18
| | |\ \ \
| | * | | | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-272-4/+6
| | | |_|/ | | |/| | | | | | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
| * | | | Move more xrange to sixAdrian Tschira2018-04-284-4/+12
| | |/ / | |/| | | | | | | | | | | | | | | | | | plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptionsRichard van der Hoff2018-04-271-7/+17
| |\ \ \ | | | | | | | | | | Improve exception handling for background processes
| | * | | Improve exception handling for background processesRichard van der Hoff2018-04-271-7/+17
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
| * / / Add missing consumeErrorsRichard van der Hoff2018-04-271-1/+1
| |/ / | | | | | | | | | | | | In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.
| * | Fix media admin APIsErik Johnston2018-04-261-2/+2
| | |
| * | Also fix reindexing of searchErik Johnston2018-04-251-1/+1
| | |
| * | Fix quarantine media admin APIErik Johnston2018-04-251-1/+1
| |/
| * Merge pull request #3110 from NotAFile/py3-six-queueRichard van der Hoff2018-04-201-2/+4
| |\ | | | | | | Replace Queue with six.moves.queue
| | * Replace Queue with six.moves.queueAdrian Tschira2018-04-161-2/+4
| | | | | | | | | | | | | | | | | | and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | Merge pull request #3117 from matrix-org/rav/refactor_have_eventsRichard van der Hoff2018-04-201-7/+42
| |\ \ | | | | | | | | Refactor store.have_events
| | * | Refactor store.have_eventsRichard van der Hoff2018-04-201-7/+42
| | |/ | | | | | | | | | | | | | | | It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.
| * / Avoid creating events with huge numbers of prev_eventsRichard van der Hoff2018-04-161-16/+41
| |/ | | | | | | | | | | In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.
| * Format docstringErik Johnston2018-04-121-2/+3
| |
| * Doc we raise on unknown eventErik Johnston2018-04-121-2/+3
| |
| * Track last processed event received_tsErik Johnston2018-04-111-0/+18
| |
| * Track where event stream processing have gotten up toErik Johnston2018-04-111-0/+3
| |
| * Merge pull request #2760 from Valodim/pypyRichard van der Hoff2018-04-112-1/+6
| |\ | | | | | | Synapse on PyPy
| | * Fix pep8 error on psycopg2cffi hackRichard van der Hoff2018-04-101-6/+5
| | |
| | * Use psycopg2cffi module instead of psycopg2 if running on pypyVincent Breitmoser2018-04-102-2/+8
| | | | | | | | | | | | | | | | | | | | | The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.
| * | fix typoNeil Johnson2018-04-101-2/+2
| | |
| * | remove errant print v0.27.3-rc2Neil Johnson2018-04-091-1/+1
| | |
| * | Fix msec to sec, againNeil Johnson2018-04-091-3/+3
| | |
| * | Fix msec to secNeil Johnson2018-04-091-1/+1
| | |
| * | Fix psycopg2 interpolationNeil Johnson2018-04-091-5/+5
| |/
| * Don't use redundant inlineCallbacksLuke Barnard2018-04-061-4/+1
| |
| * join_rule -> join_policyLuke Barnard2018-04-061-1/+1
| |
| * is_joinable -> join_ruleLuke Barnard2018-04-061-4/+1
| |
| * pep8David Baker2018-04-061-1/+2
| |
| * Implement group join APIDavid Baker2018-04-061-3/+9
| |
| * Merge pull request #3045 from matrix-org/dbkr/group_joinableLuke Barnard2018-04-053-0/+42
| |\ | | | | | | Add joinability for groups
| | * NON NULL -> NOT NULLLuke Barnard2018-04-051-1/+1
| | |
| | * Use DEFAULT join_policy of "invite" in dbLuke Barnard2018-04-051-1/+1
| | |
| | * Document set_group_join_policyLuke Barnard2018-04-051-0/+6
| | |
| | * Use join_policy API instead of joinableLuke Barnard2018-04-032-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".
| | * OK, smallint it is thenDavid Baker2018-03-281-1/+1
| | |
| | * Grr. Copy the definition from is_adminDavid Baker2018-03-281-1/+1
| | |
| | * Make column definition that works on both dbsDavid Baker2018-03-281-1/+1
| | |
| | * Add schema delta fileDavid Baker2018-03-281-0/+16
| | |
| | * Add joinability for groupsDavid Baker2018-03-282-1/+15
| | | | | | | | | | | | | | | Adds API to set the 'joinable' flag, and corresponding flag in the table.
| * | Merge pull request #3041 from matrix-org/r30_statsNeil Johnson2018-04-054-7/+111
| |\ \ | | | | | | | | R30 stats
| | * | Review commentsNeil Johnson2018-04-051-4/+5
| | | | | | | | | | | | | | | | | | | | Use iteritems over item to loop over dict formatting
| | * | pep8Neil Johnson2018-03-291-2/+4
| | | |
| | * | Remove need for sqlite specific queryNeil Johnson2018-03-291-30/+57
| | | |
| | * | fix pep8 errorsNeil Johnson2018-03-281-3/+0
| | | |
| | * | remove twisted deferral cruftNeil Johnson2018-03-281-6/+3
| | | |
| | * | bump schema versionNeil Johnson2018-03-281-1/+1
| | | |
| | * | Support multi client R30 for psqlNeil Johnson2018-03-281-8/+28
| | | |
| | * | Add user_ips last seen indexNeil Johnson2018-03-282-0/+24
| | | |
| | * | No need to cast in count_daily_usersNeil Johnson2018-03-281-2/+2
| | | |
| | * | query and call for r30 statsNeil Johnson2018-03-281-0/+36
| | | |
| | * | count_daily_users failed if db was sqlite due to type failure - presumably ↵Neil Johnson2018-03-281-1/+1
| | |/ | | | | | | | | | this prevcented all sqlite homeservers reporting home
| * | Merge pull request #3060 from matrix-org/rav/kill_event_contentRichard van der Hoff2018-04-053-7/+12
| |\ \ | | | | | | | | Remove uses of events.content
| | * | Remove uses of events.contentRichard van der Hoff2018-03-293-7/+12
| | | |
| * | | Merge pull request #3049 from matrix-org/rav/use_staticjsonRichard van der Hoff2018-04-031-15/+8
| |\ \ \ | | | | | | | | | | Use static JSONEncoders
| | * | | Use static JSONEncodersRichard van der Hoff2018-03-291-15/+8
| | |/ / | | | | | | | | | | | | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.
| * / / Use simplejson throughoutRichard van der Hoff2018-03-291-1/+2
| |/ / | | | | | | | | | Let's use simplejson rather than json, for consistency.
| * | Fix search_user_dir multiple sqlite versions do different thingsNeil Johnson2018-03-281-2/+2
| | |
| * | Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_idErik Johnston2018-03-281-3/+1
| |\ \ | | | | | | | | Linearize calls to _generate_user_id
| | * | Don't use _cursor_to_dict in find_next_generated_user_id_localpartErik Johnston2018-03-261-3/+1
| | |/
| * | Merge pull request #3030 from matrix-org/erikj/no_ujsonErik Johnston2018-03-281-1/+1
| |\ \ | | | | | | | | Remove last usage of ujson
| | * | Remove last usage of ujsonErik Johnston2018-03-261-1/+1
| | |/
| * | CommentErik Johnston2018-03-271-4/+5
| | |
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-17 00:28:44 +0000