summary refs log tree commit diff
path: root/scripts-dev (unfollow)
Commit message (Collapse)AuthorFilesLines
2018-03-13Add is_blocked to worker storeErik Johnston1-13/+17
2018-03-13Implement RoomMemberWorkerHandlerErik Johnston4-2/+418
2018-03-13Raise, don't return, exceptionErik Johnston1-1/+1
2018-03-13Add missing param to docstringsErik Johnston1-0/+3
2018-03-13Correct import orderErik Johnston1-3/+3
2018-03-13Move user_*_room distributor stuff to master classErik Johnston1-5/+50
I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.
2018-03-13Split RoomMemberHandler into base and master classErik Johnston2-98/+139
The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.
2018-03-13s/replication_client/federation_client/Erik Johnston17-29/+29
2018-03-13s/replication_server/federation_serverErik Johnston3-4/+4
2018-03-13Add docstringErik Johnston1-0/+26
2018-03-13RoomMembershipRestServlet doesn't handle /forgetErik Johnston1-1/+1
Due to the order we register the REST handlers `/forget` was handled by the correct handler.
2018-03-13Refactor get_or_register_3pid_guestErik Johnston2-11/+25
2018-03-13Fix docstring typesErik Johnston1-2/+2
2018-03-13Factor out _remote_reject_invite in RoomMemberErik Johnston1-22/+28
2018-03-13extra_users is actually a list of UserIDsErik Johnston2-5/+5
2018-03-13Make functions private that can beErik Johnston1-7/+7
2018-03-13Don't build handlers on workers unnecessarilyErik Johnston5-5/+0
2018-03-13Remove unused ReplicationLayerErik Johnston2-59/+0
2018-03-13Fix testsErik Johnston10-21/+22
2018-03-13Split replication layer into twoErik Johnston12-27/+19
2018-03-13Move property setting from ReplicationLayer to FederationBaseErik Johnston4-22/+13
2018-03-13Fix testsErik Johnston2-10/+8
2018-03-13Split out edu/query registration to a separate classErik Johnston10-60/+90
2018-03-13Add Measure block for persist_eventsRichard van der Hoff1-4/+5
This seems like a useful thing to measure.
2018-03-12Add transactional API to history purgeRichard van der Hoff3-8/+161
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12Return an error when doing two purges on a roomRichard van der Hoff1-3/+14
Queuing up purges doesn't sound like a good thing.
2018-03-09Add some docstrings to help figure this outRichard van der Hoff1-2/+26
2018-03-09Add a metric which increments when a request is receivedRichard van der Hoff2-2/+26
It's useful to know when there are peaks in incoming requests - which isn't quite the same as there being peaks in outgoing responses, due to the time taken to handle requests.
2018-03-09refactor JsonResourceRichard van der Hoff1-32/+46
rephrase the OPTIONS and unrecognised request handling so that they look similar to the common flow.
2018-03-08Factor run_in_background out from preserve_fnRichard van der Hoff2-28/+33
It annoys me that we create temporary function objects when there's really no need for it. Let's factor the gubbins out of preserve_fn and start using it.
2018-03-07Fix up log messageErik Johnston1-1/+1
2018-03-07Fix typoErik Johnston1-2/+2
2018-03-07Fix race in sync when joining roomErik Johnston3-30/+102
The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.
2018-03-05use bcrypt.checkpwKrombel2-3/+5
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago) This makes use of that with enhancements which might get introduced by that Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05Remove ability for AS users to call /events and /syncErik Johnston4-163/+14
This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05Fix cache invalidation on deletionErik Johnston1-1/+4
2018-03-05Split Directory storeErik Johnston2-27/+26
2018-03-05Provide a means to pass a timestamp to purge_historyRichard van der Hoff4-14/+96
2018-03-05Fix comment typoRichard van der Hoff1-1/+1
2018-03-05Test and fix find_first_stream_ordering_after_tsRichard van der Hoff2-15/+120
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05Add find_first_stream_ordering_after_tsRichard van der Hoff1-0/+21
Expose this as a public function which can be called outside a txn
2018-03-02Fix bug with delayed cache invalidation streamErik Johnston2-13/+15
We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.
2018-03-02Split registration storeErik Johnston2-72/+64
2018-03-02Add missing yield during 3pid signature checksErik Johnston1-1/+1
2018-03-01Fix typo in getting replication account data processingErik Johnston1-1/+1
2018-03-01Remove unused DataStoreErik Johnston1-1/+0
2018-03-01Fewer lies are betterErik Johnston1-2/+2
2018-03-01Improve caching for read_marker APIErik Johnston3-3/+41
We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.
2018-03-01Add some caches to help read marker APIErik Johnston3-2/+5
2018-03-01Move back to hs.is_mineErik Johnston1-7/+6
2018-03-01Stub out broken function only used for cacheErik Johnston1-1/+3
2018-03-01Remove unused variablesErik Johnston1-8/+1
2018-03-01Document abstract class and method betterErik Johnston1-8/+13
2018-03-01Default stream_ordering_*_ago to NoneErik Johnston1-2/+2
2018-03-01Fix comment typoErik Johnston1-1/+1
2018-03-01Split out stream storeErik Johnston3-210/+202
2018-03-01Move RoomMemberHandler out of HandlersErik Johnston11-45/+58
2018-03-01Calculate stream_ordering_month_ago correctly on workersErik Johnston3-80/+85
2018-03-01Split out SignatureStore and EventFederationStoreErik Johnston3-174/+148
2018-03-01Split up RoomStoreErik Johnston2-135/+125
2018-03-01Rewrite make_deferred_yieldable avoiding inlineCallbacksRichard van der Hoff1-9/+11
... because (a) it's actually simpler (b) it might be marginally more performant?
2018-03-01Log in the correct placesErik Johnston1-2/+4
2018-03-01Don't do preserve_fn for every requestErik Johnston1-1/+2
2018-03-01Add some loggingErik Johnston1-0/+2
2018-03-01Make repl send_event idempotent and retry on timeoutsErik Johnston1-6/+38
If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event.
2018-03-01Check event auth on the workerErik Johnston1-16/+16
2018-03-01Correctly send ratelimit and extra_users paramsErik Johnston2-1/+15
2018-03-01Fixup commentsErik Johnston1-4/+4
2018-02-28Calculate push actions on workerErik Johnston3-26/+70
2018-02-27Move storage functions for push calculationsErik Johnston6-98/+101
This will allow push actions for an event to be calculated on workers.
2018-02-27Add commentErik Johnston1-0/+4
2018-02-23Update copyrightErik Johnston2-0/+2
2018-02-23Update copyrightErik Johnston3-0/+3
2018-02-23Split EventsWorkerStore into separate fileErik Johnston3-361/+401
2018-02-23Update copyrightErik Johnston2-0/+2
2018-02-23Remove redundant clockErik Johnston1-3/+0
2018-02-23_event_persist_queue shouldn't be in worker storeErik Johnston1-4/+4
2018-02-23Update copyrightErik Johnston2-0/+2
2018-02-23Update copyrightErik Johnston2-0/+2
2018-02-23Update copyrightErik Johnston3-0/+3
2018-02-23Update copyrightErik Johnston3-0/+3
2018-02-23Use absolute importsErik Johnston3-5/+5
2018-02-21Fix missing invalidations for receipt storageErik Johnston2-14/+16
2018-02-21Split out RoomMemberStoreErik Johnston2-206/+184
2018-02-21Split AS storesErik Johnston2-37/+23
2018-02-21Split out get_events and co into a worker storeErik Johnston2-356/+352
2018-02-21Update commentsErik Johnston1-3/+4
2018-02-21Actually use new paramErik Johnston1-1/+3
2018-02-21Split out EventPushActionWorkerStoreErik Johnston2-91/+76
2018-02-21Split PusherStoreErik Johnston2-13/+8
2018-02-21Split PushRulesStoreErik Johnston3-45/+61
2018-02-21Split AccountDataStore and TagStoreErik Johnston4-67/+69
2018-02-20Raise exception in abstract methodErik Johnston1-1/+1
2018-02-20Fix commentErik Johnston2-2/+2
2018-02-20Use abstract base class to access stream IDsErik Johnston2-17/+34
2018-02-20Split ReceiptsStoreErik Johnston3-76/+69
2018-02-20Fix testErik Johnston1-1/+1
2018-02-20Ensure all push actions are deleted from stagingErik Johnston2-2/+19
2018-02-20Refactor _set_push_actions_for_event_and_users_txn to use events_and_contextsErik Johnston2-33/+41
2018-02-20Fix unit testsErik Johnston2-5/+7
2018-02-20Delete from push_actions_staging in federation tooErik Johnston1-9/+17
2018-02-20Batch inserts into event_push_actions_stagingErik Johnston2-23/+45
2018-02-19Update pynacl dependency to 1.2.1 or higherPascal Bach1-1/+1
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-16(Really) fix tablescan of event_push_actions on purgeRichard van der Hoff1-1/+0
commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/
2018-02-16Fix typo of double is_highlightErik Johnston1-1/+1
2018-02-16CommentsErik Johnston3-1/+9
2018-02-15Fix unit testErik Johnston1-4/+6
2018-02-15Ensure that we delete staging push actions on errorsErik Johnston2-3/+25
2018-02-15Remove context.push_actionsErik Johnston5-20/+11
2018-02-15Update event_push_actions table from staging tableErik Johnston2-22/+39
2018-02-15Store push actions in staging areaErik Johnston3-0/+54
2018-02-15Don't serialize current state over replicationErik Johnston3-8/+51
2018-02-15Fix state group storage bug in workersErik Johnston1-41/+41
We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.
2018-02-15Don't log errors propogated from send_eventErik Johnston1-1/+10
2018-02-14Make store_file use store_into_fileErik Johnston1-26/+11
2018-02-14purge_history: fix sqlite syntax errorRichard van der Hoff1-1/+4
apparently sqlite insists on indexes being named
2018-02-14purge_history: handle sqlite asshatteryRichard van der Hoff1-19/+27
apparently creating a temporary table commits the transaction. because that's a useful thing.
2018-02-14remove overzealous exception handlingRichard van der Hoff1-18/+10
2018-02-14purge_history: fix index useRichard van der Hoff1-0/+14
event_push_actions doesn't have an index on event_id, so we need to specify room_id.
2018-02-14Rework event purge to use a temporary tableRichard van der Hoff1-35/+58
... which should speed things up by reducing the amount of data being shuffled across the connection
2018-02-13Update docsErik Johnston1-1/+4
2018-02-13Fix log message in purge_historyRichard van der Hoff1-2/+1
(we don't just remove remote events)
2018-02-13move search reindex to schema 47Richard van der Hoff2-1/+1
We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.
2018-02-13GIN reindex: Fix syntax errors, improve exception handlingRichard van der Hoff1-13/+27
2018-02-13style nitRichard van der Hoff1-1/+1
2018-02-13Fix typos in purge api & docRichard van der Hoff2-4/+2
* It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json
2018-02-12Update workers docs to include http portErik Johnston1-5/+20
2018-02-09purge: move room_depth update to endRichard van der Hoff1-6/+12
... to avoid locking the table for too long
2018-02-09delete_local_events for purge_historyRichard van der Hoff5-15/+67
Add a flag which makes the purger delete local events
2018-02-09purge: Move cache invalidation to more appropriate placeRichard van der Hoff1-4/+4
it was a bit of a non-sequitur there
2018-02-09bump purge logging to infoRichard van der Hoff1-13/+14
this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.
2018-02-09rename delete_old_state -> purge_historyRichard van der Hoff2-8/+8
(beacause it deletes more than state)
2018-02-07Tell storage providers about new file so they can uploadErik Johnston1-0/+6
2018-02-07Add note in docs/workers.rstErik Johnston1-0/+11
2018-02-07Move presence handling into handle_new_client_eventErik Johnston1-6/+6
As we want to have it run on the main synapse instance
2018-02-07Add event_creator workerErik Johnston3-1/+190
2018-02-07Add replication http endpoint for event sendingErik Johnston8-13/+304
2018-02-07Remove pointless ratelimit checkErik Johnston1-5/+0
The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.
2018-02-06Update copyrightErik Johnston6-1/+6
2018-02-06s/_create_new_client_event/create_new_client_event/Erik Johnston4-12/+12
2018-02-06Store state groups separately from events (#2784)Erik Johnston12-189/+326
* Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
2018-02-05Use StateResolutionHandler to resolve state in persist eventsRichard van der Hoff1-48/+24
... and thus benefit (hopefully) from its cache.
2018-02-05Flatten _get_new_state_after_eventsRichard van der Hoff1-44/+46
rejig the if statements to simplify the logic and reduce indentation
2018-02-05Check that events being persisted have state_groupRichard van der Hoff1-4/+9
2018-02-05Add event_map param to resolve_state_groupsRichard van der Hoff2-4/+31
2018-02-05Remove redundant return value from _calculate_state_deltaRichard van der Hoff1-10/+23
we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.
2018-02-05Factor out common code for search insertRichard van der Hoff1-33/+62
we can reuse the same code as is used for event insert, for doing the background index population.
2018-02-05Updates testsErik Johnston2-8/+6
2018-02-05Update places where we create eventsErik Johnston7-38/+42
2018-02-05Split event creation into a separate handlerErik Johnston1-146/+160
2018-02-05Move store_event_search_txn to SearchStoreRichard van der Hoff2-35/+41
... as a precursor to making event storing and doing the bg update share some code.
2018-02-05report metrics on number of cache evictionsRichard van der Hoff5-5/+56
2018-02-05Fix broken unit test for media storageErik Johnston1-1/+6
2018-02-05Add .vscode to gitignoreErik Johnston1-0/+2
2018-02-04Factor out common code for search insertRichard van der Hoff1-33/+56
we can reuse the same code as is used for event insert, for doing the background index population.
2018-02-03Clean up work_mem handlingRichard van der Hoff1-11/+41
Add some comments and improve exception handling when twiddling work_mem for the search update
2018-02-03Move store_event_search_txn to SearchStoreRichard van der Hoff2-37/+43
... as a precursor to making event storing and doing the bg update share some code.
2018-02-02Reinstate event_search_postgres_gist handlerRichard van der Hoff3-6/+31
People may have queued updates for this, so we can't just delete it.
2018-02-02Handle url_previews with no content-typeRichard van der Hoff1-21/+34
avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.
2018-02-01doc arg types for _seperateRichard van der Hoff1-0/+15
2018-02-01More docstring fixesRichard van der Hoff1-4/+4
Fix a couple of errors in docstrings
2018-02-01Fix docstring for StateHandler.resolve_state_groupsRichard van der Hoff1-4/+9
The return type was a complete lie, so fix it
2018-02-01Factor out resolve_state_groups to a separate handlerRichard van der Hoff4-54/+108
We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.
2018-02-01Rename resolve_state_groups -> resolve_state_groups_for_eventsRichard van der Hoff2-12/+11
(to make way for a method that actually just does the state group resolution)
2018-01-31pep8Travis Ralston1-9/+12
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31