summary refs log tree commit diff
path: root/synapse/replication/slave/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Reintroduce #14376, with bugfix for monoliths (#14468)David Robertson2022-11-162-63/+0
| | | | | | | | | | | | | | | | | | | | | | * Add tests for StreamIdGenerator * Drive-by: annotate all defs * Revert "Revert "Remove slaved id tracker (#14376)" (#14463)" This reverts commit d63814fd736fed5d3d45ff3af5e6d3bfae50c439, which in turn reverted 36097e88c4da51fce6556a58c49bd675f4cf20ab. This restores the latter. * Fix StreamIdGenerator not handling unpersisted IDs Spotted by @erikjohnston. Closes #14456. * Changelog Co-authored-by: Nick Mills-Barrett <nick@fizzadar.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Revert "Remove slaved id tracker (#14376)" (#14463)Erik Johnston2022-11-162-0/+63
| | | This reverts commit 36097e88c4da51fce6556a58c49bd675f4cf20ab.
* Remove slaved id tracker (#14376)Nick Mills-Barrett2022-11-142-63/+0
| | | | | This matches the multi instance writer ID generator class which can both handle advancing the current token over replication and by calling the database.
* Merge/remove `Slaved*` stores into `WorkerStores` (#14375)Nick Mills-Barrett2022-11-116-295/+0
|
* Speed up fetching large numbers of push rules (#13592)Erik Johnston2022-08-231-1/+0
|
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-217-142/+0
|
* Use cache store remove base slaved (#13329)Nick Mills-Barrett2022-07-2111-83/+10
| | | This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
* Revert "Make all `process_replication_rows` methods async (#13304)" (#13312)Erik Johnston2022-07-183-12/+6
| | | This reverts commit 5d4028f217f178fcd384d5bfddd92225b4e78c51.
* Make all `process_replication_rows` methods async (#13304)Nick Mills-Barrett2022-07-173-6/+12
| | | | | More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so). Signed off by Nick @ Beeper (@Fizzadar)
* Type annotations in `synapse.databases.main.devices` (#13025)David Robertson2022-06-151-2/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Remove groups replication code. (#12900)Patrick Cloke2022-05-311-58/+0
| | | | The replication logic for groups is no longer used, so the message passing infrastructure can be removed.
* Prefill more stream change caches. (#12372)Erik Johnston2022-04-051-23/+2
|
* Prefill the device_list_stream_cache (#12367)Erik Johnston2022-04-041-1/+11
| | | | | | | * Prefill the device_list_stream_cache * Newsfile * Newsfile
* Track device list updates per room. (#12321)Erik Johnston2022-04-041-0/+1
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Move `update_client_ip` background job from the main process to the ↵reivilibre2022-04-011-59/+0
| | | | background worker. (#12251)
* Rename get_tcp_replication to get_replication_command_handler. (#12192)Patrick Cloke2022-03-101-1/+1
| | | | | | Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.
* Add missing type hints to synapse.replication. (#11938)Patrick Cloke2022-02-086-13/+24
|
* Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff2022-01-211-1/+1
| | | | Preparation for dropping this table altogether. Part of #6574.
* Remove redundant `get_current_events_token` (#11643)Richard van der Hoff2022-01-041-9/+0
| | | | | | | | | | | | | | | | | * Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
* Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-136-12/+42
|
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-262-16/+10
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-226-7/+34
|
* Annotate synapse.storage.util (#10892)David Robertson2021-10-082-5/+9
| | | | | Also mark `synapse.streams` as having has no untyped defs Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Remove the unused public_room_list_stream (#10565)Andrew Morgan2021-08-171-37/+0
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-152-4/+6
|
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-271-1/+1
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-241-1/+1
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-21/+0
|
* Split presence out of master (#9820)Erik Johnston2021-04-231-50/+0
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1421-21/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-1/+1
|
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-183-76/+9
|
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-071-31/+1
|
* Some cleanups to device inbox store. (#9041)Erik Johnston2021-01-071-8/+0
|
* Convert internal pusher dicts to attrs classes. (#8940)Patrick Cloke2020-12-162-10/+27
| | | This improves type hinting and should use less memory.
* Replace DeferredCache with LruCache where possible (#8563)Richard van der Hoff2020-10-191-5/+5
| | | Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
* move DeferredCache into its own moduleRichard van der Hoff2020-10-141-1/+1
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-3/+3
|
* Add some more type annotations to CacheRichard van der Hoff2020-10-141-1/+1
|
* Fix MultiWriteIdGenerator's handling of restarts. (#8374)Erik Johnston2020-09-241-0/+2
| | | | | | | | | | | | | | | | | | | On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-1812-12/+12
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-1/+1
|
* Move and rename `get_devices_with_keys_by_user` (#8204)Richard van der Hoff2020-09-011-0/+3
| | | | | | | | | | | | | | | | | | * Move `get_devices_with_keys_by_user` to `EndToEndKeyWorkerStore` this seems a better fit for it. This commit simply moves the existing code: no other changes at all. * Rename `get_devices_with_keys_by_user` to better reflect what it does. * get_device_stream_token abstract method To avoid referencing fields which are declared in the derived classes, make `get_device_stream_token` abstract, and define that in the classes which define `_device_list_id_gen`.
* Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator ↵Erik Johnston2020-08-2610-13/+13
| | | | (#8171)
* Remove `ChainedIdGenerator`. (#8123)Erik Johnston2020-08-191-6/+4
| | | | | It's just a thin wrapper around two ID gens to make `get_current_token` and `get_next` return tuples. This can easily be replaced by calling the appropriate methods on the underlying ID gens directly.
* Separate `get_current_token` into two. (#8113)Erik Johnston2020-08-191-0/+8
| | | | | | | | | | | | The function is used for two purposes: 1) for subscribers of streams to get a token they can use to get further updates with, and 2) for replication to track position of the writers of the stream. For streams with a single writer the two scenarios produce the same result, however the situation becomes complicated for streams with multiple writers. The current `MultiWriterIdGenerator` does not correctly handle the first case (which is not an issue as its only used for the `caches` stream which nothing subscribes to outside of replication).
* Convert synapse.api to async/await (#8031)Patrick Cloke2020-08-061-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-0519-54/+54
|
* Stop using 'device_max_stream_id' (#7882)Erik Johnston2020-07-171-1/+1
| | | | | It serves no purpose and updating everytime we write to the device inbox stream means all such transactions will conflict, causing lots of transaction failures and retries.
* Use symbolic names for replication stream names (#7768)Richard van der Hoff2020-07-018-17/+17
| | | This makes it much easier to find where streams are referenced.
* Fix bug in account data replication stream. (#7656)Erik Johnston2020-06-091-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | * Ensure account data stream IDs are unique. The account data stream is shared between three tables, and the maximum allocated ID was tracked in a dedicated table. Updating the max ID happened outside the transaction that allocated the ID, leading to a race where if the server was restarted then the same ID could be allocated but the max ID failed to be updated, leading it to be reused. The ID generators have support for tracking across multiple tables, so we may as well use that instead of a dedicated table. * Fix bug in account data replication stream. If the same stream ID was used in both global and room account data then the getting updates for the replication stream would fail due to `heapq.merge(..)` trying to compare a `str` with a `None`. (This is because you'd have two rows like `(534, '!room')` and `(534, None)` from the room and global account data tables). Fix is just to order by stream ID, since we don't rely on the ordering beyond that. The bug where stream IDs can be reused should be fixed now, so this case shouldn't happen going forward. Fixes #7617
* Merge pull request #7519 from matrix-org/rav/kill_py2_codeRichard van der Hoff2020-05-182-13/+4
|\ | | | | Kill off some old python 2 code
| * remove redundant `__func__`Richard van der Hoff2020-05-152-13/+4
| | | | | | | | this is a no-op under python 3
* | Move event stream handling out of slave store. (#7491)Erik Johnston2020-05-152-97/+0
|/ | | | | This allows us to have the logic on both master and workers, which is necessary to move event persistence off master. We also combine the instantiation of ID generators from DataStore and slave stores to the base worker stores. This allows us to select which process writes events independently of the master/worker splits.
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+1
| | | | variables (#6391)
* Support any process writing to cache invalidation stream. (#7436)Erik Johnston2020-05-0711-77/+31
|
* Use `stream.current_token()` and remove `stream_positions()` (#7172)Erik Johnston2020-05-0111-77/+1
| | | | We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
* Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-252-3/+14
| | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* Comments from reviewErik Johnston2020-03-181-0/+3
|
* Change device list replication to match new semantics.Erik Johnston2020-02-281-12/+13
| | | | | Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host).
* Add 'device_lists_outbound_pokes' as extra table.Erik Johnston2020-02-281-1/+7
| | | | | | | | | This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change.
* Merge worker apps into one. (#6964)Erik Johnston2020-02-251-0/+20
|
* Allow moving group read APIs to workers (#6866)Erik Johnston2020-02-071-8/+6
|
* Detect unknown remote devices and mark cache as stale (#6776)Erik Johnston2020-01-281-1/+1
| | | | We just mark the fact that the cache may be stale in the database for now.
* Allow streaming cache invalidate all to workers. (#6749)Erik Johnston2020-01-221-1/+6
|
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Fixup synapse.replication to pass mypy checks (#6667)Erik Johnston2020-01-142-4/+5
|
* Change DataStores to accept 'database' param.Erik Johnston2019-12-0613-26/+39
|
* _CURRENT_STATE_CACHE_NAME is publicErik Johnston2019-12-041-2/+2
|
* Move cache invalidation to main data storeErik Johnston2019-12-041-1/+2
|
* document the REPLICATE command a bit better (#6305)Richard van der Hoff2019-11-041-1/+9
| | | | since I found myself wonder how it works
* clean up code a bitHubert Chathi2019-10-311-5/+9
|
* make user signatures a separate streamHubert Chathi2019-10-301-2/+6
|
* make notification of signatures work with workersHubert Chathi2019-10-241-0/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-221-0/+3
|\ | | | | | | erikj/refactor_stores
| * add user signature stream change cache to slaved device storeHubert Chathi2019-09-041-0/+3
| |
* | Move storage classes into a main "data store".Erik Johnston2019-10-2117-27/+29
|/ | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Run Black. (#5482)Amber Brown2019-06-2013-84/+73
|
* Fixup bsaed on review commentsErik Johnston2019-05-171-1/+1
|
* Add basic editing supportErik Johnston2019-05-161-0/+1
|
* Fix relations in worker modeErik Johnston2019-05-161-3/+10
|
* Replace SlavedKeyStore with a shimRichard van der Hoff2019-04-081-14/+4
| | | | | since we're pulling everything out of KeyStore anyway, we may as well simplify it.
* Remove unused server_tls_certificates functions (#5028)Richard van der Hoff2019-04-081-3/+0
| | | | These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.
* Remove presence lists (#4989)Neil Johnson2019-04-031-10/+0
| | | Remove presence list support as per MSC 1819
* Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-021-9/+22
| | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
* Make EventStream rows have a typeRichard van der Hoff2019-03-271-2/+6
| | | | ... as a precursor to combining it with the CurrentStateDelta stream.
* Prefill client IPs cache on workersErik Johnston2019-03-061-0/+2
|
* Fixup slave storesErik Johnston2019-03-043-36/+26
|
* When presence is enabled don't send over replicationErik Johnston2019-02-271-2/+5
|
* Fix state cache invalidation on workersErik Johnston2019-02-221-6/+1
|
* Batch cache invalidation over replicationErik Johnston2019-02-181-7/+12
| | | | | | | | | | Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.
* Make workers work on Py3 (#4027)Amber Brown2018-10-136-30/+30
|
* merge (#3576)Amber Brown2018-09-141-7/+16
|
* Rename slave TransactionStore to SlaveTransactionStoreErik Johnston2018-08-151-1/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-1/+1
|\ | | | | | | erikj/split_federation
| * Basic support for room versioningRichard van der Hoff2018-08-031-1/+1
| | | | | | | | | | | | | | | | This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event
* | Import all functions from TransactionStoreErik Johnston2018-08-061-11/+2
|/
* Use stream cache in get_linearized_receipts_for_roomErik Johnston2018-07-101-1/+1
| | | | | This avoids us from uncessarily hitting the database when there has been no change for the room
* run isortAmber Brown2018-07-0917-28/+41
|
* Add UserErasureWorkerStore to workersErik Johnston2018-06-251-0/+2
|
* Split up ProfileStoreErik Johnston2018-03-131-0/+21
|
* Split Directory storeErik Johnston2018-03-051-5/+3
|
* Split registration storeErik Johnston2018-03-021-15/+3
|
* Merge pull request #2928 from matrix-org/erikj/read_marker_cachesErik Johnston2018-03-011-1/+1
|\ | | | | Fix typo in getting replication account data processing
| * Fix typo in getting replication account data processingErik Johnston2018-03-011-1/+1
| |
* | Merge pull request #2925 from matrix-org/erikj/split_sig_fedErik Johnston2018-03-011-47/+9
|\ \ | | | | | | Split out SignatureStore and EventFederationStore
| * | Remove unused DataStoreErik Johnston2018-03-011-1/+0
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fedErik Johnston2018-03-012-65/+11
| |\ \
| * | | Split out SignatureStore and EventFederationStoreErik Johnston2018-03-011-42/+8
| | | |
* | | | Merge pull request #2927 from matrix-org/erikj/read_marker_cachesErik Johnston2018-03-011-0/+4
|\ \ \ \ | |_|/ / |/| | / | | |/ | |/| Improve caching for read_marker API
| * | Improve caching for read_marker APIErik Johnston2018-03-011-0/+3
| | | | | | | | | | | | | | | | | | We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.
| * | Add some caches to help read marker APIErik Johnston2018-03-011-0/+1
| |/
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-03-012-19/+4
|\ \ | | | | | | | | | erikj/split_stream_store
| * \ Merge pull request #2923 from matrix-org/erikj/stream_ago_workerErik Johnston2018-03-011-1/+0
| |\ \ | | | | | | | | Calculate stream_ordering_month_ago correctly on workers
| | * | Calculate stream_ordering_month_ago correctly on workersErik Johnston2018-03-011-1/+0
| | |/
| * / Split up RoomStoreErik Johnston2018-03-011-17/+4
| |/
* / Split out stream storeErik Johnston2018-03-011-45/+9
|/
* Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidationErik Johnston2018-02-271-0/+2
|\ | | | | Fix missing invalidations for receipt storage
| * Fix missing invalidations for receipt storageErik Johnston2018-02-211-0/+2
| |
* | Merge pull request #2903 from matrix-org/erikj/split_roommember_storeErik Johnston2018-02-271-28/+5
|\ \ | | | | | | Split out RoomMemberStore
| * \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-234-84/+23
| |\ \ | | | | | | | | | | | | erikj/split_roommember_store
| * | | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| * | | Split out RoomMemberStoreErik Johnston2018-02-211-27/+3
| | | |
* | | | Merge pull request #2901 from matrix-org/erikj/split_as_storesErik Johnston2018-02-271-28/+7
|\ \ \ \ | |_|/ / |/| | | Split AS stores
| * | | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| * | | Split AS storesErik Johnston2018-02-211-28/+6
| |/ /
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-234-74/+21
|\ \ \ | | | | | | | | | | | | erikj/split_event_push_actions
| * \ \ Merge pull request #2902 from matrix-org/erikj/split_events_storeErik Johnston2018-02-231-11/+3
| |\ \ \ | | | | | | | | | | Split out get_events and co into a worker store
| | * | | Split EventsWorkerStore into separate fileErik Johnston2018-02-231-1/+1
| | | | |
| | * | | Update copyrightErik Johnston2018-02-231-0/+1
| | |/ /
| | * / Split out get_events and co into a worker storeErik Johnston2018-02-211-11/+2
| | |/
| * | Merge pull request #2899 from matrix-org/erikj/split_pushersErik Johnston2018-02-231-9/+3
| |\ \ | | | | | | | | Split PusherStore
| | * | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| | * | Split PusherStoreErik Johnston2018-02-211-9/+2
| | |/
| * | Merge pull request #2898 from matrix-org/erikj/split_push_rules_storeErik Johnston2018-02-231-17/+7
| |\ \ | | | | | | | | Split PushRulesStore
| | * | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| | * | Split PushRulesStoreErik Johnston2018-02-211-17/+6
| | |/
| * | Update copyrightErik Johnston2018-02-231-0/+1
| | |
| * | Use absolute importsErik Johnston2018-02-231-2/+2
| | |
| * | Split AccountDataStore and TagStoreErik Johnston2018-02-211-34/+4
| |/
* / Split out EventPushActionWorkerStoreErik Johnston2018-02-211-20/+3
|/
* Fix commentErik Johnston2018-02-201-1/+1
|
* Use abstract base class to access stream IDsErik Johnston2018-02-201-2/+7
|
* Split ReceiptsStoreErik Johnston2018-02-201-28/+5
|
* Add event_creator workerErik Johnston2018-02-071-0/+20
|
* Store state groups separately from events (#2784)Erik Johnston2018-02-061-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Pull out bits of StateStore to a mixinRichard van der Hoff2017-11-141-33/+6
| | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.
* Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-131-1/+1
| | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* Fix replication. And notifyErik Johnston2017-07-201-0/+54
|
* Fix upErik Johnston2017-06-271-2/+1
|
* Make workers report to master for user ip updatesErik Johnston2017-06-271-0/+48
|
* Initialise exclusive_user_regexErik Johnston2017-06-211-0/+2
|
* Fix for workersErik Johnston2017-06-211-0/+3
|
* Add cache for is_host_joinedErik Johnston2017-06-131-1/+1
|
* Merge pull request #2266 from matrix-org/erikj/host_in_roomErik Johnston2017-06-121-1/+0
|\ | | | | Change is_host_joined to use current_state table
| * Fix replicationErik Johnston2017-06-091-1/+0
| |
* | Fix replicationErik Johnston2017-06-091-1/+1
|/
* Faster cache for get_joined_hostsErik Johnston2017-05-251-0/+2
|
* Add missing storage function to slave storeErik Johnston2017-05-221-0/+2
|
* Add new storage functions to slave storeErik Johnston2017-05-041-0/+3
|
* Remove unused cacheErik Johnston2017-04-241-3/+0
|
* Move get_presence_list_* to SlaveStoreErik Johnston2017-04-111-0/+10
|
* Reduce federation presence replication trafficErik Johnston2017-04-101-0/+1
| | | | | | | | This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.
* Change slave storage to use new replication interfaceErik Johnston2017-04-0310-171/+106
| | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication.
* Use presence replication stream to invalidate cacheErik Johnston2017-03-241-0/+1
| | | | | | Instead of using the cache invalidation replication stream to invalidate the _get_presence_cache, we can instead rely on the presence replication stream. This reduces the amount of replication traffic considerably.
* Replace some calls to cursor_to_dictErik Johnston2017-03-241-1/+0
| | | | | cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to.
* PEP8Erik Johnston2017-03-201-2/+0
|
* Don't send the full event json over replicationErik Johnston2017-03-171-28/+14
|
* Fix assertion to stop transaction queue getting wedgedRichard van der Hoff2017-03-151-0/+5
| | | | | | | | ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop.
* Add new storage function to slave storeErik Johnston2017-03-131-0/+4
|
* Fix it for the workersErik Johnston2017-03-101-0/+8
|
* Don't log unknown cache warnings in workersErik Johnston2017-02-281-1/+3
|
* Aggregate event push actionsErik Johnston2017-02-141-0/+6
|
* Cache get_presence storageErik Johnston2017-02-131-1/+3
|
* Merge pull request #1912 from matrix-org/markjh/roominitialsyncErik Johnston2017-02-131-0/+6
|\ | | | | Add db functions needed for room initial sync to slave
| * Add db functions needed for room initial sync to slaveMark Haines2017-02-131-0/+6
| |
* | Fix replicationErik Johnston2017-02-021-0/+3
| |
* | Only invalidate membership caches based on the cache streamErik Johnston2017-01-311-16/+5
|/ | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive.
* Hook device list updates to replicationErik Johnston2017-01-271-0/+72
|
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-10/+0
|
* Add cache to get_public_room_ids_at_stream_idErik Johnston2016-12-121-1/+2
|
* Invalidate retry cache in both directionsErik Johnston2016-11-222-6/+22
|
* Store federation stream positions in the databaseErik Johnston2016-11-211-0/+3
|
* Handle sending events and device messages over federationErik Johnston2016-11-173-6/+24
|
* Hook up the send queue and create a federation sender workerErik Johnston2016-11-162-0/+8
|
* Fix public room pagination for client_reader appErik Johnston2016-09-161-0/+1
|
* Add new storage function to slave storeErik Johnston2016-09-161-0/+1
|
* Use stream_change cache to make get_forward_extremeties_for_room cache more ↵Erik Johnston2016-09-151-1/+4
| | | | effective
* Add cache to get_forward_extremeties_for_roomErik Johnston2016-09-151-1/+1
|
* Stream public room changes down replicationErik Johnston2016-09-152-0/+39
|
* Track the max_stream_device_id in a separate table, since we delete from the ↵Mark Haines2016-09-091-1/+1
| | | | inbox table
* Merge branch 'develop' into markjh/direct_to_device_federationMark Haines2016-09-081-0/+3
|\
| * Add upgrade script that will slowly prune state_groups_state entriesErik Johnston2016-09-051-0/+3
| |
* | Fix the stream change cache to work over replicationMark Haines2016-09-071-0/+11
|/
* Merge pull request #1060 from matrix-org/erikj/state_idsErik Johnston2016-09-011-0/+1
|\ | | | | Assign state groups in state handler.
| * Add storage function to SlaveStoreErik Johnston2016-08-311-0/+1
| |
* | Add a replication stream for direct to device messagesMark Haines2016-08-311-0/+12
|/
* Merge pull request #1049 from matrix-org/erikj/presence_users_in_roomErik Johnston2016-08-301-2/+5
|\ | | | | Use state handler instead of get_users_in_room/get_joined_hosts
| * Add to slave storeErik Johnston2016-08-301-0/+5
| |
| * Use state handler instead of get_users_in_room/get_joined_hostsErik Johnston2016-08-261-2/+0
| |
* | Add new direct message storage functions to slave storeErik Johnston2016-08-301-0/+30
|/
* Add is_host_joined to slave storageErik Johnston2016-08-261-0/+2
|
* Add new state storage funcs to replicationErik Johnston2016-08-261-0/+3
|
* Add appservice workerErik Johnston2016-08-182-0/+13
|
* Use cached get_user_by_access_token in slavesErik Johnston2016-08-162-2/+2
|
* Rename table. Add docs.Erik Johnston2016-08-151-1/+1
|
* Use cached version of get_aliases_for_roomErik Johnston2016-08-151-1/+1
|
* Implement cache replication streamErik Johnston2016-08-151-1/+29
|
* Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_readerErik Johnston2016-07-291-2/+5
|\
| * Create separate methods for getting messages to pushMark Haines2016-07-281-2/+5
| | | | | | | | | | | | | | | | | | | | | | for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages.
* | Add destination retry to slave storeErik Johnston2016-07-281-0/+30
| |
* | Add slace storage functions for public room listErik Johnston2016-07-282-0/+44
| |
* | Add get_auth_chain to slave storeErik Johnston2016-07-281-0/+4
| |
* | Add more key storage funcs into slave storeErik Johnston2016-07-281-6/+10
| |
* | Split out a FederationReader processErik Johnston2016-07-212-0/+34
|/
* pep8David Baker2016-06-241-1/+0
|
* Use similar naming we use in email notifs for pushDavid Baker2016-06-241-8/+0
| | | | Fixes https://github.com/vector-im/vector-web/issues/1654
* Remove event fetching from DB threadsErik Johnston2016-06-031-5/+0
|
* Merge pull request #827 from matrix-org/markjh/more_slaved_methodsMark Haines2016-06-033-6/+81
|\ | | | | Add methods to events, account data and receipt slaves
| * Add methods to events, account data and receipt slavesMark Haines2016-06-033-6/+81
| | | | | | | | | | Adds the methods needed by /sync to the slaved events, account data and receipt stores.
* | Merge pull request #824 from matrix-org/markjh/slaved_presence_storeMark Haines2016-06-031-0/+59
|\ \ | | | | | | Add a slaved store for presence
| * | Add a slaved store for presenceMark Haines2016-06-031-0/+59
| |/
* | Add a comment explaining why the filter cache doesn't need exipiringMark Haines2016-06-031-0/+1
| |
* | Add slaved stores for filters, tokens, and push rulesMark Haines2016-06-034-0/+151
|/
* Assert that stream replicated stream positions are intsMark Haines2016-05-133-7/+7
|
* Merge branch 'develop' into dbkr/email_notifs_on_pusherMark Haines2016-05-132-0/+63
|\
| * Add a slaved datastore for account dataMark Haines2016-05-131-0/+61
| |
| * Make sure we advance our stream positionMark Haines2016-05-131-0/+2
| |
* | Shift some of the state_group methods into the SlavedEventStoreMark Haines2016-05-131-0/+19
|/
* Pass through get_events to pusher tooDavid Baker2016-05-101-0/+1
|
* Pass through _get_event_txnDavid Baker2016-05-101-0/+1
|
* Fix backfill replication to advance the stream correctlyMark Haines2016-04-271-2/+2
|
* Merge pull request #743 from matrix-org/markjh/slave_pushersMark Haines2016-04-211-0/+52
|\ | | | | Replicate the pushers
| * Replicate the pushersMark Haines2016-04-211-0/+52
| |
* | Replicate push actionsMark Haines2016-04-211-0/+14
|/
* Merge pull request #738 from matrix-org/markjh/slaved_receiptsMark Haines2016-04-191-0/+61
|\ | | | | Add a slaved receipts store
| * Add a slaved receipts storeMark Haines2016-04-191-0/+61
| |
* | Replicate get_invited_rooms_for_userMark Haines2016-04-191-2/+7
|/
* Add tests for redactionsMark Haines2016-04-071-2/+2
|
* Add tests for get_latest_event_ids_in_room and get_current_stateMark Haines2016-04-071-0/+5
|
* Test that room membership is replicatedMark Haines2016-04-061-3/+4
|
* Add a slaved events store classMark Haines2016-04-064-0/+270
Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-10 09:56:47 +0000