summary refs log tree commit diff
path: root/synapse/replication/slave (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-252-3/+14
| | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* Comments from reviewErik Johnston2020-03-181-0/+3
|
* Change device list replication to match new semantics.Erik Johnston2020-02-281-12/+13
| | | | | Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host).
* Add 'device_lists_outbound_pokes' as extra table.Erik Johnston2020-02-281-1/+7
| | | | | | | | | This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change.
* Merge worker apps into one. (#6964)Erik Johnston2020-02-251-0/+20
|
* Allow moving group read APIs to workers (#6866)Erik Johnston2020-02-071-8/+6
|
* Detect unknown remote devices and mark cache as stale (#6776)Erik Johnston2020-01-281-1/+1
| | | | We just mark the fact that the cache may be stale in the database for now.
* Allow streaming cache invalidate all to workers. (#6749)Erik Johnston2020-01-221-1/+6
|
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Fixup synapse.replication to pass mypy checks (#6667)Erik Johnston2020-01-142-4/+5
|
* Change DataStores to accept 'database' param.Erik Johnston2019-12-0613-26/+39
|
* _CURRENT_STATE_CACHE_NAME is publicErik Johnston2019-12-041-2/+2
|
* Move cache invalidation to main data storeErik Johnston2019-12-041-1/+2
|
* document the REPLICATE command a bit better (#6305)Richard van der Hoff2019-11-041-1/+9
| | | | since I found myself wonder how it works
* clean up code a bitHubert Chathi2019-10-311-5/+9
|
* make user signatures a separate streamHubert Chathi2019-10-301-2/+6
|
* make notification of signatures work with workersHubert Chathi2019-10-241-0/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-221-0/+3
|\ | | | | | | erikj/refactor_stores
| * add user signature stream change cache to slaved device storeHubert Chathi2019-09-041-0/+3
| |
* | Move storage classes into a main "data store".Erik Johnston2019-10-2117-27/+29
|/ | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Run Black. (#5482)Amber Brown2019-06-2013-84/+73
|
* Fixup bsaed on review commentsErik Johnston2019-05-171-1/+1
|
* Add basic editing supportErik Johnston2019-05-161-0/+1
|
* Fix relations in worker modeErik Johnston2019-05-161-3/+10
|
* Replace SlavedKeyStore with a shimRichard van der Hoff2019-04-081-14/+4
| | | | | since we're pulling everything out of KeyStore anyway, we may as well simplify it.
* Remove unused server_tls_certificates functions (#5028)Richard van der Hoff2019-04-081-3/+0
| | | | These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.
* Remove presence lists (#4989)Neil Johnson2019-04-031-10/+0
| | | Remove presence list support as per MSC 1819
* Fix sync bug when accepting invites (#4956)Richard van der Hoff2019-04-021-9/+22
| | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.
* Make EventStream rows have a typeRichard van der Hoff2019-03-271-2/+6
| | | | ... as a precursor to combining it with the CurrentStateDelta stream.
* Prefill client IPs cache on workersErik Johnston2019-03-061-0/+2
|
* Fixup slave storesErik Johnston2019-03-043-36/+26
|
* When presence is enabled don't send over replicationErik Johnston2019-02-271-2/+5
|
* Fix state cache invalidation on workersErik Johnston2019-02-221-6/+1
|
* Batch cache invalidation over replicationErik Johnston2019-02-181-7/+12
| | | | | | | | | | Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.
* Make workers work on Py3 (#4027)Amber Brown2018-10-136-30/+30
|
* merge (#3576)Amber Brown2018-09-141-7/+16
|
* Rename slave TransactionStore to SlaveTransactionStoreErik Johnston2018-08-151-1/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-1/+1
|\ | | | | | | erikj/split_federation
| * Basic support for room versioningRichard van der Hoff2018-08-031-1/+1
| | | | | | | | | | | | | | | | This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event
* | Import all functions from TransactionStoreErik Johnston2018-08-061-11/+2
|/
* Use stream cache in get_linearized_receipts_for_roomErik Johnston2018-07-101-1/+1
| | | | | This avoids us from uncessarily hitting the database when there has been no change for the room
* run isortAmber Brown2018-07-0917-28/+41
|
* Add UserErasureWorkerStore to workersErik Johnston2018-06-251-0/+2
|
* Split up ProfileStoreErik Johnston2018-03-131-0/+21
|
* Split Directory storeErik Johnston2018-03-051-5/+3
|
* Split registration storeErik Johnston2018-03-021-15/+3
|
* Merge pull request #2928 from matrix-org/erikj/read_marker_cachesErik Johnston2018-03-011-1/+1
|\ | | | | Fix typo in getting replication account data processing
| * Fix typo in getting replication account data processingErik Johnston2018-03-011-1/+1
| |
* | Merge pull request #2925 from matrix-org/erikj/split_sig_fedErik Johnston2018-03-011-47/+9
|\ \ | | | | | | Split out SignatureStore and EventFederationStore
| * | Remove unused DataStoreErik Johnston2018-03-011-1/+0
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fedErik Johnston2018-03-012-65/+11
| |\ \
| * | | Split out SignatureStore and EventFederationStoreErik Johnston2018-03-011-42/+8
| | | |
* | | | Merge pull request #2927 from matrix-org/erikj/read_marker_cachesErik Johnston2018-03-011-0/+4
|\ \ \ \ | |_|/ / |/| | / | | |/ | |/| Improve caching for read_marker API
| * | Improve caching for read_marker APIErik Johnston2018-03-011-0/+3
| | | | | | | | | | | | | | | | | | We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.
| * | Add some caches to help read marker APIErik Johnston2018-03-011-0/+1
| |/
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-03-012-19/+4
|\ \ | | | | | | | | | erikj/split_stream_store
| * \ Merge pull request #2923 from matrix-org/erikj/stream_ago_workerErik Johnston2018-03-011-1/+0
| |\ \ | | | | | | | | Calculate stream_ordering_month_ago correctly on workers
| | * | Calculate stream_ordering_month_ago correctly on workersErik Johnston2018-03-011-1/+0
| | |/
| * / Split up RoomStoreErik Johnston2018-03-011-17/+4
| |/
* / Split out stream storeErik Johnston2018-03-011-45/+9
|/
* Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidationErik Johnston2018-02-271-0/+2
|\ | | | | Fix missing invalidations for receipt storage
| * Fix missing invalidations for receipt storageErik Johnston2018-02-211-0/+2
| |
* | Merge pull request #2903 from matrix-org/erikj/split_roommember_storeErik Johnston2018-02-271-28/+5
|\ \ | | | | | | Split out RoomMemberStore
| * \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-234-84/+23
| |\ \ | | | | | | | | | | | | erikj/split_roommember_store
| * | | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| * | | Split out RoomMemberStoreErik Johnston2018-02-211-27/+3
| | | |
* | | | Merge pull request #2901 from matrix-org/erikj/split_as_storesErik Johnston2018-02-271-28/+7
|\ \ \ \ | |_|/ / |/| | | Split AS stores
| * | | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| * | | Split AS storesErik Johnston2018-02-211-28/+6
| |/ /
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-02-234-74/+21
|\ \ \ | | | | | | | | | | | | erikj/split_event_push_actions
| * \ \ Merge pull request #2902 from matrix-org/erikj/split_events_storeErik Johnston2018-02-231-11/+3
| |\ \ \ | | | | | | | | | | Split out get_events and co into a worker store
| | * | | Split EventsWorkerStore into separate fileErik Johnston2018-02-231-1/+1
| | | | |
| | * | | Update copyrightErik Johnston2018-02-231-0/+1
| | |/ /
| | * / Split out get_events and co into a worker storeErik Johnston2018-02-211-11/+2
| | |/
| * | Merge pull request #2899 from matrix-org/erikj/split_pushersErik Johnston2018-02-231-9/+3
| |\ \ | | | | | | | | Split PusherStore
| | * | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| | * | Split PusherStoreErik Johnston2018-02-211-9/+2
| | |/
| * | Merge pull request #2898 from matrix-org/erikj/split_push_rules_storeErik Johnston2018-02-231-17/+7
| |\ \ | | | | | | | | Split PushRulesStore
| | * | Update copyrightErik Johnston2018-02-231-0/+1
| | | |
| | * | Split PushRulesStoreErik Johnston2018-02-211-17/+6
| | |/
| * | Update copyrightErik Johnston2018-02-231-0/+1
| | |
| * | Use absolute importsErik Johnston2018-02-231-2/+2
| | |
| * | Split AccountDataStore and TagStoreErik Johnston2018-02-211-34/+4
| |/
* / Split out EventPushActionWorkerStoreErik Johnston2018-02-211-20/+3
|/
* Fix commentErik Johnston2018-02-201-1/+1
|
* Use abstract base class to access stream IDsErik Johnston2018-02-201-2/+7
|
* Split ReceiptsStoreErik Johnston2018-02-201-28/+5
|
* Add event_creator workerErik Johnston2018-02-071-0/+20
|
* Store state groups separately from events (#2784)Erik Johnston2018-02-061-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Pull out bits of StateStore to a mixinRichard van der Hoff2017-11-141-33/+6
| | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.
* Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-131-1/+1
| | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* Fix replication. And notifyErik Johnston2017-07-201-0/+54
|
* Fix upErik Johnston2017-06-271-2/+1
|
* Make workers report to master for user ip updatesErik Johnston2017-06-271-0/+48
|
* Initialise exclusive_user_regexErik Johnston2017-06-211-0/+2
|
* Fix for workersErik Johnston2017-06-211-0/+3
|
* Add cache for is_host_joinedErik Johnston2017-06-131-1/+1
|
* Merge pull request #2266 from matrix-org/erikj/host_in_roomErik Johnston2017-06-121-1/+0
|\ | | | | Change is_host_joined to use current_state table
| * Fix replicationErik Johnston2017-06-091-1/+0
| |
* | Fix replicationErik Johnston2017-06-091-1/+1
|/
* Faster cache for get_joined_hostsErik Johnston2017-05-251-0/+2
|
* Add missing storage function to slave storeErik Johnston2017-05-221-0/+2
|
* Add new storage functions to slave storeErik Johnston2017-05-041-0/+3
|
* Remove unused cacheErik Johnston2017-04-241-3/+0
|
* Move get_presence_list_* to SlaveStoreErik Johnston2017-04-111-0/+10
|
* Reduce federation presence replication trafficErik Johnston2017-04-101-0/+1
| | | | | | | | This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.
* Change slave storage to use new replication interfaceErik Johnston2017-04-0310-171/+106
| | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication.
* Use presence replication stream to invalidate cacheErik Johnston2017-03-241-0/+1
| | | | | | Instead of using the cache invalidation replication stream to invalidate the _get_presence_cache, we can instead rely on the presence replication stream. This reduces the amount of replication traffic considerably.
* Replace some calls to cursor_to_dictErik Johnston2017-03-241-1/+0
| | | | | cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to.
* PEP8Erik Johnston2017-03-201-2/+0
|
* Don't send the full event json over replicationErik Johnston2017-03-171-28/+14
|
* Fix assertion to stop transaction queue getting wedgedRichard van der Hoff2017-03-151-0/+5
| | | | | | | | ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop.
* Add new storage function to slave storeErik Johnston2017-03-131-0/+4
|
* Fix it for the workersErik Johnston2017-03-101-0/+8
|
* Don't log unknown cache warnings in workersErik Johnston2017-02-281-1/+3
|
* Aggregate event push actionsErik Johnston2017-02-141-0/+6
|
* Cache get_presence storageErik Johnston2017-02-131-1/+3
|
* Merge pull request #1912 from matrix-org/markjh/roominitialsyncErik Johnston2017-02-131-0/+6
|\ | | | | Add db functions needed for room initial sync to slave
| * Add db functions needed for room initial sync to slaveMark Haines2017-02-131-0/+6
| |
* | Fix replicationErik Johnston2017-02-021-0/+3
| |
* | Only invalidate membership caches based on the cache streamErik Johnston2017-01-311-16/+5
|/ | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive.
* Hook device list updates to replicationErik Johnston2017-01-271-0/+72
|
* Insert delta of current_state_events to be more efficientErik Johnston2017-01-201-10/+0
|
* Add cache to get_public_room_ids_at_stream_idErik Johnston2016-12-121-1/+2
|
* Invalidate retry cache in both directionsErik Johnston2016-11-222-6/+22
|
* Store federation stream positions in the databaseErik Johnston2016-11-211-0/+3
|
* Handle sending events and device messages over federationErik Johnston2016-11-173-6/+24
|
* Hook up the send queue and create a federation sender workerErik Johnston2016-11-162-0/+8
|
* Fix public room pagination for client_reader appErik Johnston2016-09-161-0/+1
|
* Add new storage function to slave storeErik Johnston2016-09-161-0/+1
|
* Use stream_change cache to make get_forward_extremeties_for_room cache more ↵Erik Johnston2016-09-151-1/+4
| | | | effective
* Add cache to get_forward_extremeties_for_roomErik Johnston2016-09-151-1/+1
|
* Stream public room changes down replicationErik Johnston2016-09-152-0/+39
|
* Track the max_stream_device_id in a separate table, since we delete from the ↵Mark Haines2016-09-091-1/+1
| | | | inbox table
* Merge branch 'develop' into markjh/direct_to_device_federationMark Haines2016-09-081-0/+3
|\
| * Add upgrade script that will slowly prune state_groups_state entriesErik Johnston2016-09-051-0/+3
| |
* | Fix the stream change cache to work over replicationMark Haines2016-09-071-0/+11
|/
* Merge pull request #1060 from matrix-org/erikj/state_idsErik Johnston2016-09-011-0/+1
|\ | | | | Assign state groups in state handler.
| * Add storage function to SlaveStoreErik Johnston2016-08-311-0/+1
| |
* | Add a replication stream for direct to device messagesMark Haines2016-08-311-0/+12
|/
* Merge pull request #1049 from matrix-org/erikj/presence_users_in_roomErik Johnston2016-08-301-2/+5
|\ | | | | Use state handler instead of get_users_in_room/get_joined_hosts
| * Add to slave storeErik Johnston2016-08-301-0/+5
| |
| * Use state handler instead of get_users_in_room/get_joined_hostsErik Johnston2016-08-261-2/+0
| |
* | Add new direct message storage functions to slave storeErik Johnston2016-08-301-0/+30
|/
* Add is_host_joined to slave storageErik Johnston2016-08-261-0/+2
|
* Add new state storage funcs to replicationErik Johnston2016-08-261-0/+3
|
* Add appservice workerErik Johnston2016-08-182-0/+13
|
* Use cached get_user_by_access_token in slavesErik Johnston2016-08-162-2/+2
|
* Rename table. Add docs.Erik Johnston2016-08-151-1/+1
|
* Use cached version of get_aliases_for_roomErik Johnston2016-08-151-1/+1
|
* Implement cache replication streamErik Johnston2016-08-151-1/+29
|
* Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_readerErik Johnston2016-07-291-2/+5
|\
| * Create separate methods for getting messages to pushMark Haines2016-07-281-2/+5
| | | | | | | | | | | | | | | | | | | | | | for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages.
* | Add destination retry to slave storeErik Johnston2016-07-281-0/+30
| |
* | Add slace storage functions for public room listErik Johnston2016-07-282-0/+44
| |
* | Add get_auth_chain to slave storeErik Johnston2016-07-281-0/+4
| |
* | Add more key storage funcs into slave storeErik Johnston2016-07-281-6/+10
| |
* | Split out a FederationReader processErik Johnston2016-07-212-0/+34
|/
* pep8David Baker2016-06-241-1/+0
|
* Use similar naming we use in email notifs for pushDavid Baker2016-06-241-8/+0
| | | | Fixes https://github.com/vector-im/vector-web/issues/1654
* Remove event fetching from DB threadsErik Johnston2016-06-031-5/+0
|
* Merge pull request #827 from matrix-org/markjh/more_slaved_methodsMark Haines2016-06-033-6/+81
|\ | | | | Add methods to events, account data and receipt slaves
| * Add methods to events, account data and receipt slavesMark Haines2016-06-033-6/+81
| | | | | | | | | | Adds the methods needed by /sync to the slaved events, account data and receipt stores.
* | Merge pull request #824 from matrix-org/markjh/slaved_presence_storeMark Haines2016-06-031-0/+59
|\ \ | | | | | | Add a slaved store for presence
| * | Add a slaved store for presenceMark Haines2016-06-031-0/+59
| |/
* | Add a comment explaining why the filter cache doesn't need exipiringMark Haines2016-06-031-0/+1
| |
* | Add slaved stores for filters, tokens, and push rulesMark Haines2016-06-034-0/+151
|/
* Assert that stream replicated stream positions are intsMark Haines2016-05-133-7/+7
|
* Merge branch 'develop' into dbkr/email_notifs_on_pusherMark Haines2016-05-132-0/+63
|\
| * Add a slaved datastore for account dataMark Haines2016-05-131-0/+61
| |
| * Make sure we advance our stream positionMark Haines2016-05-131-0/+2
| |
* | Shift some of the state_group methods into the SlavedEventStoreMark Haines2016-05-131-0/+19
|/
* Pass through get_events to pusher tooDavid Baker2016-05-101-0/+1
|
* Pass through _get_event_txnDavid Baker2016-05-101-0/+1
|
* Fix backfill replication to advance the stream correctlyMark Haines2016-04-271-2/+2
|
* Merge pull request #743 from matrix-org/markjh/slave_pushersMark Haines2016-04-211-0/+52
|\ | | | | Replicate the pushers
| * Replicate the pushersMark Haines2016-04-211-0/+52
| |
* | Replicate push actionsMark Haines2016-04-211-0/+14
|/
* Merge pull request #738 from matrix-org/markjh/slaved_receiptsMark Haines2016-04-191-0/+61
|\ | | | | Add a slaved receipts store
| * Add a slaved receipts storeMark Haines2016-04-191-0/+61
| |
* | Replicate get_invited_rooms_for_userMark Haines2016-04-191-2/+7
|/
* Add tests for redactionsMark Haines2016-04-071-2/+2
|
* Add tests for get_latest_event_ids_in_room and get_current_stateMark Haines2016-04-071-0/+5
|
* Test that room membership is replicatedMark Haines2016-04-061-3/+4
|
* Add a slaved events store classMark Haines2016-04-065-0/+284
Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-06 19:36:31 +0000