Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Comments | Erik Johnston | 2017-04-10 | 1 | -2/+2 |
| | |||||
* | Merge pull request #2103 from matrix-org/erikj/no-double-encode | Erik Johnston | 2017-04-07 | 1 | -28/+76 |
|\ | | | | | Don't double encode replication data | ||||
| * | Document types of the replication streams | Erik Johnston | 2017-04-06 | 1 | -28/+76 |
| | | |||||
* | | Fix incorrect type when using InvalidateCacheCommand | Erik Johnston | 2017-04-06 | 1 | -1/+1 |
| | | |||||
* | | Add log lines | Erik Johnston | 2017-04-05 | 1 | -1/+2 |
| | | |||||
* | | Rearrange metrics | Erik Johnston | 2017-04-05 | 1 | -16/+31 |
| | | |||||
* | | Fix typo | Erik Johnston | 2017-04-05 | 1 | -2/+2 |
| | | |||||
* | | Fixup some metrics for tcp repl | Erik Johnston | 2017-04-05 | 1 | -0/+16 |
|/ | |||||
* | Merge pull request #2097 from matrix-org/erikj/repl_tcp_client | Erik Johnston | 2017-04-05 | 11 | -171/+302 |
|\ | | | | | Move to using TCP replication | ||||
| * | Change slave storage to use new replication interface | Erik Johnston | 2017-04-03 | 10 | -171/+106 |
| | | | | | | | | | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication. | ||||
| * | Add basic replication client handler and factory | Erik Johnston | 2017-04-03 | 1 | -0/+196 |
| | | |||||
* | | Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix | Erik Johnston | 2017-04-04 | 3 | -6/+15 |
|\ \ | | | | | | | Advance replication streams even if nothing is listening | ||||
| * | | Advance replication streams even if nothing is listening | Erik Johnston | 2017-04-04 | 3 | -6/+15 |
| |/ | | | | | | | | | | | Otherwise the streams don't advance and steadily fall behind, so when a worker does connect either a) they'll be streamed lots of old updates or b) the connection will fail as the streams are too far behind. | ||||
* / | Fiddle tcp replication logging | Erik Johnston | 2017-04-04 | 1 | -2/+2 |
|/ | |||||
* | Always advance stream tokens | Erik Johnston | 2017-04-03 | 1 | -1/+4 |
| | |||||
* | Use callbacks to notify tcp replication rather than deferreds | Erik Johnston | 2017-03-31 | 1 | -14/+1 |
| | |||||
* | Add a timestamp to USER_SYNC command | Erik Johnston | 2017-03-31 | 3 | -9/+17 |
| | | | | This timestamp is used to indicate when the user last sync'd | ||||
* | Fix up docs | Erik Johnston | 2017-03-31 | 2 | -19/+3 |
| | |||||
* | Add server side resource for tcp replication | Erik Johnston | 2017-03-30 | 1 | -0/+300 |
| | |||||
* | Initial TCP protocol implementation | Erik Johnston | 2017-03-30 | 3 | -0/+974 |
| | | | | This defines the low level TCP replication protocol | ||||
* | Define the various streams we will replicate | Erik Johnston | 2017-03-30 | 2 | -0/+423 |
| | |||||
* | Make federation send queue take the current position | Erik Johnston | 2017-03-30 | 1 | -1/+1 |
| | |||||
* | Add new storage functions for new replication | Erik Johnston | 2017-03-30 | 1 | -1/+1 |
| | | | | | The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one. | ||||
* | Use presence replication stream to invalidate cache | Erik Johnston | 2017-03-24 | 1 | -0/+1 |
| | | | | | | Instead of using the cache invalidation replication stream to invalidate the _get_presence_cache, we can instead rely on the presence replication stream. This reduces the amount of replication traffic considerably. | ||||
* | Replace some calls to cursor_to_dict | Erik Johnston | 2017-03-24 | 1 | -1/+0 |
| | | | | | cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to. | ||||
* | PEP8 | Erik Johnston | 2017-03-20 | 1 | -2/+0 |
| | |||||
* | Don't send the full event json over replication | Erik Johnston | 2017-03-17 | 2 | -30/+16 |
| | |||||
* | Fix assertion to stop transaction queue getting wedged | Richard van der Hoff | 2017-03-15 | 1 | -0/+5 |
| | | | | | | | | ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop. | ||||
* | Add new storage function to slave store | Erik Johnston | 2017-03-13 | 1 | -0/+4 |
| | |||||
* | Fix it for the workers | Erik Johnston | 2017-03-10 | 1 | -0/+8 |
| | |||||
* | Don't log unknown cache warnings in workers | Erik Johnston | 2017-02-28 | 1 | -1/+3 |
| | |||||
* | Aggregate event push actions | Erik Johnston | 2017-02-14 | 1 | -0/+6 |
| | |||||
* | Cache get_presence storage | Erik Johnston | 2017-02-13 | 1 | -1/+3 |
| | |||||
* | Merge pull request #1912 from matrix-org/markjh/roominitialsync | Erik Johnston | 2017-02-13 | 1 | -0/+6 |
|\ | | | | | Add db functions needed for room initial sync to slave | ||||
| * | Add db functions needed for room initial sync to slave | Mark Haines | 2017-02-13 | 1 | -0/+6 |
| | | |||||
* | | Fix replication | Erik Johnston | 2017-02-02 | 1 | -0/+3 |
| | | |||||
* | | Only invalidate membership caches based on the cache stream | Erik Johnston | 2017-01-31 | 2 | -19/+5 |
|/ | | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive. | ||||
* | Rename func | Erik Johnston | 2017-01-30 | 1 | -1/+1 |
| | |||||
* | Hook device list updates to replication | Erik Johnston | 2017-01-27 | 2 | -1/+91 |
| | |||||
* | Insert delta of current_state_events to be more efficient | Erik Johnston | 2017-01-20 | 1 | -10/+0 |
| | |||||
* | Add cache to get_public_room_ids_at_stream_id | Erik Johnston | 2016-12-12 | 1 | -1/+2 |
| | |||||
* | Add new API appservice specific public room list | Erik Johnston | 2016-12-06 | 1 | -1/+1 |
| | |||||
* | Ensure only main or federation_sender process can send federation traffic | Erik Johnston | 2016-11-23 | 1 | -1/+1 |
| | |||||
* | Explicit federation ack | Erik Johnston | 2016-11-23 | 1 | -5/+10 |
| | |||||
* | Invalidate retry cache in both directions | Erik Johnston | 2016-11-22 | 4 | -6/+84 |
| | |||||
* | Store federation stream positions in the database | Erik Johnston | 2016-11-21 | 1 | -0/+3 |
| | |||||
* | Handle sending events and device messages over federation | Erik Johnston | 2016-11-17 | 4 | -7/+25 |
| | |||||
* | Hook up the send queue and create a federation sender worker | Erik Johnston | 2016-11-16 | 3 | -0/+32 |
| | |||||
* | Fix check of wrong variable | Erik Johnston | 2016-10-11 | 1 | -1/+1 |
| | |||||
* | Reduce DB hits for replication | Erik Johnston | 2016-09-23 | 1 | -36/+103 |
| | | | | | | | | | | | | | | Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly. | ||||
* | Fix public room pagination for client_reader app | Erik Johnston | 2016-09-16 | 1 | -0/+1 |
| | |||||
* | Add new storage function to slave store | Erik Johnston | 2016-09-16 | 1 | -0/+1 |
| | |||||
* | Use stream_change cache to make get_forward_extremeties_for_room cache more ↵ | Erik Johnston | 2016-09-15 | 1 | -1/+4 |
| | | | | effective | ||||
* | Add cache to get_forward_extremeties_for_room | Erik Johnston | 2016-09-15 | 1 | -1/+1 |
| | |||||
* | Stream public room changes down replication | Erik Johnston | 2016-09-15 | 3 | -1/+58 |
| | |||||
* | Correctly handle typing stream id resetting | Erik Johnston | 2016-09-09 | 1 | -1/+8 |
| | |||||
* | Drop replication log levels | Erik Johnston | 2016-09-09 | 1 | -2/+2 |
| | |||||
* | Track the max_stream_device_id in a separate table, since we delete from the ↵ | Mark Haines | 2016-09-09 | 1 | -1/+1 |
| | | | | inbox table | ||||
* | Merge branch 'develop' into markjh/direct_to_device_federation | Mark Haines | 2016-09-08 | 1 | -0/+3 |
|\ | |||||
| * | Add upgrade script that will slowly prune state_groups_state entries | Erik Johnston | 2016-09-05 | 1 | -0/+3 |
| | | |||||
* | | Fix the stream change cache to work over replication | Mark Haines | 2016-09-07 | 1 | -0/+11 |
|/ | |||||
* | Merge pull request #1060 from matrix-org/erikj/state_ids | Erik Johnston | 2016-09-01 | 2 | -23/+2 |
|\ | | | | | Assign state groups in state handler. | ||||
| * | Add storage function to SlaveStore | Erik Johnston | 2016-08-31 | 1 | -0/+1 |
| | | |||||
| * | Remove state replication stream | Erik Johnston | 2016-08-30 | 1 | -23/+1 |
| | | |||||
* | | Add a replication stream for direct to device messages | Mark Haines | 2016-08-31 | 2 | -1/+30 |
|/ | |||||
* | Merge pull request #1049 from matrix-org/erikj/presence_users_in_room | Erik Johnston | 2016-08-30 | 1 | -2/+5 |
|\ | | | | | Use state handler instead of get_users_in_room/get_joined_hosts | ||||
| * | Add to slave store | Erik Johnston | 2016-08-30 | 1 | -0/+5 |
| | | |||||
| * | Use state handler instead of get_users_in_room/get_joined_hosts | Erik Johnston | 2016-08-26 | 1 | -2/+0 |
| | | |||||
* | | Add new direct message storage functions to slave store | Erik Johnston | 2016-08-30 | 1 | -0/+30 |
|/ | |||||
* | Add is_host_joined to slave storage | Erik Johnston | 2016-08-26 | 1 | -0/+2 |
| | |||||
* | Add new state storage funcs to replication | Erik Johnston | 2016-08-26 | 1 | -0/+3 |
| | |||||
* | Add appservice worker | Erik Johnston | 2016-08-18 | 2 | -0/+13 |
| | |||||
* | Use cached get_user_by_access_token in slaves | Erik Johnston | 2016-08-16 | 2 | -2/+2 |
| | |||||
* | Rename table. Add docs. | Erik Johnston | 2016-08-15 | 1 | -1/+1 |
| | |||||
* | Use cached version of get_aliases_for_room | Erik Johnston | 2016-08-15 | 1 | -1/+1 |
| | |||||
* | Implement cache replication stream | Erik Johnston | 2016-08-15 | 2 | -2/+49 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader | Erik Johnston | 2016-07-29 | 1 | -2/+5 |
|\ | |||||
| * | Create separate methods for getting messages to push | Mark Haines | 2016-07-28 | 1 | -2/+5 |
| | | | | | | | | | | | | | | | | | | | | | | for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages. | ||||
* | | Add destination retry to slave store | Erik Johnston | 2016-07-28 | 1 | -0/+30 |
| | | |||||
* | | Add slace storage functions for public room list | Erik Johnston | 2016-07-28 | 2 | -0/+44 |
| | | |||||
* | | Add get_auth_chain to slave store | Erik Johnston | 2016-07-28 | 1 | -0/+4 |
| | | |||||
* | | Add more key storage funcs into slave store | Erik Johnston | 2016-07-28 | 1 | -6/+10 |
| | | |||||
* | | Split out a FederationReader process | Erik Johnston | 2016-07-21 | 2 | -0/+34 |
|/ | |||||
* | pep8 | David Baker | 2016-06-24 | 1 | -1/+0 |
| | |||||
* | Use similar naming we use in email notifs for push | David Baker | 2016-06-24 | 1 | -8/+0 |
| | | | | Fixes https://github.com/vector-im/vector-web/issues/1654 | ||||
* | Remove event fetching from DB threads | Erik Johnston | 2016-06-03 | 1 | -5/+0 |
| | |||||
* | Merge pull request #827 from matrix-org/markjh/more_slaved_methods | Mark Haines | 2016-06-03 | 3 | -6/+81 |
|\ | | | | | Add methods to events, account data and receipt slaves | ||||
| * | Add methods to events, account data and receipt slaves | Mark Haines | 2016-06-03 | 3 | -6/+81 |
| | | | | | | | | | | Adds the methods needed by /sync to the slaved events, account data and receipt stores. | ||||
* | | Merge pull request #824 from matrix-org/markjh/slaved_presence_store | Mark Haines | 2016-06-03 | 1 | -0/+59 |
|\ \ | | | | | | | Add a slaved store for presence | ||||
| * | | Add a slaved store for presence | Mark Haines | 2016-06-03 | 1 | -0/+59 |
| |/ | |||||
* | | Add a comment explaining why the filter cache doesn't need exipiring | Mark Haines | 2016-06-03 | 1 | -0/+1 |
| | | |||||
* | | Add slaved stores for filters, tokens, and push rules | Mark Haines | 2016-06-03 | 4 | -0/+151 |
|/ | |||||
* | Allow external processes to mark a user as syncing. (#812) | Mark Haines | 2016-06-02 | 2 | -0/+61 |
| | | | | | | | | | | | | * Add infrastructure to the presence handler to track sync requests in external processes * Expire stale entries for dead external processes * Add an http endpoint for making users as syncing Add some docstrings and comments. * Fixes | ||||
* | Move typing handler out of the Handlers object | Mark Haines | 2016-05-17 | 1 | -1/+1 |
| | |||||
* | Move the presence handler out of the Handlers object | Mark Haines | 2016-05-16 | 1 | -1/+1 |
| | |||||
* | Assert that stream replicated stream positions are ints | Mark Haines | 2016-05-13 | 3 | -7/+7 |
| | |||||
* | Merge branch 'develop' into dbkr/email_notifs_on_pusher | Mark Haines | 2016-05-13 | 3 | -1/+64 |
|\ | |||||
| * | Add a slaved datastore for account data | Mark Haines | 2016-05-13 | 1 | -0/+61 |
| | | |||||
| * | Log the stream IDs in an order that makes sense | Mark Haines | 2016-05-13 | 1 | -1/+1 |
| | | |||||
| * | Make sure we advance our stream position | Mark Haines | 2016-05-13 | 1 | -0/+2 |
| | | |||||
* | | Shift some of the state_group methods into the SlavedEventStore | Mark Haines | 2016-05-13 | 1 | -0/+19 |
|/ | |||||
* | Pass through get_events to pusher too | David Baker | 2016-05-10 | 1 | -0/+1 |
| | |||||
* | Pass through _get_event_txn | David Baker | 2016-05-10 | 1 | -0/+1 |
| | |||||
* | Don't warn | Erik Johnston | 2016-05-05 | 1 | -2/+0 |
| | |||||
* | Add some log information at returned replication streams | Erik Johnston | 2016-05-05 | 1 | -0/+11 |
| | |||||
* | Report per request metrics for all of the things using request_handler | Mark Haines | 2016-04-28 | 2 | -2/+4 |
| | |||||
* | Fix backfill replication to advance the stream correctly | Mark Haines | 2016-04-27 | 2 | -3/+3 |
| | |||||
* | Merge pull request #743 from matrix-org/markjh/slave_pushers | Mark Haines | 2016-04-21 | 1 | -0/+52 |
|\ | | | | | Replicate the pushers | ||||
| * | Replicate the pushers | Mark Haines | 2016-04-21 | 1 | -0/+52 |
| | | |||||
* | | Merge pull request #744 from matrix-org/markjh/replication_remove_pusher | Mark Haines | 2016-04-21 | 2 | -3/+57 |
|\ \ | | | | | | | Add a replication endpoint for deleting pushers | ||||
| * | | Add a replication endpoint for deleting pushers | Mark Haines | 2016-04-21 | 2 | -3/+57 |
| |/ | |||||
* / | Replicate push actions | Mark Haines | 2016-04-21 | 1 | -0/+14 |
|/ | |||||
* | Merge pull request #738 from matrix-org/markjh/slaved_receipts | Mark Haines | 2016-04-19 | 1 | -0/+61 |
|\ | | | | | Add a slaved receipts store | ||||
| * | Add a slaved receipts store | Mark Haines | 2016-04-19 | 1 | -0/+61 |
| | | |||||
* | | Replicate get_invited_rooms_for_user | Mark Haines | 2016-04-19 | 1 | -2/+7 |
|/ | |||||
* | Add tests for redactions | Mark Haines | 2016-04-07 | 1 | -2/+2 |
| | |||||
* | Add tests for get_latest_event_ids_in_room and get_current_state | Mark Haines | 2016-04-07 | 1 | -0/+5 |
| | |||||
* | Test that room membership is replicated | Mark Haines | 2016-04-06 | 1 | -3/+4 |
| | |||||
* | Add a slaved events store class | Mark Haines | 2016-04-06 | 5 | -0/+284 |
| | | | | | Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store. | ||||
* | Separate generating the replication response... | Mark Haines | 2016-04-05 | 1 | -44/+55 |
| | | | | | from doing the http request parsing to make it easier to write unit tests for replication. | ||||
* | Merge pull request #676 from matrix-org/markjh/replicate_stateIII | Mark Haines | 2016-03-31 | 1 | -3/+14 |
|\ | | | | | Add replication streams for ex outliers and current state resets | ||||
| * | Use a namedtuple rather than tuple unpacking | Mark Haines | 2016-03-31 | 1 | -10/+6 |
| | | |||||
| * | Add replication streams for ex outliers and current state resets | Mark Haines | 2016-03-30 | 1 | -1/+16 |
| | | |||||
* | | typo | Matthew Hodgson | 2016-03-30 | 1 | -1/+1 |
|/ | |||||
* | Add a replication stream for state groups | Mark Haines | 2016-03-30 | 1 | -7/+29 |
| | |||||
* | Add replication stream for pushers | Mark Haines | 2016-03-15 | 1 | -1/+24 |
| | |||||
* | s/stream_ordering/event_stream_ordering/ in push | Mark Haines | 2016-03-04 | 1 | -1/+1 |
| | |||||
* | Hook push rules up to the replication API | Mark Haines | 2016-03-02 | 1 | -2/+26 |
| | |||||
* | Add a /replication API for extracting the updates that happened on | Mark Haines | 2016-03-01 | 2 | -0/+334 |
synapse This is necessary for replicating the data in synapse to be visible to a separate service because presence and typing notifications aren't stored in a database so won't be visible to another process. This API can be used to either get the raw data by requesting the tables themselves or to just receive notifications for updates by following the streams meta-stream. Returns updates for each table requested a JSON array of arrays with a row for each row in the table. Each table is prefixed by a header row with the: name of the table, current stream_id position for the table, number of rows, number of columns and the names of the columns. This is followed by the rows that have been added to the server since the requester last asked. The API has a timeout and is hooked up to the notifier so that a slave can long poll for updates. |