Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Logcontexts for replication command handlers | Richard van der Hoff | 2018-08-17 | 3 | -15/+43 |
| | | | | | | | | | | Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky. | ||||
* | Use federation handler function rather than duplicate | Erik Johnston | 2018-08-15 | 1 | -41/+3 |
| | | | | This involves renaming _persist_events to be a public function. | ||||
* | Rename slave TransactionStore to SlaveTransactionStore | Erik Johnston | 2018-08-15 | 1 | -1/+1 |
| | |||||
* | Move clean_room_for_join to master | Erik Johnston | 2018-08-09 | 1 | -0/+35 |
| | |||||
* | Fixup doc comments | Erik Johnston | 2018-08-09 | 1 | -0/+17 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-09 | 4 | -17/+63 |
|\ | | | | | | | erikj/split_federation | ||||
| * | Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet | Erik Johnston | 2018-08-09 | 3 | -243/+374 |
| |\ | | | | | | | Add helper base class for generating new replication endpoints | ||||
| | * | Fixup wording and remove dead code | Erik Johnston | 2018-08-09 | 1 | -2/+1 |
| | | | |||||
| | * | Rename POST param to METHOD | Erik Johnston | 2018-08-08 | 2 | -13/+22 |
| | | | |||||
| | * | Fixup logging and docstrings | Erik Johnston | 2018-08-08 | 2 | -2/+40 |
| | | | |||||
| * | | Basic support for room versioning | Richard van der Hoff | 2018-08-03 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | | | | | | | | | This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event | ||||
* | | | Import all functions from TransactionStore | Erik Johnston | 2018-08-06 | 1 | -11/+2 |
| | | | |||||
* | | | Add EDU/query handling over replication | Erik Johnston | 2018-08-06 | 1 | -1/+1 |
| | | | |||||
* | | | Add replication APIs for persisting federation events | Erik Johnston | 2018-08-06 | 2 | -1/+247 |
| |/ |/| | |||||
* | | Fix isort | Erik Johnston | 2018-08-06 | 1 | -4/+1 |
| | | |||||
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-03 | 1 | -4/+3 |
|\| | | | | | | | erikj/refactor_repl_servlet | ||||
| * | Kill off MatrixCodeMessageException | Richard van der Hoff | 2018-08-01 | 2 | -16/+12 |
| | | | | | | | | | | | | | | | | | | | | | | This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error. | ||||
* | | Use new helper base class for membership requests | Erik Johnston | 2018-07-31 | 1 | -171/+91 |
| | | |||||
* | | Use new helper base class for ReplicationSendEventRestServlet | Erik Johnston | 2018-07-31 | 1 | -79/+36 |
| | | |||||
* | | Add helper base class for generating new replication endpoints | Erik Johnston | 2018-07-31 | 1 | -0/+208 |
|/ | | | | | This will hopefully reduce the boiler plate required to implement new internal HTTP requests. | ||||
* | Fix unit tests | Richard van der Hoff | 2018-07-25 | 1 | -1/+1 |
| | | | | | | on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival. | ||||
* | Wrap a number of things that run in the background | Richard van der Hoff | 2018-07-25 | 1 | -6/+8 |
| | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics. | ||||
* | Fix missing attributes on workers. | Erik Johnston | 2018-07-23 | 1 | -2/+5 |
| | | | | | This was missed during the transition from attribute to getter for getting state from context. | ||||
* | Use stream cache in get_linearized_receipts_for_room | Erik Johnston | 2018-07-10 | 1 | -1/+1 |
| | | | | | This avoids us from uncessarily hitting the database when there has been no change for the room | ||||
* | run isort | Amber Brown | 2018-07-09 | 24 | -66/+91 |
| | |||||
* | Attempt to be more performant on PyPy (#3462) | Amber Brown | 2018-06-28 | 1 | -6/+10 |
| | |||||
* | Merge pull request #3441 from matrix-org/erikj/redo_erasure | Erik Johnston | 2018-06-25 | 1 | -0/+2 |
|\ | | | | | Fix user erasure and re-enable | ||||
| * | Add UserErasureWorkerStore to workers | Erik Johnston | 2018-06-25 | 1 | -0/+2 |
| | | |||||
* | | Remove all global reactor imports & pass it around explicitly (#3424) | Amber Brown | 2018-06-25 | 2 | -5/+5 |
|/ | |||||
* | Pass around the reactor explicitly (#3385) | Amber Brown | 2018-06-22 | 1 | -3/+3 |
| | |||||
* | Fix tcp protocol metrics naming (#3410) | Amber Brown | 2018-06-21 | 1 | -18/+35 |
| | |||||
* | Fix replication metrics | Richard van der Hoff | 2018-06-04 | 1 | -2/+2 |
| | | | | fix bug introduced in #3256 | ||||
* | Merge remote-tracking branch 'origin/develop' into 3218-official-prom | Amber Brown | 2018-05-28 | 2 | -8/+9 |
|\ | |||||
| * | Merge pull request #3244 from NotAFile/py3-six-4 | Amber Brown | 2018-05-24 | 2 | -5/+7 |
| |\ | | | | | | | replace some iteritems with six | ||||
| | * | replace some iteritems with six | Adrian Tschira | 2018-05-19 | 2 | -5/+7 |
| | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | | more cleanup | Amber Brown | 2018-05-22 | 2 | -6/+10 |
| | | | |||||
* | | | fix the test failures | Amber Brown | 2018-05-22 | 1 | -1/+1 |
| | | | |||||
* | | | cleanups, self-registration | Amber Brown | 2018-05-22 | 1 | -4/+5 |
| | | | |||||
* | | | Merge remote-tracking branch 'origin/develop' into 3218-official-prom | Amber Brown | 2018-05-22 | 1 | -0/+2 |
|\| | | |||||
| * | | Send users a server notice about consent | Richard van der Hoff | 2018-05-22 | 1 | -0/+2 |
| |/ | | | | | | | | | When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so. | ||||
* | | rest of the changes | Amber Brown | 2018-05-21 | 1 | -16/+14 |
| | | |||||
* | | replacing portions | Amber Brown | 2018-05-21 | 1 | -54/+34 |
|/ | |||||
* | make imports local | Adrian Tschira | 2018-04-28 | 2 | -4/+4 |
| | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | Refactor ResponseCache usage | Richard van der Hoff | 2018-04-12 | 1 | -12/+6 |
| | | | | | | | | | | | | | | | Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production. | ||||
* | Add metrics for ResponseCache | Richard van der Hoff | 2018-04-10 | 1 | -1/+1 |
| | |||||
* | Fix json encoding bug in replication | Richard van der Hoff | 2018-04-03 | 1 | -1/+1 |
| | | | | json encoders have an encode method, not a dumps method. | ||||
* | Use static JSONEncoders | Richard van der Hoff | 2018-03-29 | 1 | -3/+5 |
| | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it. | ||||
* | Explicitly use simplejson | Erik Johnston | 2018-03-20 | 1 | -7/+7 |
| | |||||
* | Fix replication after switch to simplejson | Erik Johnston | 2018-03-19 | 1 | -2/+4 |
| | | | | | Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default. | ||||
* | Merge branch 'master' of github.com:matrix-org/synapse into develop | Erik Johnston | 2018-03-19 | 1 | -1/+1 |
|\ | |||||
| * | Replace ujson with simplejson | Erik Johnston | 2018-03-15 | 1 | -1/+1 |
| | | |||||
* | | Merge pull request #2992 from matrix-org/erikj/implement_member_workre | Erik Johnston | 2018-03-14 | 2 | -3/+336 |
|\ \ | | | | | | | Implement RoomMemberWorkerHandler | ||||
| * | | Fix imports | Erik Johnston | 2018-03-14 | 2 | -7/+4 |
| | | | |||||
| * | | s/join/joined/ in notify_user_membership_change | Erik Johnston | 2018-03-14 | 1 | -3/+3 |
| | | | |||||
| * | | Implement RoomMemberWorkerHandler | Erik Johnston | 2018-03-13 | 2 | -0/+336 |
| | | | |||||
* | | | Split up ProfileStore | Erik Johnston | 2018-03-13 | 1 | -0/+21 |
|/ / | |||||
* | | extra_users is actually a list of UserIDs | Erik Johnston | 2018-03-13 | 1 | -4/+4 |
| | | |||||
* | | Split Directory store | Erik Johnston | 2018-03-05 | 1 | -5/+3 |
| | | |||||
* | | Split registration store | Erik Johnston | 2018-03-02 | 1 | -15/+3 |
| | | |||||
* | | Merge pull request #2928 from matrix-org/erikj/read_marker_caches | Erik Johnston | 2018-03-01 | 1 | -1/+1 |
|\ \ | | | | | | | Fix typo in getting replication account data processing | ||||
| * | | Fix typo in getting replication account data processing | Erik Johnston | 2018-03-01 | 1 | -1/+1 |
| | | | |||||
* | | | Merge pull request #2925 from matrix-org/erikj/split_sig_fed | Erik Johnston | 2018-03-01 | 1 | -47/+9 |
|\ \ \ | | | | | | | | | Split out SignatureStore and EventFederationStore | ||||
| * | | | Remove unused DataStore | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| | | | | |||||
| * | | | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed | Erik Johnston | 2018-03-01 | 2 | -65/+11 |
| |\ \ \ | |||||
| * | | | | Split out SignatureStore and EventFederationStore | Erik Johnston | 2018-03-01 | 1 | -42/+8 |
| | | | | | |||||
* | | | | | Merge pull request #2927 from matrix-org/erikj/read_marker_caches | Erik Johnston | 2018-03-01 | 1 | -0/+4 |
|\ \ \ \ \ | |_|/ / / |/| | / / | | |/ / | |/| | | Improve caching for read_marker API | ||||
| * | | | Improve caching for read_marker API | Erik Johnston | 2018-03-01 | 1 | -0/+3 |
| | | | | | | | | | | | | | | | | | | | | | | | | We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data. | ||||
| * | | | Add some caches to help read marker API | Erik Johnston | 2018-03-01 | 1 | -0/+1 |
| |/ / | |||||
* | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-03-01 | 2 | -19/+4 |
|\ \ \ | | | | | | | | | | | | | erikj/split_stream_store | ||||
| * \ \ | Merge pull request #2923 from matrix-org/erikj/stream_ago_worker | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| |\ \ \ | | | | | | | | | | | Calculate stream_ordering_month_ago correctly on workers | ||||
| | * | | | Calculate stream_ordering_month_ago correctly on workers | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| | |/ / | |||||
| * / / | Split up RoomStore | Erik Johnston | 2018-03-01 | 1 | -17/+4 |
| |/ / | |||||
* / / | Split out stream store | Erik Johnston | 2018-03-01 | 1 | -45/+9 |
|/ / | |||||
* | | Log in the correct places | Erik Johnston | 2018-03-01 | 1 | -2/+4 |
| | | |||||
* | | Don't do preserve_fn for every request | Erik Johnston | 2018-03-01 | 1 | -1/+2 |
| | | |||||
* | | Add some logging | Erik Johnston | 2018-03-01 | 1 | -0/+2 |
| | | |||||
* | | Make repl send_event idempotent and retry on timeouts | Erik Johnston | 2018-03-01 | 1 | -6/+38 |
| | | | | | | | | | | | | If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event. | ||||
* | | Correctly send ratelimit and extra_users params | Erik Johnston | 2018-03-01 | 1 | -1/+13 |
| | | |||||
* | | Calculate push actions on worker | Erik Johnston | 2018-02-28 | 1 | -1/+1 |
| | | |||||
* | | Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation | Erik Johnston | 2018-02-27 | 1 | -0/+2 |
|\ \ | | | | | | | Fix missing invalidations for receipt storage | ||||
| * | | Fix missing invalidations for receipt storage | Erik Johnston | 2018-02-21 | 1 | -0/+2 |
| | | | |||||
* | | | Merge pull request #2903 from matrix-org/erikj/split_roommember_store | Erik Johnston | 2018-02-27 | 1 | -28/+5 |
|\ \ \ | | | | | | | | | Split out RoomMemberStore | ||||
| * \ \ | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-02-23 | 4 | -84/+23 |
| |\ \ \ | | | | | | | | | | | | | | | | erikj/split_roommember_store | ||||
| * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Split out RoomMemberStore | Erik Johnston | 2018-02-21 | 1 | -27/+3 |
| | | | | | |||||
* | | | | | Merge pull request #2901 from matrix-org/erikj/split_as_stores | Erik Johnston | 2018-02-27 | 1 | -28/+7 |
|\ \ \ \ \ | |_|/ / / |/| | | | | Split AS stores | ||||
| * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Split AS stores | Erik Johnston | 2018-02-21 | 1 | -28/+6 |
| |/ / / | |||||
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-02-23 | 4 | -74/+21 |
|\ \ \ \ | | | | | | | | | | | | | | | | erikj/split_event_push_actions | ||||
| * \ \ \ | Merge pull request #2902 from matrix-org/erikj/split_events_store | Erik Johnston | 2018-02-23 | 1 | -11/+3 |
| |\ \ \ \ | | | | | | | | | | | | | Split out get_events and co into a worker store | ||||
| | * | | | | Split EventsWorkerStore into separate file | Erik Johnston | 2018-02-23 | 1 | -1/+1 |
| | | | | | | |||||
| | * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | |/ / / | |||||
| | * / / | Split out get_events and co into a worker store | Erik Johnston | 2018-02-21 | 1 | -11/+2 |
| | |/ / | |||||
| * | | | Merge pull request #2899 from matrix-org/erikj/split_pushers | Erik Johnston | 2018-02-23 | 1 | -9/+3 |
| |\ \ \ | | | | | | | | | | | Split PusherStore | ||||
| | * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| | * | | | Split PusherStore | Erik Johnston | 2018-02-21 | 1 | -9/+2 |
| | |/ / | |||||
| * | | | Merge pull request #2898 from matrix-org/erikj/split_push_rules_store | Erik Johnston | 2018-02-23 | 1 | -17/+7 |
| |\ \ \ | | | | | | | | | | | Split PushRulesStore | ||||
| | * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| | * | | | Split PushRulesStore | Erik Johnston | 2018-02-21 | 1 | -17/+6 |
| | |/ / | |||||
| * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | |||||
| * | | | Use absolute imports | Erik Johnston | 2018-02-23 | 1 | -2/+2 |
| | | | | |||||
| * | | | Split AccountDataStore and TagStore | Erik Johnston | 2018-02-21 | 1 | -34/+4 |
| |/ / | |||||
* / / | Split out EventPushActionWorkerStore | Erik Johnston | 2018-02-21 | 1 | -20/+3 |
|/ / | |||||
* | | Fix comment | Erik Johnston | 2018-02-20 | 1 | -1/+1 |
| | | |||||
* | | Use abstract base class to access stream IDs | Erik Johnston | 2018-02-20 | 1 | -2/+7 |
| | | |||||
* | | Split ReceiptsStore | Erik Johnston | 2018-02-20 | 1 | -28/+5 |
| | | |||||
* | | Don't serialize current state over replication | Erik Johnston | 2018-02-15 | 1 | -2/+2 |
| | | |||||
* | | Don't log errors propogated from send_event | Erik Johnston | 2018-02-15 | 1 | -1/+10 |
| | | |||||
* | | Add event_creator worker | Erik Johnston | 2018-02-07 | 1 | -0/+20 |
| | | |||||
* | | Add replication http endpoint for event sending | Erik Johnston | 2018-02-07 | 2 | -0/+139 |
| | | |||||
* | | Store state groups separately from events (#2784) | Erik Johnston | 2018-02-06 | 1 | -2/+2 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints | ||||
* | | Metrics for number of RDATA commands received | Richard van der Hoff | 2018-01-15 | 1 | -5/+14 |
|/ | | | | I found myself wishing we had this. | ||||
* | Fix some logcontext leaks in replication resource | Richard van der Hoff | 2017-11-23 | 1 | -2/+4 |
| | | | | | The @measure_func annotations rely on the wrapped function respecting the logcontext rules. Add the necessary yields to make this work. | ||||
* | Pull out bits of StateStore to a mixin | Richard van der Hoff | 2017-11-14 | 1 | -33/+6 |
| | | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655. | ||||
* | Make __init__ consitstent across Store heirarchy | Richard van der Hoff | 2017-11-13 | 1 | -1/+1 |
| | | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores) | ||||
* | replace 'except:' with 'except Exception:' | Richard van der Hoff | 2017-10-23 | 1 | -1/+1 |
| | | | | what could possibly go wrong | ||||
* | log when we get an exception handling replication updates | hera | 2017-10-12 | 1 | -1/+5 |
| | |||||
* | Fix replication. And notify | Erik Johnston | 2017-07-20 | 2 | -0/+74 |
| | |||||
* | Reduce log levels in tcp replication | Erik Johnston | 2017-07-11 | 1 | -2/+2 |
| | |||||
* | Serialize user ip command as json | Erik Johnston | 2017-06-27 | 1 | -5/+9 |
| | |||||
* | Fix up | Erik Johnston | 2017-06-27 | 1 | -2/+1 |
| | |||||
* | Make workers report to master for user ip updates | Erik Johnston | 2017-06-27 | 5 | -0/+103 |
| | |||||
* | Initialise exclusive_user_regex | Erik Johnston | 2017-06-21 | 1 | -0/+2 |
| | |||||
* | Fix for workers | Erik Johnston | 2017-06-21 | 1 | -0/+3 |
| | |||||
* | Initial worker impl | Erik Johnston | 2017-06-16 | 1 | -0/+22 |
| | |||||
* | Add cache for is_host_joined | Erik Johnston | 2017-06-13 | 1 | -1/+1 |
| | |||||
* | Merge pull request #2266 from matrix-org/erikj/host_in_room | Erik Johnston | 2017-06-12 | 1 | -1/+0 |
|\ | | | | | Change is_host_joined to use current_state table | ||||
| * | Fix replication | Erik Johnston | 2017-06-09 | 1 | -1/+0 |
| | | |||||
* | | Fix replication | Erik Johnston | 2017-06-09 | 1 | -1/+1 |
| | | |||||
* | | Add missing notifier | Erik Johnston | 2017-06-09 | 1 | -1/+2 |
|/ | |||||
* | Faster cache for get_joined_hosts | Erik Johnston | 2017-05-25 | 1 | -0/+2 |
| | |||||
* | Add missing storage function to slave store | Erik Johnston | 2017-05-22 | 1 | -0/+2 |
| | |||||
* | Add new storage functions to slave store | Erik Johnston | 2017-05-04 | 1 | -0/+3 |
| | |||||
* | Remove unused cache | Erik Johnston | 2017-04-24 | 1 | -3/+0 |
| | |||||
* | Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl | Erik Johnston | 2017-04-12 | 2 | -0/+11 |
|\ | | | | | Reduce federation replication traffic | ||||
| * | Move get_presence_list_* to SlaveStore | Erik Johnston | 2017-04-11 | 1 | -0/+10 |
| | | |||||
| * | Reduce federation presence replication traffic | Erik Johnston | 2017-04-10 | 1 | -0/+1 |
| | | | | | | | | | | | | | | | | This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination. | ||||
* | | Remove HTTP replication APIs | Erik Johnston | 2017-04-11 | 4 | -749/+0 |
|/ | |||||
* | Typo | Erik Johnston | 2017-04-10 | 1 | -1/+1 |
| | |||||
* | Merge pull request #2109 from matrix-org/erikj/send_queue_fix | Erik Johnston | 2017-04-10 | 1 | -2/+2 |
|\ | | | | | Fix up federation SendQueue and document types | ||||
| * | Comments | Erik Johnston | 2017-04-10 | 1 | -2/+2 |
| | | |||||
* | | Up replication ping timeout | Erik Johnston | 2017-04-10 | 1 | -2/+4 |
|/ | |||||
* | Merge pull request #2103 from matrix-org/erikj/no-double-encode | Erik Johnston | 2017-04-07 | 1 | -28/+76 |
|\ | | | | | Don't double encode replication data | ||||
| * | Document types of the replication streams | Erik Johnston | 2017-04-06 | 1 | -28/+76 |
| | | |||||
* | | Fix incorrect type when using InvalidateCacheCommand | Erik Johnston | 2017-04-06 | 1 | -1/+1 |
| | | |||||
* | | Add log lines | Erik Johnston | 2017-04-05 | 1 | -1/+2 |
| | | |||||
* | | Rearrange metrics | Erik Johnston | 2017-04-05 | 1 | -16/+31 |
| | | |||||
* | | Fix typo | Erik Johnston | 2017-04-05 | 1 | -2/+2 |
| | | |||||
* | | Fixup some metrics for tcp repl | Erik Johnston | 2017-04-05 | 1 | -0/+16 |
|/ | |||||
* | Merge pull request #2097 from matrix-org/erikj/repl_tcp_client | Erik Johnston | 2017-04-05 | 11 | -171/+302 |
|\ | | | | | Move to using TCP replication | ||||
| * | Change slave storage to use new replication interface | Erik Johnston | 2017-04-03 | 10 | -171/+106 |
| | | | | | | | | | | | | | | As the TCP replication uses a slightly different API and streams than the HTTP replication. This breaks HTTP replication. | ||||
| * | Add basic replication client handler and factory | Erik Johnston | 2017-04-03 | 1 | -0/+196 |
| | | |||||
* | | Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix | Erik Johnston | 2017-04-04 | 3 | -6/+15 |
|\ \ | | | | | | | Advance replication streams even if nothing is listening | ||||
| * | | Advance replication streams even if nothing is listening | Erik Johnston | 2017-04-04 | 3 | -6/+15 |
| |/ | | | | | | | | | | | Otherwise the streams don't advance and steadily fall behind, so when a worker does connect either a) they'll be streamed lots of old updates or b) the connection will fail as the streams are too far behind. | ||||
* / | Fiddle tcp replication logging | Erik Johnston | 2017-04-04 | 1 | -2/+2 |
|/ | |||||
* | Always advance stream tokens | Erik Johnston | 2017-04-03 | 1 | -1/+4 |
| | |||||
* | Use callbacks to notify tcp replication rather than deferreds | Erik Johnston | 2017-03-31 | 1 | -14/+1 |
| | |||||
* | Add a timestamp to USER_SYNC command | Erik Johnston | 2017-03-31 | 3 | -9/+17 |
| | | | | This timestamp is used to indicate when the user last sync'd | ||||
* | Fix up docs | Erik Johnston | 2017-03-31 | 2 | -19/+3 |
| | |||||
* | Add server side resource for tcp replication | Erik Johnston | 2017-03-30 | 1 | -0/+300 |
| | |||||
* | Initial TCP protocol implementation | Erik Johnston | 2017-03-30 | 3 | -0/+974 |
| | | | | This defines the low level TCP replication protocol | ||||
* | Define the various streams we will replicate | Erik Johnston | 2017-03-30 | 2 | -0/+423 |
| | |||||
* | Make federation send queue take the current position | Erik Johnston | 2017-03-30 | 1 | -1/+1 |
| | |||||
* | Add new storage functions for new replication | Erik Johnston | 2017-03-30 | 1 | -1/+1 |
| | | | | | The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one. | ||||
* | Use presence replication stream to invalidate cache | Erik Johnston | 2017-03-24 | 1 | -0/+1 |
| | | | | | | Instead of using the cache invalidation replication stream to invalidate the _get_presence_cache, we can instead rely on the presence replication stream. This reduces the amount of replication traffic considerably. | ||||
* | Replace some calls to cursor_to_dict | Erik Johnston | 2017-03-24 | 1 | -1/+0 |
| | | | | | cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to. | ||||
* | PEP8 | Erik Johnston | 2017-03-20 | 1 | -2/+0 |
| | |||||
* | Don't send the full event json over replication | Erik Johnston | 2017-03-17 | 2 | -30/+16 |
| | |||||
* | Fix assertion to stop transaction queue getting wedged | Richard van der Hoff | 2017-03-15 | 1 | -0/+5 |
| | | | | | | | | ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop. | ||||
* | Add new storage function to slave store | Erik Johnston | 2017-03-13 | 1 | -0/+4 |
| | |||||
* | Fix it for the workers | Erik Johnston | 2017-03-10 | 1 | -0/+8 |
| | |||||
* | Don't log unknown cache warnings in workers | Erik Johnston | 2017-02-28 | 1 | -1/+3 |
| | |||||
* | Aggregate event push actions | Erik Johnston | 2017-02-14 | 1 | -0/+6 |
| | |||||
* | Cache get_presence storage | Erik Johnston | 2017-02-13 | 1 | -1/+3 |
| | |||||
* | Merge pull request #1912 from matrix-org/markjh/roominitialsync | Erik Johnston | 2017-02-13 | 1 | -0/+6 |
|\ | | | | | Add db functions needed for room initial sync to slave | ||||
| * | Add db functions needed for room initial sync to slave | Mark Haines | 2017-02-13 | 1 | -0/+6 |
| | | |||||
* | | Fix replication | Erik Johnston | 2017-02-02 | 1 | -0/+3 |
| | | |||||
* | | Only invalidate membership caches based on the cache stream | Erik Johnston | 2017-01-31 | 2 | -19/+5 |
|/ | | | | | Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive. | ||||
* | Rename func | Erik Johnston | 2017-01-30 | 1 | -1/+1 |
| | |||||
* | Hook device list updates to replication | Erik Johnston | 2017-01-27 | 2 | -1/+91 |
| | |||||
* | Insert delta of current_state_events to be more efficient | Erik Johnston | 2017-01-20 | 1 | -10/+0 |
| | |||||
* | Add cache to get_public_room_ids_at_stream_id | Erik Johnston | 2016-12-12 | 1 | -1/+2 |
| | |||||
* | Add new API appservice specific public room list | Erik Johnston | 2016-12-06 | 1 | -1/+1 |
| | |||||
* | Ensure only main or federation_sender process can send federation traffic | Erik Johnston | 2016-11-23 | 1 | -1/+1 |
| | |||||
* | Explicit federation ack | Erik Johnston | 2016-11-23 | 1 | -5/+10 |
| | |||||
* | Invalidate retry cache in both directions | Erik Johnston | 2016-11-22 | 4 | -6/+84 |
| | |||||
* | Store federation stream positions in the database | Erik Johnston | 2016-11-21 | 1 | -0/+3 |
| | |||||
* | Handle sending events and device messages over federation | Erik Johnston | 2016-11-17 | 4 | -7/+25 |
| | |||||
* | Hook up the send queue and create a federation sender worker | Erik Johnston | 2016-11-16 | 3 | -0/+32 |
| | |||||
* | Fix check of wrong variable | Erik Johnston | 2016-10-11 | 1 | -1/+1 |
| | |||||
* | Reduce DB hits for replication | Erik Johnston | 2016-09-23 | 1 | -36/+103 |
| | | | | | | | | | | | | | | Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly. | ||||
* | Fix public room pagination for client_reader app | Erik Johnston | 2016-09-16 | 1 | -0/+1 |
| | |||||
* | Add new storage function to slave store | Erik Johnston | 2016-09-16 | 1 | -0/+1 |
| | |||||
* | Use stream_change cache to make get_forward_extremeties_for_room cache more ↵ | Erik Johnston | 2016-09-15 | 1 | -1/+4 |
| | | | | effective | ||||
* | Add cache to get_forward_extremeties_for_room | Erik Johnston | 2016-09-15 | 1 | -1/+1 |
| | |||||
* | Stream public room changes down replication | Erik Johnston | 2016-09-15 | 3 | -1/+58 |
| | |||||
* | Correctly handle typing stream id resetting | Erik Johnston | 2016-09-09 | 1 | -1/+8 |
| | |||||
* | Drop replication log levels | Erik Johnston | 2016-09-09 | 1 | -2/+2 |
| | |||||
* | Track the max_stream_device_id in a separate table, since we delete from the ↵ | Mark Haines | 2016-09-09 | 1 | -1/+1 |
| | | | | inbox table | ||||
* | Merge branch 'develop' into markjh/direct_to_device_federation | Mark Haines | 2016-09-08 | 1 | -0/+3 |
|\ | |||||
| * | Add upgrade script that will slowly prune state_groups_state entries | Erik Johnston | 2016-09-05 | 1 | -0/+3 |
| | | |||||
* | | Fix the stream change cache to work over replication | Mark Haines | 2016-09-07 | 1 | -0/+11 |
|/ | |||||
* | Merge pull request #1060 from matrix-org/erikj/state_ids | Erik Johnston | 2016-09-01 | 2 | -23/+2 |
|\ | | | | | Assign state groups in state handler. | ||||
| * | Add storage function to SlaveStore | Erik Johnston | 2016-08-31 | 1 | -0/+1 |
| | | |||||
| * | Remove state replication stream | Erik Johnston | 2016-08-30 | 1 | -23/+1 |
| | | |||||
* | | Add a replication stream for direct to device messages | Mark Haines | 2016-08-31 | 2 | -1/+30 |
|/ | |||||
* | Merge pull request #1049 from matrix-org/erikj/presence_users_in_room | Erik Johnston | 2016-08-30 | 1 | -2/+5 |
|\ | | | | | Use state handler instead of get_users_in_room/get_joined_hosts | ||||
| * | Add to slave store | Erik Johnston | 2016-08-30 | 1 | -0/+5 |
| | | |||||
| * | Use state handler instead of get_users_in_room/get_joined_hosts | Erik Johnston | 2016-08-26 | 1 | -2/+0 |
| | | |||||
* | | Add new direct message storage functions to slave store | Erik Johnston | 2016-08-30 | 1 | -0/+30 |
|/ | |||||
* | Add is_host_joined to slave storage | Erik Johnston | 2016-08-26 | 1 | -0/+2 |
| | |||||
* | Add new state storage funcs to replication | Erik Johnston | 2016-08-26 | 1 | -0/+3 |
| | |||||
* | Add appservice worker | Erik Johnston | 2016-08-18 | 2 | -0/+13 |
| | |||||
* | Use cached get_user_by_access_token in slaves | Erik Johnston | 2016-08-16 | 2 | -2/+2 |
| | |||||
* | Rename table. Add docs. | Erik Johnston | 2016-08-15 | 1 | -1/+1 |
| | |||||
* | Use cached version of get_aliases_for_room | Erik Johnston | 2016-08-15 | 1 | -1/+1 |
| | |||||
* | Implement cache replication stream | Erik Johnston | 2016-08-15 | 2 | -2/+49 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader | Erik Johnston | 2016-07-29 | 1 | -2/+5 |
|\ | |||||
| * | Create separate methods for getting messages to push | Mark Haines | 2016-07-28 | 1 | -2/+5 |
| | | | | | | | | | | | | | | | | | | | | | | for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages. | ||||
* | | Add destination retry to slave store | Erik Johnston | 2016-07-28 | 1 | -0/+30 |
| | | |||||
* | | Add slace storage functions for public room list | Erik Johnston | 2016-07-28 | 2 | -0/+44 |
| | | |||||
* | | Add get_auth_chain to slave store | Erik Johnston | 2016-07-28 | 1 | -0/+4 |
| | | |||||
* | | Add more key storage funcs into slave store | Erik Johnston | 2016-07-28 | 1 | -6/+10 |
| | | |||||
* | | Split out a FederationReader process | Erik Johnston | 2016-07-21 | 2 | -0/+34 |
|/ | |||||
* | pep8 | David Baker | 2016-06-24 | 1 | -1/+0 |
| | |||||
* | Use similar naming we use in email notifs for push | David Baker | 2016-06-24 | 1 | -8/+0 |
| | | | | Fixes https://github.com/vector-im/vector-web/issues/1654 | ||||
* | Remove event fetching from DB threads | Erik Johnston | 2016-06-03 | 1 | -5/+0 |
| | |||||
* | Merge pull request #827 from matrix-org/markjh/more_slaved_methods | Mark Haines | 2016-06-03 | 3 | -6/+81 |
|\ | | | | | Add methods to events, account data and receipt slaves | ||||
| * | Add methods to events, account data and receipt slaves | Mark Haines | 2016-06-03 | 3 | -6/+81 |
| | | | | | | | | | | Adds the methods needed by /sync to the slaved events, account data and receipt stores. | ||||
* | | Merge pull request #824 from matrix-org/markjh/slaved_presence_store | Mark Haines | 2016-06-03 | 1 | -0/+59 |
|\ \ | | | | | | | Add a slaved store for presence | ||||
| * | | Add a slaved store for presence | Mark Haines | 2016-06-03 | 1 | -0/+59 |
| |/ | |||||
* | | Add a comment explaining why the filter cache doesn't need exipiring | Mark Haines | 2016-06-03 | 1 | -0/+1 |
| | | |||||
* | | Add slaved stores for filters, tokens, and push rules | Mark Haines | 2016-06-03 | 4 | -0/+151 |
|/ | |||||
* | Allow external processes to mark a user as syncing. (#812) | Mark Haines | 2016-06-02 | 2 | -0/+61 |
| | | | | | | | | | | | | * Add infrastructure to the presence handler to track sync requests in external processes * Expire stale entries for dead external processes * Add an http endpoint for making users as syncing Add some docstrings and comments. * Fixes | ||||
* | Move typing handler out of the Handlers object | Mark Haines | 2016-05-17 | 1 | -1/+1 |
| | |||||
* | Move the presence handler out of the Handlers object | Mark Haines | 2016-05-16 | 1 | -1/+1 |
| | |||||
* | Assert that stream replicated stream positions are ints | Mark Haines | 2016-05-13 | 3 | -7/+7 |
| | |||||
* | Merge branch 'develop' into dbkr/email_notifs_on_pusher | Mark Haines | 2016-05-13 | 3 | -1/+64 |
|\ | |||||
| * | Add a slaved datastore for account data | Mark Haines | 2016-05-13 | 1 | -0/+61 |
| | | |||||
| * | Log the stream IDs in an order that makes sense | Mark Haines | 2016-05-13 | 1 | -1/+1 |
| | | |||||
| * | Make sure we advance our stream position | Mark Haines | 2016-05-13 | 1 | -0/+2 |
| | | |||||
* | | Shift some of the state_group methods into the SlavedEventStore | Mark Haines | 2016-05-13 | 1 | -0/+19 |
|/ | |||||
* | Pass through get_events to pusher too | David Baker | 2016-05-10 | 1 | -0/+1 |
| | |||||
* | Pass through _get_event_txn | David Baker | 2016-05-10 | 1 | -0/+1 |
| | |||||
* | Don't warn | Erik Johnston | 2016-05-05 | 1 | -2/+0 |
| | |||||
* | Add some log information at returned replication streams | Erik Johnston | 2016-05-05 | 1 | -0/+11 |
| | |||||
* | Report per request metrics for all of the things using request_handler | Mark Haines | 2016-04-28 | 2 | -2/+4 |
| | |||||
* | Fix backfill replication to advance the stream correctly | Mark Haines | 2016-04-27 | 2 | -3/+3 |
| | |||||
* | Merge pull request #743 from matrix-org/markjh/slave_pushers | Mark Haines | 2016-04-21 | 1 | -0/+52 |
|\ | | | | | Replicate the pushers | ||||
| * | Replicate the pushers | Mark Haines | 2016-04-21 | 1 | -0/+52 |
| | |