Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Store room version on invite (#6983) | Richard van der Hoff | 2020-02-26 | 2 | -2/+36 |
| | | | | | When we get an invite over federation, store the room version in the rooms table. The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere... | ||||
* | Port PresenceHandler to async/await (#6991) | Erik Johnston | 2020-02-26 | 1 | -1/+5 |
| | |||||
* | Merge worker apps into one. (#6964) | Erik Johnston | 2020-02-25 | 1 | -0/+20 |
| | |||||
* | Increase MAX_EVENTS_BEHIND for replication clients | Erik Johnston | 2020-02-21 | 1 | -1/+1 |
| | |||||
* | Allow moving group read APIs to workers (#6866) | Erik Johnston | 2020-02-07 | 1 | -8/+6 |
| | |||||
* | Fix sending server up commands from workers (#6811) | Erik Johnston | 2020-01-30 | 1 | -0/+4 |
| | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> | ||||
* | Detect unknown remote devices and mark cache as stale (#6776) | Erik Johnston | 2020-01-28 | 1 | -1/+1 |
| | | | | We just mark the fact that the cache may be stale in the database for now. | ||||
* | Propagate cache invalidates from workers to other workers. (#6748) | Erik Johnston | 2020-01-27 | 2 | -4/+7 |
| | | | Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers. | ||||
* | Allow streaming cache invalidate all to workers. (#6749) | Erik Johnston | 2020-01-22 | 2 | -6/+27 |
| | |||||
* | Wake up transaction queue when remote server comes back online (#6706) | Erik Johnston | 2020-01-17 | 4 | -0/+44 |
| | | | | | This will be used to retry outbound transactions to a remote server if we think it might have come back up. | ||||
* | Port synapse.replication.tcp to async/await (#6666) | Erik Johnston | 2020-01-16 | 5 | -85/+63 |
| | | | | | | | | | | * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing.... | ||||
* | Add `local_current_membership` table (#6655) | Erik Johnston | 2020-01-15 | 1 | -1/+1 |
| | | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information. | ||||
* | Fixup synapse.replication to pass mypy checks (#6667) | Erik Johnston | 2020-01-14 | 10 | -86/+103 |
| | |||||
* | Reduce the reconnect time when replication fails. (#6617) | Richard van der Hoff | 2020-01-03 | 1 | -1/+2 |
| | |||||
* | Change EventContext to use the Storage class (#6564) | Erik Johnston | 2019-12-20 | 2 | -2/+6 |
| | |||||
* | Change DataStores to accept 'database' param. | Erik Johnston | 2019-12-06 | 13 | -26/+39 |
| | |||||
* | _CURRENT_STATE_CACHE_NAME is public | Erik Johnston | 2019-12-04 | 1 | -2/+2 |
| | |||||
* | Move cache invalidation to main data store | Erik Johnston | 2019-12-04 | 1 | -1/+2 |
| | |||||
* | Propagate reason in remotely rejected invites | Erik Johnston | 2019-11-28 | 1 | -2/+5 |
| | |||||
* | Prevent account_data content from being sent over TCP replication (#6333) | Andrew Morgan | 2019-11-26 | 1 | -4/+3 |
|\ | |||||
| * | lint | Andrew Morgan | 2019-11-08 | 1 | -2/+1 |
| | | |||||
| * | Remove content from being sent for account data rdata stream | Andrew Morgan | 2019-11-08 | 1 | -3/+3 |
| | | |||||
* | | Merge pull request #6332 from matrix-org/erikj/query_devices_fix | Erik Johnston | 2019-11-26 | 2 | -1/+82 |
|\ \ | |/ |/| | Fix caching devices for remote servers in worker. | ||||
| * | Fixup docs | Erik Johnston | 2019-11-26 | 1 | -1/+5 |
| | | |||||
| * | Fix caching devices for remote servers in worker. | Erik Johnston | 2019-11-05 | 2 | -1/+78 |
| | | | | | | | | | | | | | | | | When the `/keys/query` API is hit on client_reader worker Synapse may decide that it needs to resync some remote deivces. Usually this happens on master, and then gets cached. However, that fails on workers and so it falls back to fetching devices from remotes directly, which may in turn fail if the remote is down. | ||||
* | | Address review comments | Andrew Morgan | 2019-11-06 | 1 | -1/+1 |
| | | |||||
* | | Don't forget to ratelimit calls outside of RegistrationHandler | Andrew Morgan | 2019-11-06 | 1 | -0/+2 |
|/ | |||||
* | document the REPLICATE command a bit better (#6305) | Richard van der Hoff | 2019-11-04 | 3 | -9/+95 |
| | | | | since I found myself wonder how it works | ||||
* | Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify | Hubert Chathi | 2019-10-31 | 4 | -4/+4 |
|\ | |||||
| * | Remove usage of deprecated logger.warn method from codebase (#6271) | Andrew Morgan | 2019-10-31 | 4 | -4/+4 |
| | | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated. | ||||
* | | clean up code a bit | Hubert Chathi | 2019-10-31 | 1 | -5/+9 |
| | | |||||
* | | make user signatures a separate stream | Hubert Chathi | 2019-10-30 | 3 | -2/+25 |
| | | |||||
* | | Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify | Hubert Chathi | 2019-10-30 | 6 | -44/+26 |
|\| | |||||
| * | Port replication http server endpoints to async/await | Erik Johnston | 2019-10-29 | 6 | -44/+26 |
| | | |||||
* | | make notification of signatures work with workers | Hubert Chathi | 2019-10-24 | 1 | -0/+1 |
|/ | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2019-10-22 | 1 | -0/+3 |
|\ | | | | | | | erikj/refactor_stores | ||||
| * | Merge branch 'develop' into uhoreg/e2e_cross-signing_merged | Hubert Chathi | 2019-09-07 | 1 | -10/+11 |
| |\ | |||||
| * | | add user signature stream change cache to slaved device store | Hubert Chathi | 2019-09-04 | 1 | -0/+3 |
| | | | |||||
* | | | Move storage classes into a main "data store". | Erik Johnston | 2019-10-21 | 17 | -27/+29 |
| |/ |/| | | | | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage. | ||||
* | | Trace how long it takes for the send trasaction to complete, including ↵ | Jorik Schellekens | 2019-09-05 | 1 | -1/+6 |
| | | | | | | | | retrys (#5986) | ||||
* | | Add opentracing to all client servlets (#5983) | Jorik Schellekens | 2019-09-05 | 1 | -10/+6 |
|/ | |||||
* | Remove bind_email and bind_msisdn (#5964) | Andrew Morgan | 2019-09-04 | 1 | -18/+3 |
| | | | Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107). | ||||
* | Remove unnecessary parentheses around return statements (#5931) | Andrew Morgan | 2019-08-30 | 6 | -15/+15 |
| | | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :) | ||||
* | Opentracing across workers (#5771) | Jorik Schellekens | 2019-08-22 | 1 | -2/+14 |
| | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet | ||||
* | Revert "Add "require_consent" parameter for registration" | Brendan Abolivier | 2019-08-22 | 1 | -2/+0 |
| | | | | This reverts commit 3320aaab3a9bba3f5872371aba7053b41af9d0a0. | ||||
* | Add "require_consent" parameter for registration | Half-Shot | 2019-08-22 | 1 | -0/+2 |
| | |||||
* | Merge tag 'v1.2.0rc2' into develop | Andrew Morgan | 2019-07-24 | 1 | -1/+1 |
|\ | | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734)) | ||||
| * | Fix servlet metric names (#5734) | Jorik Schellekens | 2019-07-24 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths | ||||
* | | Replace returnValue with return (#5736) | Amber Brown | 2019-07-23 | 8 | -20/+20 |
|/ | |||||
* | Remove access-token support from RegistrationHandler.register (#5641) | Richard van der Hoff | 2019-07-08 | 1 | -6/+0 |
| | | | | | | | | Nothing uses this now, so we can remove the dead code, and clean up the API. Since we're changing the shape of the return value anyway, we take the opportunity to give the method a better name. | ||||
* | Remove support for invite_3pid_guest. (#5625) | Richard van der Hoff | 2019-07-05 | 1 | -65/+0 |
| | | | | | | | | | This has never been documented, and I'm not sure it's ever been used outside sytest. It's quite a lot of poorly-maintained code, so I'd like to get rid of it. For now I haven't removed the database table; I suggest we leave that for a future clearout. | ||||
* | Move logging utilities out of the side drawer of util/ and into logging/ (#5606) | Amber Brown | 2019-07-04 | 1 | -1/+1 |
| | |||||
* | Run Black. (#5482) | Amber Brown | 2019-06-20 | 26 | -355/+357 |
| | |||||
* | Handle failing to talk to master over replication | Erik Johnston | 2019-06-07 | 1 | -1/+9 |
| | |||||
* | Fixup bsaed on review comments | Erik Johnston | 2019-05-17 | 1 | -1/+1 |
| | |||||
* | Add basic editing support | Erik Johnston | 2019-05-16 | 1 | -0/+1 |
| | |||||
* | Fix relations in worker mode | Erik Johnston | 2019-05-16 | 3 | -8/+17 |
| | |||||
* | Replace SlavedKeyStore with a shim | Richard van der Hoff | 2019-04-08 | 1 | -14/+4 |
| | | | | | since we're pulling everything out of KeyStore anyway, we may as well simplify it. | ||||
* | Remove unused server_tls_certificates functions (#5028) | Richard van der Hoff | 2019-04-08 | 1 | -3/+0 |
| | | | | These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again. | ||||
* | Remove presence lists (#4989) | Neil Johnson | 2019-04-03 | 1 | -10/+0 |
| | | | Remove presence list support as per MSC 1819 | ||||
* | Fix sync bug when accepting invites (#4956) | Richard van der Hoff | 2019-04-02 | 1 | -9/+22 |
| | | | | | | | | | | Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream. | ||||
* | Combine the CurrentStateDeltaStream into the EventStream | Richard van der Hoff | 2019-03-27 | 3 | -23/+33 |
| | |||||
* | Make EventStream rows have a type | Richard van der Hoff | 2019-03-27 | 3 | -16/+94 |
| | | | | ... as a precursor to combining it with the CurrentStateDelta stream. | ||||
* | Skip building a ROW_TYPE when building updates | Richard van der Hoff | 2019-03-27 | 1 | -2/+2 |
| | | | | | We're about to turn it straight into a JSON object anyway so building a ROW_TYPE is a bit pointless, and reduces flexibility in the update_function. | ||||
* | Add parse_row method to replication stream class | Richard van der Hoff | 2019-03-27 | 3 | -3/+19 |
| | | | | This will allow individual stream classes to override how a row is parsed. | ||||
* | move FederationStream out to its own file | Richard van der Hoff | 2019-03-27 | 4 | -23/+43 |
| | |||||
* | move EventsStream out to its own file | Richard van der Hoff | 2019-03-27 | 3 | -23/+42 |
| | |||||
* | Move replication.tcp.streams into a package | Richard van der Hoff | 2019-03-27 | 2 | -33/+51 |
| | |||||
* | Fix/improve some docstrings in the replication code. (#4949) | Richard van der Hoff | 2019-03-27 | 2 | -7/+19 |
| | |||||
* | Fix ClientReplicationStreamProtocol.__str__ (#4929) | Richard van der Hoff | 2019-03-25 | 2 | -4/+5 |
| | | | | | | | | `__str__` depended on `self.addr`, which was absent from ClientReplicationStreamProtocol, so attempting to call str on such an object would raise an exception. We can calculate the peer addr from the transport, so there is no need for addr anyway. | ||||
* | Fix bug where read-receipts lost their timestamps (#4927) | Richard van der Hoff | 2019-03-25 | 2 | -11/+27 |
| | | | | | Make sure that they are sent correctly over the replication stream. Fixes: #4898 | ||||
* | Add a config option for torture-testing worker replication. (#4902) | Richard van der Hoff | 2019-03-20 | 1 | -1/+17 |
| | | | Setting this to 50 or so makes a bunch of sytests fail in worker mode. | ||||
* | Prefill client IPs cache on workers | Erik Johnston | 2019-03-06 | 1 | -0/+2 |
| | |||||
* | Merge pull request #4792 from matrix-org/anoa/replication_tokens | Andrew Morgan | 2019-03-06 | 1 | -3/+28 |
|\ | | | | | Support batch updates in the worker sender | ||||
| * | Simplify token replication logic | Andrew Morgan | 2019-03-05 | 1 | -23/+14 |
| | | |||||
| * | Clean up logic and add comments | Andrew Morgan | 2019-03-04 | 1 | -11/+18 |
| | | |||||
| * | Clearer branching, fix missing list clear | Andrew Morgan | 2019-03-04 | 1 | -4/+11 |
| | | |||||
| * | Prevent replication wedging | Andrew Morgan | 2019-03-04 | 1 | -4/+24 |
| | | |||||
* | | Add rate-limiting on registration (#4735) | Brendan Abolivier | 2019-03-05 | 1 | -2/+6 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Rate-limiting for registration * Add unit test for registration rate limiting * Add config parameters for rate limiting on auth endpoints * Doc * Fix doc of rate limiting function Co-Authored-By: babolivier <contact@brendanabolivier.com> * Incorporate review * Fix config parsing * Fix linting errors * Set default config for auth rate limiting * Fix tests * Add changelog * Advance reactor instead of mocked clock * Move parameters to registration specific config and give them more sensible default values * Remove unused config options * Don't mock the rate limiter un MAU tests * Rename _register_with_store into register_with_store * Make CI happy * Remove unused import * Update sample config * Fix ratelimiting test for py2 * Add non-guest test | ||||
* | | Fixup slave stores | Erik Johnston | 2019-03-04 | 3 | -36/+26 |
|/ | |||||
* | When presence is enabled don't send over replication | Erik Johnston | 2019-02-27 | 1 | -2/+5 |
| | |||||
* | Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff | Erik Johnston | 2019-02-27 | 3 | -5/+39 |
|\ | | | | | Fix tightloop over connecting to replication server | ||||
| * | Move connecting logic into ClientReplicationStreamProtocol | Erik Johnston | 2019-02-27 | 2 | -18/+17 |
| | | |||||
| * | Increase the max delay between retry attempts | Erik Johnston | 2019-02-26 | 1 | -1/+1 |
| | | | | | | | | | | Otherwise if you have many workers they can easily take out master with their connection attempts | ||||
| * | Fix tightloop over connecting to replication server | Erik Johnston | 2019-02-26 | 2 | -4/+39 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the client failed to process incoming commands during the initial set up of the replication connection it would immediately disconnect and reconnect, resulting in a tightloop. This can happen, for example, when subscribing to a stream that has a row that is too long in the backlog. The fix here is to not consider the connection successfully set up until the client has succesfully subscribed and caught up with the streams. This ensures that the retry logic timers aren't reset until then, meaning that if an error does happen during start up the client will continue backing off before retrying again. | ||||
* | | Limit cache invalidation replication line length (#4748) | Erik Johnston | 2019-02-27 | 1 | -1/+16 |
|/ | |||||
* | Fix state cache invalidation on workers | Erik Johnston | 2019-02-22 | 1 | -6/+1 |
| | |||||
* | Fix registration on workers (#4682) | Erik Johnston | 2019-02-20 | 3 | -3/+58 |
| | | | | | | | | | | * Move RegistrationHandler init to HomeServer * Move post registration actions to RegistrationHandler * Add post regisration replication endpoint * Newsfile | ||||
* | Batch cache invalidation over replication | Erik Johnston | 2019-02-18 | 1 | -7/+12 |
| | | | | | | | | | | Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic. | ||||
* | Move register_device into handler | Erik Johnston | 2019-02-18 | 1 | -14/+3 |
| | |||||
* | Split out registration to worker | Erik Johnston | 2019-02-18 | 3 | -1/+179 |
| | | | | | | | | This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker. | ||||
* | Fix replication for room v3 (#4523) | Erik Johnston | 2019-01-30 | 1 | -1/+4 |
| | | | | | | | | | * Fix replication for room v3 We were not correctly quoting the path fragments over http replication, which meant that it exploded when the event IDs had a slash in them * Newsfile | ||||
* | Fix receiving events from federation via a worker | Erik Johnston | 2019-01-29 | 1 | -1/+1 |
| | | | | This bug was introduced in PR #4470, commit 678a92cb56d547dcadffa723e29b4855a27d0901 | ||||
* | Replace missed usages of FrozenEvent | Erik Johnston | 2019-01-25 | 2 | -4/+12 |
| | |||||
* | Revert "Require event format version to parse or create events" | Erik Johnston | 2019-01-25 | 2 | -12/+4 |
| | |||||
* | Replace missed usages of FrozenEvent | Erik Johnston | 2019-01-24 | 2 | -4/+12 |
| | |||||
* | Don't truncate command name in metrics | Erik Johnston | 2018-10-29 | 1 | -2/+2 |
| | |||||
* | Make the replication logger quieter (#4108) | Amber Brown | 2018-10-29 | 1 | -1/+1 |
| | |||||
* | Make workers work on Py3 (#4027) | Amber Brown | 2018-10-13 | 6 | -30/+30 |
| | |||||
* | Fix minor typo in exception | Travis Ralston | 2018-09-13 | 1 | -1/+1 |
| | |||||
* | merge (#3576) | Amber Brown | 2018-09-14 | 1 | -7/+16 |
| | |||||
* | Remove conn_id | Erik Johnston | 2018-09-04 | 1 | -2/+2 |
| | |||||
* | Remove conn_id from repl prometheus metrics | Erik Johnston | 2018-09-03 | 1 | -10/+10 |
| | | | | | `conn_id` gets set to a random string, and so we end up filling up prometheus with tonnes of data series, which is bad. | ||||
* | Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging | Erik Johnston | 2018-08-20 | 1 | -1/+1 |
|\ | | | | | Fix logging bug in EDU handling over replication | ||||
| * | Fix logging bug in EDU handling over replication | Erik Johnston | 2018-08-17 | 1 | -1/+1 |
| | | |||||
* | | Logcontexts for replication command handlers | Richard van der Hoff | 2018-08-17 | 3 | -15/+43 |
|/ | | | | | | | | | | Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky. | ||||
* | Use federation handler function rather than duplicate | Erik Johnston | 2018-08-15 | 1 | -41/+3 |
| | | | | This involves renaming _persist_events to be a public function. | ||||
* | Rename slave TransactionStore to SlaveTransactionStore | Erik Johnston | 2018-08-15 | 1 | -1/+1 |
| | |||||
* | Move clean_room_for_join to master | Erik Johnston | 2018-08-09 | 1 | -0/+35 |
| | |||||
* | Fixup doc comments | Erik Johnston | 2018-08-09 | 1 | -0/+17 |
| | |||||
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-09 | 4 | -17/+63 |
|\ | | | | | | | erikj/split_federation | ||||
| * | Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet | Erik Johnston | 2018-08-09 | 3 | -243/+374 |
| |\ | | | | | | | Add helper base class for generating new replication endpoints | ||||
| | * | Fixup wording and remove dead code | Erik Johnston | 2018-08-09 | 1 | -2/+1 |
| | | | |||||
| | * | Rename POST param to METHOD | Erik Johnston | 2018-08-08 | 2 | -13/+22 |
| | | | |||||
| | * | Fixup logging and docstrings | Erik Johnston | 2018-08-08 | 2 | -2/+40 |
| | | | |||||
| * | | Basic support for room versioning | Richard van der Hoff | 2018-08-03 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | | | | | | | | | This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event | ||||
* | | | Import all functions from TransactionStore | Erik Johnston | 2018-08-06 | 1 | -11/+2 |
| | | | |||||
* | | | Add EDU/query handling over replication | Erik Johnston | 2018-08-06 | 1 | -1/+1 |
| | | | |||||
* | | | Add replication APIs for persisting federation events | Erik Johnston | 2018-08-06 | 2 | -1/+247 |
| |/ |/| | |||||
* | | Fix isort | Erik Johnston | 2018-08-06 | 1 | -4/+1 |
| | | |||||
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-08-03 | 1 | -4/+3 |
|\| | | | | | | | erikj/refactor_repl_servlet | ||||
| * | Kill off MatrixCodeMessageException | Richard van der Hoff | 2018-08-01 | 2 | -16/+12 |
| | | | | | | | | | | | | | | | | | | | | | | This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error. | ||||
* | | Use new helper base class for membership requests | Erik Johnston | 2018-07-31 | 1 | -171/+91 |
| | | |||||
* | | Use new helper base class for ReplicationSendEventRestServlet | Erik Johnston | 2018-07-31 | 1 | -79/+36 |
| | | |||||
* | | Add helper base class for generating new replication endpoints | Erik Johnston | 2018-07-31 | 1 | -0/+208 |
|/ | | | | | This will hopefully reduce the boiler plate required to implement new internal HTTP requests. | ||||
* | Fix unit tests | Richard van der Hoff | 2018-07-25 | 1 | -1/+1 |
| | | | | | | on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival. | ||||
* | Wrap a number of things that run in the background | Richard van der Hoff | 2018-07-25 | 1 | -6/+8 |
| | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics. | ||||
* | Fix missing attributes on workers. | Erik Johnston | 2018-07-23 | 1 | -2/+5 |
| | | | | | This was missed during the transition from attribute to getter for getting state from context. | ||||
* | Use stream cache in get_linearized_receipts_for_room | Erik Johnston | 2018-07-10 | 1 | -1/+1 |
| | | | | | This avoids us from uncessarily hitting the database when there has been no change for the room | ||||
* | run isort | Amber Brown | 2018-07-09 | 24 | -66/+91 |
| | |||||
* | Attempt to be more performant on PyPy (#3462) | Amber Brown | 2018-06-28 | 1 | -6/+10 |
| | |||||
* | Merge pull request #3441 from matrix-org/erikj/redo_erasure | Erik Johnston | 2018-06-25 | 1 | -0/+2 |
|\ | | | | | Fix user erasure and re-enable | ||||
| * | Add UserErasureWorkerStore to workers | Erik Johnston | 2018-06-25 | 1 | -0/+2 |
| | | |||||
* | | Remove all global reactor imports & pass it around explicitly (#3424) | Amber Brown | 2018-06-25 | 2 | -5/+5 |
|/ | |||||
* | Pass around the reactor explicitly (#3385) | Amber Brown | 2018-06-22 | 1 | -3/+3 |
| | |||||
* | Fix tcp protocol metrics naming (#3410) | Amber Brown | 2018-06-21 | 1 | -18/+35 |
| | |||||
* | Fix replication metrics | Richard van der Hoff | 2018-06-04 | 1 | -2/+2 |
| | | | | fix bug introduced in #3256 | ||||
* | Merge remote-tracking branch 'origin/develop' into 3218-official-prom | Amber Brown | 2018-05-28 | 2 | -8/+9 |
|\ | |||||
| * | Merge pull request #3244 from NotAFile/py3-six-4 | Amber Brown | 2018-05-24 | 2 | -5/+7 |
| |\ | | | | | | | replace some iteritems with six | ||||
| | * | replace some iteritems with six | Adrian Tschira | 2018-05-19 | 2 | -5/+7 |
| | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | | more cleanup | Amber Brown | 2018-05-22 | 2 | -6/+10 |
| | | | |||||
* | | | fix the test failures | Amber Brown | 2018-05-22 | 1 | -1/+1 |
| | | | |||||
* | | | cleanups, self-registration | Amber Brown | 2018-05-22 | 1 | -4/+5 |
| | | | |||||
* | | | Merge remote-tracking branch 'origin/develop' into 3218-official-prom | Amber Brown | 2018-05-22 | 1 | -0/+2 |
|\| | | |||||
| * | | Send users a server notice about consent | Richard van der Hoff | 2018-05-22 | 1 | -0/+2 |
| |/ | | | | | | | | | When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so. | ||||
* | | rest of the changes | Amber Brown | 2018-05-21 | 1 | -16/+14 |
| | | |||||
* | | replacing portions | Amber Brown | 2018-05-21 | 1 | -54/+34 |
|/ | |||||
* | make imports local | Adrian Tschira | 2018-04-28 | 2 | -4/+4 |
| | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | Refactor ResponseCache usage | Richard van der Hoff | 2018-04-12 | 1 | -12/+6 |
| | | | | | | | | | | | | | | | Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production. | ||||
* | Add metrics for ResponseCache | Richard van der Hoff | 2018-04-10 | 1 | -1/+1 |
| | |||||
* | Fix json encoding bug in replication | Richard van der Hoff | 2018-04-03 | 1 | -1/+1 |
| | | | | json encoders have an encode method, not a dumps method. | ||||
* | Use static JSONEncoders | Richard van der Hoff | 2018-03-29 | 1 | -3/+5 |
| | | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it. | ||||
* | Explicitly use simplejson | Erik Johnston | 2018-03-20 | 1 | -7/+7 |
| | |||||
* | Fix replication after switch to simplejson | Erik Johnston | 2018-03-19 | 1 | -2/+4 |
| | | | | | Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default. | ||||
* | Merge branch 'master' of github.com:matrix-org/synapse into develop | Erik Johnston | 2018-03-19 | 1 | -1/+1 |
|\ | |||||
| * | Replace ujson with simplejson | Erik Johnston | 2018-03-15 | 1 | -1/+1 |
| | | |||||
* | | Merge pull request #2992 from matrix-org/erikj/implement_member_workre | Erik Johnston | 2018-03-14 | 2 | -3/+336 |
|\ \ | | | | | | | Implement RoomMemberWorkerHandler | ||||
| * | | Fix imports | Erik Johnston | 2018-03-14 | 2 | -7/+4 |
| | | | |||||
| * | | s/join/joined/ in notify_user_membership_change | Erik Johnston | 2018-03-14 | 1 | -3/+3 |
| | | | |||||
| * | | Implement RoomMemberWorkerHandler | Erik Johnston | 2018-03-13 | 2 | -0/+336 |
| | | | |||||
* | | | Split up ProfileStore | Erik Johnston | 2018-03-13 | 1 | -0/+21 |
|/ / | |||||
* | | extra_users is actually a list of UserIDs | Erik Johnston | 2018-03-13 | 1 | -4/+4 |
| | | |||||
* | | Split Directory store | Erik Johnston | 2018-03-05 | 1 | -5/+3 |
| | | |||||
* | | Split registration store | Erik Johnston | 2018-03-02 | 1 | -15/+3 |
| | | |||||
* | | Merge pull request #2928 from matrix-org/erikj/read_marker_caches | Erik Johnston | 2018-03-01 | 1 | -1/+1 |
|\ \ | | | | | | | Fix typo in getting replication account data processing | ||||
| * | | Fix typo in getting replication account data processing | Erik Johnston | 2018-03-01 | 1 | -1/+1 |
| | | | |||||
* | | | Merge pull request #2925 from matrix-org/erikj/split_sig_fed | Erik Johnston | 2018-03-01 | 1 | -47/+9 |
|\ \ \ | | | | | | | | | Split out SignatureStore and EventFederationStore | ||||
| * | | | Remove unused DataStore | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| | | | | |||||
| * | | | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed | Erik Johnston | 2018-03-01 | 2 | -65/+11 |
| |\ \ \ | |||||
| * | | | | Split out SignatureStore and EventFederationStore | Erik Johnston | 2018-03-01 | 1 | -42/+8 |
| | | | | | |||||
* | | | | | Merge pull request #2927 from matrix-org/erikj/read_marker_caches | Erik Johnston | 2018-03-01 | 1 | -0/+4 |
|\ \ \ \ \ | |_|/ / / |/| | / / | | |/ / | |/| | | Improve caching for read_marker API | ||||
| * | | | Improve caching for read_marker API | Erik Johnston | 2018-03-01 | 1 | -0/+3 |
| | | | | | | | | | | | | | | | | | | | | | | | | We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data. | ||||
| * | | | Add some caches to help read marker API | Erik Johnston | 2018-03-01 | 1 | -0/+1 |
| |/ / | |||||
* | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-03-01 | 2 | -19/+4 |
|\ \ \ | | | | | | | | | | | | | erikj/split_stream_store | ||||
| * \ \ | Merge pull request #2923 from matrix-org/erikj/stream_ago_worker | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| |\ \ \ | | | | | | | | | | | Calculate stream_ordering_month_ago correctly on workers | ||||
| | * | | | Calculate stream_ordering_month_ago correctly on workers | Erik Johnston | 2018-03-01 | 1 | -1/+0 |
| | |/ / | |||||
| * / / | Split up RoomStore | Erik Johnston | 2018-03-01 | 1 | -17/+4 |
| |/ / | |||||
* / / | Split out stream store | Erik Johnston | 2018-03-01 | 1 | -45/+9 |
|/ / | |||||
* | | Log in the correct places | Erik Johnston | 2018-03-01 | 1 | -2/+4 |
| | | |||||
* | | Don't do preserve_fn for every request | Erik Johnston | 2018-03-01 | 1 | -1/+2 |
| | | |||||
* | | Add some logging | Erik Johnston | 2018-03-01 | 1 | -0/+2 |
| | | |||||
* | | Make repl send_event idempotent and retry on timeouts | Erik Johnston | 2018-03-01 | 1 | -6/+38 |
| | | | | | | | | | | | | If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event. | ||||
* | | Correctly send ratelimit and extra_users params | Erik Johnston | 2018-03-01 | 1 | -1/+13 |
| | | |||||
* | | Calculate push actions on worker | Erik Johnston | 2018-02-28 | 1 | -1/+1 |
| | | |||||
* | | Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation | Erik Johnston | 2018-02-27 | 1 | -0/+2 |
|\ \ | | | | | | | Fix missing invalidations for receipt storage | ||||
| * | | Fix missing invalidations for receipt storage | Erik Johnston | 2018-02-21 | 1 | -0/+2 |
| | | | |||||
* | | | Merge pull request #2903 from matrix-org/erikj/split_roommember_store | Erik Johnston | 2018-02-27 | 1 | -28/+5 |
|\ \ \ | | | | | | | | | Split out RoomMemberStore | ||||
| * \ \ | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-02-23 | 4 | -84/+23 |
| |\ \ \ | | | | | | | | | | | | | | | | erikj/split_roommember_store | ||||
| * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Split out RoomMemberStore | Erik Johnston | 2018-02-21 | 1 | -27/+3 |
| | | | | | |||||
* | | | | | Merge pull request #2901 from matrix-org/erikj/split_as_stores | Erik Johnston | 2018-02-27 | 1 | -28/+7 |
|\ \ \ \ \ | |_|/ / / |/| | | | | Split AS stores | ||||
| * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Split AS stores | Erik Johnston | 2018-02-21 | 1 | -28/+6 |
| |/ / / | |||||
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵ | Erik Johnston | 2018-02-23 | 4 | -74/+21 |
|\ \ \ \ | | | | | | | | | | | | | | | | erikj/split_event_push_actions | ||||
| * \ \ \ | Merge pull request #2902 from matrix-org/erikj/split_events_store | Erik Johnston | 2018-02-23 | 1 | -11/+3 |
| |\ \ \ \ | | | | | | | | | | | | | Split out get_events and co into a worker store | ||||
| | * | | | | Split EventsWorkerStore into separate file | Erik Johnston | 2018-02-23 | 1 | -1/+1 |
| | | | | | | |||||
| | * | | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | |/ / / | |||||
| | * / / | Split out get_events and co into a worker store | Erik Johnston | 2018-02-21 | 1 | -11/+2 |
| | |/ / | |||||
| * | | | Merge pull request #2899 from matrix-org/erikj/split_pushers | Erik Johnston | 2018-02-23 | 1 | -9/+3 |
| |\ \ \ | | | | | | | | | | | Split PusherStore | ||||
| | * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| | * | | | Split PusherStore | Erik Johnston | 2018-02-21 | 1 | -9/+2 |
| | |/ / | |||||
| * | | | Merge pull request #2898 from matrix-org/erikj/split_push_rules_store | Erik Johnston | 2018-02-23 | 1 | -17/+7 |
| |\ \ \ | | | | | | | | | | | Split PushRulesStore | ||||
| | * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | | |||||
| | * | | | Split PushRulesStore | Erik Johnston | 2018-02-21 | 1 | -17/+6 |
| | |/ / | |||||
| * | | | Update copyright | Erik Johnston | 2018-02-23 | 1 | -0/+1 |
| | | | | |||||
| * | | | Use absolute imports | Erik Johnston | 2018-02-23 | 1 | -2/+2 |
| | | | | |||||
| * | | | Split AccountDataStore and TagStore | Erik Johnston | 2018-02-21 | 1 | -34/+4 |
| |/ / | |||||
* / / | Split out EventPushActionWorkerStore | Erik Johnston | 2018-02-21 | 1 | -20/+3 |
|/ / | |||||
* | | Fix comment | Erik Johnston | 2018-02-20 | 1 | -1/+1 |
| | | |||||
* | | Use abstract base class to access stream IDs | Erik Johnston | 2018-02-20 | 1 | -2/+7 |
| | | |||||
* | | Split ReceiptsStore | Erik Johnston | 2018-02-20 | 1 | -28/+5 |
| | | |||||
* | | Don't serialize current state over replication | Erik Johnston | 2018-02-15 | 1 | -2/+2 |
| | | |||||
* | | Don't log errors propogated from send_event | Erik Johnston | 2018-02-15 | 1 | -1/+10 |
| | | |||||
* | | Add event_creator worker | Erik Johnston | 2018-02-07 | 1 | -0/+20 |
| | | |||||
* | | Add replication http endpoint for event sending | Erik Johnston | 2018-02-07 | 2 | -0/+139 |
| | | |||||
* | | Store state groups separately from events (#2784) | Erik Johnston | 2018-02-06 | 1 | -2/+2 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints | ||||
* | | Metrics for number of RDATA commands received | Richard van der Hoff | 2018-01-15 | 1 | -5/+14 |
|/ | | | | I found myself wishing we had this. | ||||
* | Fix some logcontext leaks in replication resource | Richard van der Hoff | 2017-11-23 | 1 | -2/+4 |
| | | | | | The @measure_func annotations rely on the wrapped function respecting the logcontext rules. Add the necessary yields to make this work. | ||||
* | Pull out bits of StateStore to a mixin | Richard van der Hoff | 2017-11-14 | 1 | -33/+6 |
| | | | | | | | | | ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655. | ||||
* | Make __init__ consitstent across Store heirarchy | Richard van der Hoff | 2017-11-13 | 1 | -1/+1 |
| | | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores) | ||||
* | replace 'except:' with 'except Exception:' | Richard van der Hoff | 2017-10-23 | 1 | -1/+1 |
| | | | | what could possibly go wrong | ||||
* | log when we get an exception handling replication updates | hera | 2017-10-12 | 1 | -1/+5 |
| | |||||
* | Fix replication. And notify | Erik Johnston | 2017-07-20 | 2 | -0/+74 |
| | |||||
* | Reduce log levels in tcp replication | Erik Johnston | 2017-07-11 | 1 | -2/+2 |
| | |||||
* | Serialize user ip command as json | Erik Johnston | 2017-06-27 | 1 | -5/+9 |
| | |||||
* | Fix up | Erik Johnston | 2017-06-27 | 1 | -2/+1 |
| | |||||
* | Make workers report to master for user ip updates | Erik Johnston | 2017-06-27 | 5 | -0/+103 |
| | |||||
* | Initialise exclusive_user_regex | Erik Johnston | 2017-06-21 | 1 | -0/+2 |
| | |||||
* | Fix for workers | Erik Johnston | 2017-06-21 | 1 | -0/+3 |
| | |||||
* | Initial worker impl | Erik Johnston | 2017-06-16 | 1 | -0/+22 |
| | |||||
* | Add cache for is_host_joined | Erik Johnston | 2017-06-13 | 1 | -1/+1 |
| | |||||
* | Merge pull request #2266 from matrix-org/erikj/host_in_room | Erik Johnston | 2017-06-12 | 1 | -1/+0 |
|\ | | | | | Change is_host_joined to use current_state table | ||||
| * | Fix replication | Erik Johnston | 2017-06-09 | 1 | -1/+0 |
| | | |||||
* | | Fix replication | Erik Johnston | 2017-06-09 | 1 | -1/+1 |
| | | |||||
* | | Add missing notifier | Erik Johnston | 2017-06-09 | 1 | -1/+2 |
|/ | |||||
* | Faster cache for get_joined_hosts | Erik Johnston | 2017-05-25 | 1 | -0/+2 |
| | |||||
* | Add missing storage function to slave store | Erik Johnston | 2017-05-22 | 1 | -0/+2 |
| | |||||
* | Add new storage functions to slave store | Erik Johnston | 2017-05-04 | 1 | -0/+3 |
| | |||||
* | Remove unused cache | Erik Johnston | 2017-04-24 | 1 | -3/+0 |
| | |||||
* | Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl | Erik Johnston | 2017-04-12 | 2 | -0/+11 |
|\ | | | | | Reduce federation replication traffic | ||||
| * | Move get_presence_list_* to SlaveStore | Erik Johnston | 2017-04-11 | 1 | -0/+10 |
| | | |||||
| * | Reduce federation presence replication traffic | Erik Johnston | 2017-04-10 | 1 | -0/+1 |
| | | | | | | | | | | | | | | | | This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination. | ||||
* | | Remove HTTP replication APIs | Erik Johnston | 2017-04-11 | 4 | -749/+0 |
|/ | |||||
* | Typo | Erik Johnston | 2017-04-10 | 1 | -1/+1 |
| | |||||
* | Merge pull request #2109 from matrix-org/erikj/send_queue_fix | Erik Johnston | 2017-04-10 | 1 | -2/+2 |
|\ | | | | | Fix up federation SendQueue and document types | ||||
| * | Comments | Erik Johnston | 2017-04-10 | 1 | -2/+2 |
| | | |||||
* | | Up replication ping timeout | Erik Johnston | 2017-04-10 | 1 | -2/+4 |
|/ | |||||
* | Merge pull request #2103 from matrix-org/erikj/no-double-encode | Erik Johnston | 2017-04-07 | 1 | -28/+76 |
|\ | | | | | Don't double encode replication data | ||||
| * | Document types of the replication streams | Erik Johnston | 2017-04-06 | 1 | -28/+76 |
| | | |||||
* | | Fix incorrect type when using InvalidateCacheCommand | Erik Johnston | 2017-04-06 | 1 | -1/+1 |
| | |