summary refs log tree commit diff
path: root/synapse/replication/tcp (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix sending server up commands from workers (#6811)Erik Johnston2020-01-301-0/+4
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Propagate cache invalidates from workers to other workers. (#6748)Erik Johnston2020-01-272-4/+7
| | | Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
* Allow streaming cache invalidate all to workers. (#6749)Erik Johnston2020-01-221-5/+21
|
* Wake up transaction queue when remote server comes back online (#6706)Erik Johnston2020-01-174-0/+44
| | | | | This will be used to retry outbound transactions to a remote server if we think it might have come back up.
* Port synapse.replication.tcp to async/await (#6666)Erik Johnston2020-01-165-85/+63
| | | | | | | | | | * Port synapse.replication.tcp to async/await * Newsfile * Correctly document type of on_<FOO> functions as async * Don't be overenthusiastic with the asyncing....
* Fixup synapse.replication to pass mypy checks (#6667)Erik Johnston2020-01-147-77/+93
|
* Reduce the reconnect time when replication fails. (#6617)Richard van der Hoff2020-01-031-1/+2
|
* lintAndrew Morgan2019-11-081-2/+1
|
* Remove content from being sent for account data rdata streamAndrew Morgan2019-11-081-3/+3
|
* document the REPLICATE command a bit better (#6305)Richard van der Hoff2019-11-042-8/+86
| | | | since I found myself wonder how it works
* Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notifyHubert Chathi2019-10-312-2/+2
|\
| * Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-312-2/+2
| | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* | make user signatures a separate streamHubert Chathi2019-10-302-0/+19
|/
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-4/+4
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Replace returnValue with return (#5736)Amber Brown2019-07-232-7/+7
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Run Black. (#5482)Amber Brown2019-06-207-183/+228
|
* Fix relations in worker modeErik Johnston2019-05-162-5/+7
|
* Combine the CurrentStateDeltaStream into the EventStreamRichard van der Hoff2019-03-273-23/+33
|
* Make EventStream rows have a typeRichard van der Hoff2019-03-272-14/+88
| | | | ... as a precursor to combining it with the CurrentStateDelta stream.
* Skip building a ROW_TYPE when building updatesRichard van der Hoff2019-03-271-2/+2
| | | | | We're about to turn it straight into a JSON object anyway so building a ROW_TYPE is a bit pointless, and reduces flexibility in the update_function.
* Add parse_row method to replication stream classRichard van der Hoff2019-03-273-3/+19
| | | | This will allow individual stream classes to override how a row is parsed.
* move FederationStream out to its own fileRichard van der Hoff2019-03-274-23/+43
|
* move EventsStream out to its own fileRichard van der Hoff2019-03-273-23/+42
|
* Move replication.tcp.streams into a packageRichard van der Hoff2019-03-272-33/+51
|
* Fix/improve some docstrings in the replication code. (#4949)Richard van der Hoff2019-03-272-7/+19
|
* Fix ClientReplicationStreamProtocol.__str__ (#4929)Richard van der Hoff2019-03-252-4/+5
| | | | | | | | `__str__` depended on `self.addr`, which was absent from ClientReplicationStreamProtocol, so attempting to call str on such an object would raise an exception. We can calculate the peer addr from the transport, so there is no need for addr anyway.
* Fix bug where read-receipts lost their timestamps (#4927)Richard van der Hoff2019-03-252-11/+27
| | | | | Make sure that they are sent correctly over the replication stream. Fixes: #4898
* Add a config option for torture-testing worker replication. (#4902)Richard van der Hoff2019-03-201-1/+17
| | | Setting this to 50 or so makes a bunch of sytests fail in worker mode.
* Simplify token replication logicAndrew Morgan2019-03-051-23/+14
|
* Clean up logic and add commentsAndrew Morgan2019-03-041-11/+18
|
* Clearer branching, fix missing list clearAndrew Morgan2019-03-041-4/+11
|
* Prevent replication wedgingAndrew Morgan2019-03-041-4/+24
|
* Merge pull request #4749 from matrix-org/erikj/replication_connection_backoffErik Johnston2019-02-273-5/+39
|\ | | | | Fix tightloop over connecting to replication server
| * Move connecting logic into ClientReplicationStreamProtocolErik Johnston2019-02-272-18/+17
| |
| * Increase the max delay between retry attemptsErik Johnston2019-02-261-1/+1
| | | | | | | | | | Otherwise if you have many workers they can easily take out master with their connection attempts
| * Fix tightloop over connecting to replication serverErik Johnston2019-02-262-4/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the client failed to process incoming commands during the initial set up of the replication connection it would immediately disconnect and reconnect, resulting in a tightloop. This can happen, for example, when subscribing to a stream that has a row that is too long in the backlog. The fix here is to not consider the connection successfully set up until the client has succesfully subscribed and caught up with the streams. This ensures that the retry logic timers aren't reset until then, meaning that if an error does happen during start up the client will continue backing off before retrying again.
* | Limit cache invalidation replication line length (#4748)Erik Johnston2019-02-271-1/+16
|/
* Don't truncate command name in metricsErik Johnston2018-10-291-2/+2
|
* Make the replication logger quieter (#4108)Amber Brown2018-10-291-1/+1
|
* Fix minor typo in exceptionTravis Ralston2018-09-131-1/+1
|
* Remove conn_idErik Johnston2018-09-041-2/+2
|
* Remove conn_id from repl prometheus metricsErik Johnston2018-09-031-10/+10
| | | | | `conn_id` gets set to a random string, and so we end up filling up prometheus with tonnes of data series, which is bad.
* Logcontexts for replication command handlersRichard van der Hoff2018-08-173-15/+43
| | | | | | | | | | Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky.
* Fix unit testsRichard van der Hoff2018-07-251-1/+1
| | | | | | on_notifier_poke no longer runs synchonously, so we have to do a different hack to make sure that the replication data has been sent. Let's actually listen for its arrival.
* Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-251-6/+8
| | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* run isortAmber Brown2018-07-094-30/+41
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-6/+10
|
* Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-252-5/+5
|
* Fix tcp protocol metrics naming (#3410)Amber Brown2018-06-211-18/+35
|
* Fix replication metricsRichard van der Hoff2018-06-041-2/+2
| | | | fix bug introduced in #3256
* Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-282-8/+9
|\
| * Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-242-5/+7
| |\ | | | | | | replace some iteritems with six
| | * replace some iteritems with sixAdrian Tschira2018-05-192-5/+7
| | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | more cleanupAmber Brown2018-05-222-6/+10
| | |
* | | fix the test failuresAmber Brown2018-05-221-1/+1
| | |
* | | cleanups, self-registrationAmber Brown2018-05-221-4/+5
| | |
* | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-221-0/+2
|\| |
| * | Send users a server notice about consentRichard van der Hoff2018-05-221-0/+2
| |/ | | | | | | | | When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.
* | rest of the changesAmber Brown2018-05-211-16/+14
| |
* | replacing portionsAmber Brown2018-05-211-54/+34
|/
* make imports localAdrian Tschira2018-04-282-4/+4
| | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* Fix json encoding bug in replicationRichard van der Hoff2018-04-031-1/+1
| | | | json encoders have an encode method, not a dumps method.
* Use static JSONEncodersRichard van der Hoff2018-03-291-3/+5
| | | | | using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.
* Explicitly use simplejsonErik Johnston2018-03-201-7/+7
|
* Fix replication after switch to simplejsonErik Johnston2018-03-191-2/+4
| | | | | Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default.
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+1
|\
| * Replace ujson with simplejsonErik Johnston2018-03-151-1/+1
| |
* | Metrics for number of RDATA commands receivedRichard van der Hoff2018-01-151-5/+14
|/ | | | I found myself wishing we had this.
* Fix some logcontext leaks in replication resourceRichard van der Hoff2017-11-231-2/+4
| | | | | The @measure_func annotations rely on the wrapped function respecting the logcontext rules. Add the necessary yields to make this work.
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* log when we get an exception handling replication updateshera2017-10-121-1/+5
|
* Fix replication. And notifyErik Johnston2017-07-201-0/+20
|
* Reduce log levels in tcp replicationErik Johnston2017-07-111-2/+2
|
* Serialize user ip command as jsonErik Johnston2017-06-271-5/+9
|
* Make workers report to master for user ip updatesErik Johnston2017-06-274-0/+55
|
* Initial worker implErik Johnston2017-06-161-0/+22
|
* Add missing notifierErik Johnston2017-06-091-1/+2
|
* TypoErik Johnston2017-04-101-1/+1
|
* Merge pull request #2109 from matrix-org/erikj/send_queue_fixErik Johnston2017-04-101-2/+2
|\ | | | | Fix up federation SendQueue and document types
| * CommentsErik Johnston2017-04-101-2/+2
| |
* | Up replication ping timeoutErik Johnston2017-04-101-2/+4
|/
* Merge pull request #2103 from matrix-org/erikj/no-double-encodeErik Johnston2017-04-071-28/+76
|\ | | | | Don't double encode replication data
| * Document types of the replication streamsErik Johnston2017-04-061-28/+76
| |
* | Fix incorrect type when using InvalidateCacheCommandErik Johnston2017-04-061-1/+1
| |
* | Add log linesErik Johnston2017-04-051-1/+2
| |
* | Rearrange metricsErik Johnston2017-04-051-16/+31
| |
* | Fix typoErik Johnston2017-04-051-2/+2
| |
* | Fixup some metrics for tcp replErik Johnston2017-04-051-0/+16
|/
* Merge pull request #2097 from matrix-org/erikj/repl_tcp_clientErik Johnston2017-04-051-0/+196
|\ | | | | Move to using TCP replication
| * Add basic replication client handler and factoryErik Johnston2017-04-031-0/+196
| |
* | Merge pull request #2098 from matrix-org/erikj/repl_tcp_fixErik Johnston2017-04-043-6/+15
|\ \ | | | | | | Advance replication streams even if nothing is listening
| * | Advance replication streams even if nothing is listeningErik Johnston2017-04-043-6/+15
| |/ | | | | | | | | | | Otherwise the streams don't advance and steadily fall behind, so when a worker does connect either a) they'll be streamed lots of old updates or b) the connection will fail as the streams are too far behind.
* / Fiddle tcp replication loggingErik Johnston2017-04-041-2/+2
|/
* Always advance stream tokensErik Johnston2017-04-031-1/+4
|
* Use callbacks to notify tcp replication rather than deferredsErik Johnston2017-03-311-14/+1
|
* Add a timestamp to USER_SYNC commandErik Johnston2017-03-313-9/+17
| | | | This timestamp is used to indicate when the user last sync'd
* Fix up docsErik Johnston2017-03-312-19/+3
|
* Add server side resource for tcp replicationErik Johnston2017-03-301-0/+300
|
* Initial TCP protocol implementationErik Johnston2017-03-303-0/+974
| | | | This defines the low level TCP replication protocol
* Define the various streams we will replicateErik Johnston2017-03-302-0/+423
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-11 23:17:19 +0000