summary refs log tree commit diff
path: root/synapse/app/generic_worker.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Experimental Unix socket support (#15353)Jason Little2023-04-031-11/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add IReactorUNIX to ISynapseReactor type hint. * Create listen_unix(). Two options, 'path' to the file and 'mode' of permissions(not umask, recommend 666 as default as nginx/other reverse proxies write to it and it's setup as user www-data) For the moment, leave the option to always create a PID lockfile turned on by default * Create UnixListenerConfig and wire it up. Rename ListenerConfig to TCPListenerConfig, then Union them together into ListenerConfig. This spidered around a bit, but I think I got it all. Metrics and manhole have been placed behind a conditional in case of accidental putting them onto a unix socket. Use new helpers to get if a listener is configured for TLS, and to help create a site tag for logging. There are 2 TODO things in parse_listener_def() to finish up at a later point. * Refactor SynapseRequest to handle logging correctly when using a unix socket. This prevents an exception when an IP address can not be retrieved for a request. * Make the 'Synapse now listening on Unix socket' log line a little prettier. * No silent failures on generic workers when trying to use a unix socket with metrics or manhole. * Inline variables in app/_base.py * Update docstring for listen_unix() to remove reference to a hardcoded permission of 0o666 and add a few comments saying where the default IS declared. * Disallow both a unix socket and a ip/port combo on the same listener resource * Linting * Changelog * review: simplify how listen_unix returns(and get rid of a type: ignore) * review: fix typo from ConfigError in app/homeserver.py * review: roll conditional for http_options.tag into get_site_tag() helper(and add docstring) * review: enhance the conditionals for checking if a port or path is valid, remove a TODO line * review: Try updating comment in get_client_ip_if_available to clarify what is being retrieved and why * Pretty up how 'Synapse now listening on Unix Socket' looks by decoding the byte string. * review: In parse_listener_def(), raise ConfigError if neither socket_path nor port is declared(and fix a typo)
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-221-1/+0
|
* Enable Faster Remote Room Joins against worker-mode Synapse. (#14752)reivilibre2023-01-221-7/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Enable Complement tests for Faster Remote Room Joins on worker-mode * (dangerous) Add an override to allow Complement to use FRRJ under workers * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Fix race where we didn't send out replication notification * MORE HACKS * Fix get_un_partial_stated_rooms_token to take instance_name * Fix bad merge * Remove warning * Correctly advance un_partial_stated_room_stream * Fix merge * Add another notify_replication * Fixups * Create a separate ReplicationNotifier * Fix test * Fix portdb * Create a separate ReplicationNotifier * Fix test * Fix portdb * Fix presence test * Newsfile * Apply suggestions from code review * Update changelog.d/14752.misc Co-authored-by: Erik Johnston <erik@matrix.org> * lint Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Add listener `health` (#14747)Dirk Klimpel2023-01-111-0/+3
| | | Fixes: #8780
* Use ClientRestResource on both the main process and workers. (#14528)Patrick Cloke2022-12-021-72/+2
| | | | | | | Add logic to ClientRestResource to decide whether to mount servlets or not based on whether the current process is a worker. This is clearer to see what a worker runs than the completely separate / copy & pasted list of servlets being mounted for workers.
* Remove legacy Prometheus metrics names. They were deprecated in Synapse ↵reivilibre2022-11-241-1/+0
| | | | v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
* Remove need for `worker_main_http_uri` setting to use /keys/upload. (#14400)realtyem2022-11-161-101/+2
|
* Support using SSL on worker endpoints. (#14128)Tuomas Ojamies2022-11-151-21/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix missing SSL support in worker endpoints. * Add changelog * SSL for Replication endpoint * Remove unit test change * Refactor listener creation to reduce duplicated code * Fix the logger message * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add config documentation for new TLS option Co-authored-by: Tuomas Ojamies <tojamies@palantir.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Merge/remove `Slaved*` stores into `WorkerStores` (#14375)Nick Mills-Barrett2022-11-111-12/+32
|
* Improve `RawHeaders` type hints (#14303)David Robertson2022-10-281-4/+4
|
* Use servlets for /key/ endpoints. (#14229)Patrick Cloke2022-10-201-12/+8
| | | | | To fix the response for unknown endpoints under that prefix. See MSC3743.
* Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke2022-10-121-0/+2
| | | Co-authored-by: Eric Eastwood <erice@element.io>
* Add experimental configuration option to allow disabling legacy Prometheus ↵reivilibre2022-08-241-1/+5
| | | | | metric names. (#13540) Co-authored-by: David Robertson <davidr@element.io>
* Refuse to start when `faster_joins` is enabled on a worker deployment (#13531)Sean Quah2022-08-161-0/+7
| | | | | | Synapse does not currently support faster room joins on deployments with workers. Signed-off-by: Sean Quah <seanq@matrix.org>
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-211-14/+20
|
* Use cache store remove base slaved (#13329)Nick Mills-Barrett2022-07-211-2/+0
| | | This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-071-3/+2
|
* Remove groups replication code. (#12900)Patrick Cloke2022-05-311-2/+0
| | | | The replication logic for groups is no longer used, so the message passing infrastructure can be removed.
* Remove user-visible groups/communities code (#12553)Patrick Cloke2022-05-251-4/+0
| | | | | | | | | Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove: * All API endpoints (including Client-Server, Server-Server, and admin). * Documented configuration options (and the experimental flag, which is now unused). * Special handling during room upgrades. * The `groups` section of the `/sync` response.
* Add the `batch_send` endpoint to generic workers (#12868)Nick Mills-Barrett2022-05-251-0/+2
|
* Add the `/account/whoami` endpoint to generic workers (#12866)Nick Mills-Barrett2022-05-251-1/+2
|
* Add the `update_user_directory_from_worker` configuration option ↵reivilibre2022-05-101-16/+0
| | | | | (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. (#12654) Co-authored-by: Shay <hillerys@element.io>
* Add the `notify_appservices_from_worker` configuration option (superseding ↵reivilibre2022-05-061-16/+0
| | | | `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
* Move `update_client_ip` background job from the main process to the ↵reivilibre2022-04-011-2/+0
| | | | background worker. (#12251)
* Deprecate the groups/communities endpoints and add an experimental ↵Patrick Cloke2022-03-121-1/+2
| | | | configuration flag. (#12200)
* Rename get_tcp_replication to get_replication_command_handler. (#12192)Patrick Cloke2022-03-101-1/+1
| | | | | | Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-2/+3
| | | | * Require latest matrix-common * Use the common function
* Expose worker & homeserver as entrypoints in `setup.py` (#11449)Maximilian Bosch2021-11-301-1/+5
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Fix using MSC2716 batch sending with event persistence workers (#11220)Tulir Asokan2021-11-291-0/+2
| | | | Signed-off-by: Tulir Asokan <tulir@beeper.com>
* Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-171-2/+4
| | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-101-15/+19
|
* Support for serving server well-known files (#11211)Richard van der Hoff2021-11-011-0/+3
| | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-221-2/+2
|
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-1/+1
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-3/+3
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-2/+2
|
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-4/+4
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Document that /spaces & /hierarchy can be routed to workers. (#10648)Patrick Cloke2021-09-081-24/+18
| | | Also refactors some of the registration of endpoints on workers.
* Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-1/+4
|
* Persist room hierarchy pagination sessions to the database. (#10613)Patrick Cloke2021-08-241-0/+2
|
* Implement MSC3231: Token authenticated registration (#10142)Callum Brown2021-08-211-1/+5
| | | | | Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
* Remove the unused public_room_list_stream (#10565)Andrew Morgan2021-08-171-2/+2
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-171-20/+20
|
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-4/+2
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-1/+1
|
* Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+2
| | | This adds a simple best effort locking mechanism that works cross workers.
* Ensure that errors during startup are written to the logs and the console. ↵Richard van der Hoff2021-06-211-5/+17
| | | | | | | | (#10191) * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-0/+4
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Move some more endpoints off master (#10084)Erik Johnston2021-05-271-2/+2
|
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-2/+0
|
* Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-051-0/+1
| | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* Limit how often GC happens by time. (#9902)Erik Johnston2021-05-051-0/+3
| | | | | | | | | | | | | | | Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-5/+6
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-0/+1
|
* Split presence out of master (#9820)Erik Johnston2021-04-231-27/+4
|
* remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-141-1/+1
| | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-468/+2
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-061-1/+2
| | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
* Add type hints for the federation sender. (#9681)Patrick Cloke2021-03-291-7/+0
| | | | Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.
* Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff2021-03-261-7/+2
| | | This warning is somewhat confusing to users, so let's suppress it
* Fix lintBrendan Abolivier2021-03-191-4/+4
|
* Move support for MSC3026 behind an experimental flagBrendan Abolivier2021-03-181-1/+6
|
* Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-0/+1
|
* Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539)Richard van der Hoff2021-03-031-2/+13
| | | Should fix some remaining warnings
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-241-32/+0
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Fix deleting pushers when using sharded pushers. (#9465)Erik Johnston2021-02-221-3/+0
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-2/+1
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix SSO on workers (#9271)Richard van der Hoff2021-02-011-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-1/+14
|
* Register the /devices endpoint on workers. (#9092)Patrick Cloke2021-01-131-0/+2
|
* Move more encryption endpoints off master (#9068)Erik Johnston2021-01-111-2/+10
|
* Clean up exception handling in the startup code (#9059)Richard van der Hoff2021-01-111-4/+3
| | | | Factor out the exception handling in the startup code to a utility function, and fix the some logging and exit code stuff.
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-071-0/+3
|
* Allow redacting events on workers (#8994)Patrick Cloke2020-12-291-27/+4
| | | Adds the redacts endpoint to workers that have the client listener.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-1/+0
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Fix not sending events over federation when using sharded event persisters ↵Erik Johnston2020-10-141-4/+0
| | | | | | | | | | | | | | | | | (#8536) * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.
* Move additional tasks to the background worker (#8458)Patrick Cloke2020-10-071-0/+4
|
* Allow background tasks to be run on a separate worker. (#8369)Patrick Cloke2020-10-021-0/+4
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-3/+3
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-1/+1
|
* Change HomeServer definition to work with typing.Erik Johnston2020-08-111-3/+5
| | | | | | | | | | Duplicating function signatures between server.py and server.pyi is silly. This commit changes that by changing all `build_*` methods to `get_*` methods and changing the `_make_dependency_method` to work work as a descriptor that caches the produced value. There are some changes in other files that were made to fix the typing in server.py.
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+5
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-7/+7
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2020-07-301-3/+3
|\
| * Update worker docs with recent enhancements (#7969)Erik Johnston2020-07-291-3/+3
| |
* | Ensure that remove_pusher is always async (#7981)Patrick Cloke2020-07-301-1/+1
|/
* Fix typing replication not being handled on master (#7959)Erik Johnston2020-07-271-7/+0
| | | | | | | | | | | | | | | | Handling of incoming typing stream updates from replication was not hooked up on master, effecting set ups where typing was handled on a different worker. This is really only a problem if the master process is also handling sync requests, which is unlikely for those that are at the stage of moving typing off. The other observable effect is that if a worker restarts or a replication connect drops then the typing worker will issue a `POSITION typing`, triggering master process to try and stream *all* typing updates from position 0. Fixes #7907
* Convert synapse.app to async/await. (#7868)Patrick Cloke2020-07-171-7/+5
|
* Allow moving typing off master (#7869)Erik Johnston2020-07-161-34/+2
|
* Add ability to shard the federation sender (#7798)Erik Johnston2020-07-101-43/+16
|
* Pass original request headers from workers to the main process. (#7797)Patrick Cloke2020-07-091-5/+25
|
* Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-15/+21
| | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* Discard RDATA from already seen positions. (#7648)Patrick Cloke2020-06-151-0/+5
|
* Ensure we persist and ack the same tokenErik Johnston2020-05-271-5/+7
|
* Speed up processing of federation stream RDATA rows.Erik Johnston2020-05-271-2/+17
| | | | | | Instead of storing and sending an ACK for every single row we send synchronously, we instead do it asynchronously while batching up updates.
* Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563)Richard van der Hoff2020-05-261-1/+1
| | | Introduced in #7556
* Fix recording of federation stream token (#7564)Richard van der Hoff2020-05-261-24/+44
| | | | | | | | A couple of changes of significance: * remove the `_last_ack < federation_position` condition, so that updates will still be correctly processed after restart * Correctly wire up send_federation_ack to the right class.
* Add option to move event persistence off master (#7517)Erik Johnston2020-05-221-5/+48
|
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-3/+2
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-4/+9
| | | | These are business as usual errors, rather than stuff we want to log at error.
* Stub out GET presence requests in the frontend proxy (#7545)Richard van der Hoff2020-05-211-17/+4
| | | | | We don't really make any promises about returning accurate presence data when presence is disabled, so we may as well just return a static response, rather than making the master handle a request.
* Allow ReplicationRestResource to be added to workers (#7515)Erik Johnston2020-05-181-0/+4
| | | This allows workers to talk to each other over HTTP replication.
* Add a worker store for search insertion. (#7516)Erik Johnston2020-05-151-0/+2
| | | | | This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
* Move EventStream handling into default ReplicationDataHandler (#7493)Erik Johnston2020-05-141-31/+2
| | | This is so that the logic can happen on both master and workers when we move event persistence out.
* Allow censoring of events to happen on workers. (#7492)Erik Johnston2020-05-131-0/+2
| | | This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
* Thread through instance name to replication client. (#7369)Erik Johnston2020-05-011-6/+4
| | | For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
* Use `stream.current_token()` and remove `stream_positions()` (#7172)Erik Johnston2020-05-011-16/+0
| | | | We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
* Persist user interactive authentication sessions (#7302)Patrick Cloke2020-04-301-0/+2
| | | | | By persisting the user interactive authentication sessions to the database, this fixes situations where a user hits different works throughout their auth session and also allows sessions to persist through restarts of Synapse.
* Run replication streamers on workers (#7146)Erik Johnston2020-04-281-4/+9
| | | Currently we never write to streams from workers, but that will change soon
* Stop the master relaying USER_SYNC for other workers (#7318)Richard van der Hoff2020-04-221-36/+49
| | | | | | | Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication. In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits. Fixes (I hope) #7257.
* Merge branch 'release-v1.12.4' into developRichard van der Hoff2020-04-221-0/+6
|\
| * Support GET account_data requests on a worker (#7311)Richard van der Hoff2020-04-211-0/+6
| |
| * Fix starting workers when federation sending not split out.Erik Johnston2020-03-311-0/+12
| |
* | Move client command handling out of TCP protocol (#7185)Erik Johnston2020-04-061-6/+3
| | | | | | The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
* | Remove some `run_in_background` calls in replication code (#7203)Richard van der Hoff2020-04-031-8/+8
| | | | | | | | | | | | By running this stuff with `run_in_background`, it won't be correctly reported against the relevant CPU usage stats. Fixes #7202
* | Remove usage of "conn_id" for presence. (#7128)Erik Johnston2020-03-301-4/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `conn_id` usage for UserSyncCommand. Each tcp replication connection is assigned a "conn_id", which is used to give an ID to a remotely connected worker. In a redis world, there will no longer be a one to one mapping between connection and instance, so instead we need to replace such usages with an ID generated by the remote instances and included in the replicaiton commands. This really only effects UserSyncCommand. * Add CLEAR_USER_SYNCS command that is sent on shutdown. This should help with the case where a synchrotron gets restarted gracefully, rather than rely on 5 minute timeout.
* | Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-251-0/+3
| | | | | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* | Merge pull request #7133 from matrix-org/erikj/fix_worker_startupErik Johnston2020-03-251-0/+12
|\ \ | | | | | | Fix starting workers when federation sending not split out.
| * | Fix starting workers when federation sending not split out.Erik Johnston2020-03-241-0/+12
| |/
* | Convert `*StreamRow` classes to inner classes (#7116)Richard van der Hoff2020-03-231-1/+1
| | | | | | | | | | This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.
* | Fix processing of `groups` stream, and use symbolic names for streams (#7117)Richard van der Hoff2020-03-231-12/+23
| | | | | | | | | | | | `groups` != `receipts` Introduced in #6964
* | Comments from reviewErik Johnston2020-03-181-0/+3
| |
* | Fix worker handlingErik Johnston2020-03-021-2/+3
| |
* | Change device list replication to match new semantics.Erik Johnston2020-02-281-1/+1
|/ | | | | Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host).
* Don't refuse to start worker if media listener configured. (#7002)Erik Johnston2020-02-271-14/+20
| | | | | | | | Instead lets just warn if the worker has a media listener configured but has the media repository disabled. Previously non media repository workers would just ignore the media listener.
* Merge worker apps into one. (#6964)Erik Johnston2020-02-251-0/+917