diff options
Diffstat (limited to 'docs/workers.md')
| -rw-r--r-- | docs/workers.md | 284 |
1 files changed, 284 insertions, 0 deletions
diff --git a/docs/workers.md b/docs/workers.md new file mode 100644 index 0000000000..4bd60ba0a0 --- /dev/null +++ b/docs/workers.md @@ -0,0 +1,284 @@ +# Scaling synapse via workers + +Synapse has experimental support for splitting out functionality into +multiple separate python processes, helping greatly with scalability. These +processes are called 'workers', and are (eventually) intended to scale +horizontally independently. + +All of the below is highly experimental and subject to change as Synapse evolves, +but documenting it here to help folks needing highly scalable Synapses similar +to the one running matrix.org! + +All processes continue to share the same database instance, and as such, workers +only work with postgres based synapse deployments (sharing a single sqlite +across multiple processes is a recipe for disaster, plus you should be using +postgres anyway if you care about scalability). + +The workers communicate with the master synapse process via a synapse-specific +TCP protocol called 'replication' - analogous to MySQL or Postgres style +database replication; feeding a stream of relevant data to the workers so they +can be kept in sync with the main synapse process and database state. + +## Configuration + +To make effective use of the workers, you will need to configure an HTTP +reverse-proxy such as nginx or haproxy, which will direct incoming requests to +the correct worker, or to the main synapse instance. Note that this includes +requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) +for information on setting up a reverse proxy. + +To enable workers, you need to add two replication listeners to the master +synapse, e.g.: + + listeners: + # The TCP replication port + - port: 9092 + bind_address: '127.0.0.1' + type: replication + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] + +Under **no circumstances** should these replication API listeners be exposed to +the public internet; it currently implements no authentication whatsoever and is +unencrypted. + +(Roughly, the TCP port is used for streaming data from the master to the +workers, and the HTTP port for the workers to send data to the main +synapse process.) + +You then create a set of configs for the various worker processes. These +should be worker configuration files, and should be stored in a dedicated +subdirectory, to allow synctl to manipulate them. An additional configuration +for the master synapse process will need to be created because the process will +not be started automatically. That configuration should look like this: + + worker_app: synapse.app.homeserver + daemonize: true + +Each worker configuration file inherits the configuration of the main homeserver +configuration file. You can then override configuration specific to that worker, +e.g. the HTTP listener that it provides (if any); logging configuration; etc. +You should minimise the number of overrides though to maintain a usable config. + +You must specify the type of worker application (`worker_app`). The currently +available worker applications are listed below. You must also specify the +replication endpoints that it's talking to on the main synapse process. +`worker_replication_host` should specify the host of the main synapse, +`worker_replication_port` should point to the TCP replication listener port and +`worker_replication_http_port` should point to the HTTP replication port. + +Currently, the `event_creator` and `federation_reader` workers require specifying +`worker_replication_http_port`. + +For instance: + + worker_app: synapse.app.synchrotron + + # The replication listener on the synapse to talk to. + worker_replication_host: 127.0.0.1 + worker_replication_port: 9092 + worker_replication_http_port: 9093 + + worker_listeners: + - type: http + port: 8083 + resources: + - names: + - client + + worker_daemonize: True + worker_pid_file: /home/matrix/synapse/synchrotron.pid + worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml + +...is a full configuration for a synchrotron worker instance, which will expose a +plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided +by the main synapse. + +Obviously you should configure your reverse-proxy to route the relevant +endpoints to the worker (`localhost:8083` in the above example). + +Finally, to actually run your worker-based synapse, you must pass synctl the -a +commandline option to tell it to operate on all the worker configurations found +in the given directory, e.g.: + + synctl -a $CONFIG/workers start + +Currently one should always restart all workers when restarting or upgrading +synapse, unless you explicitly know it's safe not to. For instance, restarting +synapse without restarting all the synchrotrons may result in broken typing +notifications. + +To manipulate a specific worker, you pass the -w option to synctl: + + synctl -w $CONFIG/workers/synchrotron.yaml restart + +## Available worker applications + +### `synapse.app.pusher` + +Handles sending push notifications to sygnal and email. Doesn't handle any +REST endpoints itself, but you should set `start_pushers: False` in the +shared configuration file to stop the main synapse sending these notifications. + +Note this worker cannot be load-balanced: only one instance should be active. + +### `synapse.app.synchrotron` + +The synchrotron handles `sync` requests from clients. In particular, it can +handle REST endpoints matching the following regular expressions: + + ^/_matrix/client/(v2_alpha|r0)/sync$ + ^/_matrix/client/(api/v1|v2_alpha|r0)/events$ + ^/_matrix/client/(api/v1|r0)/initialSync$ + ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ + +The above endpoints should all be routed to the synchrotron worker by the +reverse-proxy configuration. + +It is possible to run multiple instances of the synchrotron to scale +horizontally. In this case the reverse-proxy should be configured to +load-balance across the instances, though it will be more efficient if all +requests from a particular user are routed to a single instance. Extracting +a userid from the access token is currently left as an exercise for the reader. + +### `synapse.app.appservice` + +Handles sending output traffic to Application Services. Doesn't handle any +REST endpoints itself, but you should set `notify_appservices: False` in the +shared configuration file to stop the main synapse sending these notifications. + +Note this worker cannot be load-balanced: only one instance should be active. + +### `synapse.app.federation_reader` + +Handles a subset of federation endpoints. In particular, it can handle REST +endpoints matching the following regular expressions: + + ^/_matrix/federation/v1/event/ + ^/_matrix/federation/v1/state/ + ^/_matrix/federation/v1/state_ids/ + ^/_matrix/federation/v1/backfill/ + ^/_matrix/federation/v1/get_missing_events/ + ^/_matrix/federation/v1/publicRooms + ^/_matrix/federation/v1/query/ + ^/_matrix/federation/v1/make_join/ + ^/_matrix/federation/v1/make_leave/ + ^/_matrix/federation/v1/send_join/ + ^/_matrix/federation/v1/send_leave/ + ^/_matrix/federation/v1/invite/ + ^/_matrix/federation/v1/query_auth/ + ^/_matrix/federation/v1/event_auth/ + ^/_matrix/federation/v1/exchange_third_party_invite/ + ^/_matrix/federation/v1/send/ + ^/_matrix/key/v2/query + +The above endpoints should all be routed to the federation_reader worker by the +reverse-proxy configuration. + +The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single +instance. + +### `synapse.app.federation_sender` + +Handles sending federation traffic to other servers. Doesn't handle any +REST endpoints itself, but you should set `send_federation: False` in the +shared configuration file to stop the main synapse sending this traffic. + +Note this worker cannot be load-balanced: only one instance should be active. + +### `synapse.app.media_repository` + +Handles the media repository. It can handle all endpoints starting with: + + /_matrix/media/ + +And the following regular expressions matching media-specific administration APIs: + + ^/_synapse/admin/v1/purge_media_cache$ + ^/_synapse/admin/v1/room/.*/media$ + ^/_synapse/admin/v1/quarantine_media/.*$ + +You should also set `enable_media_repo: False` in the shared configuration +file to stop the main synapse running background jobs related to managing the +media repository. + +Note this worker cannot be load-balanced: only one instance should be active. + +### `synapse.app.client_reader` + +Handles client API endpoints. It can handle REST endpoints matching the +following regular expressions: + + ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ + ^/_matrix/client/(api/v1|r0|unstable)/login$ + ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ + ^/_matrix/client/(api/v1|r0|unstable)/keys/query$ + ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ + ^/_matrix/client/versions$ + ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ + +Additionally, the following REST endpoints can be handled for GET requests: + + ^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$ + +Additionally, the following REST endpoints can be handled, but all requests must +be routed to the same instance: + + ^/_matrix/client/(r0|unstable)/register$ + +Pagination requests can also be handled, but all requests with the same path +room must be routed to the same instance. Additionally, care must be taken to +ensure that the purge history admin API is not used while pagination requests +for the room are in flight: + + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$ + +### `synapse.app.user_dir` + +Handles searches in the user directory. It can handle REST endpoints matching +the following regular expressions: + + ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$ + +### `synapse.app.frontend_proxy` + +Proxies some frequently-requested client endpoints to add caching and remove +load from the main synapse. It can handle REST endpoints matching the following +regular expressions: + + ^/_matrix/client/(api/v1|r0|unstable)/keys/upload + +If `use_presence` is False in the homeserver config, it can also handle REST +endpoints matching the following regular expressions: + + ^/_matrix/client/(api/v1|r0|unstable)/presence/[^/]+/status + +This "stub" presence handler will pass through `GET` request but make the +`PUT` effectively a no-op. + +It will proxy any requests it cannot handle to the main synapse instance. It +must therefore be configured with the location of the main instance, via +the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration +file. For example: + + worker_main_http_uri: http://127.0.0.1:8008 + +### `synapse.app.event_creator` + +Handles some event creation. It can handle REST endpoints matching: + + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send + ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ + ^/_matrix/client/(api/v1|r0|unstable)/join/ + ^/_matrix/client/(api/v1|r0|unstable)/profile/ + +It will create events locally and then send them on to the main synapse +instance to be persisted and handled. |