Merge branch 'develop' into rav/saml_mapping_work

1 files changed, 284 insertions, 0 deletions

diff --git a/docs/workers.md b/docs/workers.md
new file mode 100644
index 0000000000..4bd60ba0a0
--- /dev/null
+++ b/docs/workers.md
@@ -0,0 +1,284 @@
+# Scaling synapse via workers
+
+Synapse has experimental support for splitting out functionality into
+multiple separate python processes, helping greatly with scalability.  These
+processes are called 'workers', and are (eventually) intended to scale
+horizontally independently.
+
+All of the below is highly experimental and subject to change as Synapse evolves,
+but documenting it here to help folks needing highly scalable Synapses similar
+to the one running matrix.org!
+
+All processes continue to share the same database instance, and as such, workers
+only work with postgres based synapse deployments (sharing a single sqlite
+across multiple processes is a recipe for disaster, plus you should be using
+postgres anyway if you care about scalability).
+
+The workers communicate with the master synapse process via a synapse-specific
+TCP protocol called 'replication' - analogous to MySQL or Postgres style
+database replication; feeding a stream of relevant data to the workers so they
+can be kept in sync with the main synapse process and database state.
+
+## Configuration
+
+To make effective use of the workers, you will need to configure an HTTP
+reverse-proxy such as nginx or haproxy, which will direct incoming requests to
+the correct worker, or to the main synapse instance. Note that this includes
+requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md)
+for information on setting up a reverse proxy.
+
+To enable workers, you need to add two replication listeners to the master
+synapse, e.g.:
+
+    listeners:
+      # The TCP replication port
+      - port: 9092
+        bind_address: '127.0.0.1'
+        type: replication
+      # The HTTP replication port
+      - port: 9093
+        bind_address: '127.0.0.1'
+        type: http
+        resources:
+         - names: [replication]
+
+Under **no circumstances** should these replication API listeners be exposed to
+the public internet; it currently implements no authentication whatsoever and is
+unencrypted.
+
+(Roughly, the TCP port is used for streaming data from the master to the
+workers, and the HTTP port for the workers to send data to the main
+synapse process.)
+
+You then create a set of configs for the various worker processes.  These
+should be worker configuration files, and should be stored in a dedicated
+subdirectory, to allow synctl to manipulate them. An additional configuration
+for the master synapse process will need to be created because the process will
+not be started automatically. That configuration should look like this:
+
+    worker_app: synapse.app.homeserver
+    daemonize: true
+
+Each worker configuration file inherits the configuration of the main homeserver
+configuration file.  You can then override configuration specific to that worker,
+e.g. the HTTP listener that it provides (if any); logging configuration; etc.
+You should minimise the number of overrides though to maintain a usable config.
+
+You must specify the type of worker application (`worker_app`). The currently
+available worker applications are listed below. You must also specify the
+replication endpoints that it's talking to on the main synapse process.
+`worker_replication_host` should specify the host of the main synapse,
+`worker_replication_port` should point to the TCP replication listener port and
+`worker_replication_http_port` should point to the HTTP replication port.
+
+Currently, the `event_creator` and `federation_reader` workers require specifying
+`worker_replication_http_port`.
+
+For instance:
+
+    worker_app: synapse.app.synchrotron
+
+    # The replication listener on the synapse to talk to.
+    worker_replication_host: 127.0.0.1
+    worker_replication_port: 9092
+    worker_replication_http_port: 9093
+
+    worker_listeners:
+     - type: http
+       port: 8083
+       resources:
+         - names:
+           - client
+
+    worker_daemonize: True
+    worker_pid_file: /home/matrix/synapse/synchrotron.pid
+    worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml
+
+...is a full configuration for a synchrotron worker instance, which will expose a
+plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided
+by the main synapse.
+
+Obviously you should configure your reverse-proxy to route the relevant
+endpoints to the worker (`localhost:8083` in the above example).
+
+Finally, to actually run your worker-based synapse, you must pass synctl the -a
+commandline option to tell it to operate on all the worker configurations found
+in the given directory, e.g.:
+
+    synctl -a $CONFIG/workers start
+
+Currently one should always restart all workers when restarting or upgrading
+synapse, unless you explicitly know it's safe not to.  For instance, restarting
+synapse without restarting all the synchrotrons may result in broken typing
+notifications.
+
+To manipulate a specific worker, you pass the -w option to synctl:
+
+    synctl -w $CONFIG/workers/synchrotron.yaml restart
+
+## Available worker applications
+
+### `synapse.app.pusher`
+
+Handles sending push notifications to sygnal and email. Doesn't handle any
+REST endpoints itself, but you should set `start_pushers: False` in the
+shared configuration file to stop the main synapse sending these notifications.
+
+Note this worker cannot be load-balanced: only one instance should be active.
+
+### `synapse.app.synchrotron`
+
+The synchrotron handles `sync` requests from clients. In particular, it can
+handle REST endpoints matching the following regular expressions:
+
+    ^/_matrix/client/(v2_alpha|r0)/sync$
+    ^/_matrix/client/(api/v1|v2_alpha|r0)/events$
+    ^/_matrix/client/(api/v1|r0)/initialSync$
+    ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$
+
+The above endpoints should all be routed to the synchrotron worker by the
+reverse-proxy configuration.
+
+It is possible to run multiple instances of the synchrotron to scale
+horizontally. In this case the reverse-proxy should be configured to
+load-balance across the instances, though it will be more efficient if all
+requests from a particular user are routed to a single instance. Extracting
+a userid from the access token is currently left as an exercise for the reader.
+
+### `synapse.app.appservice`
+
+Handles sending output traffic to Application Services. Doesn't handle any
+REST endpoints itself, but you should set `notify_appservices: False` in the
+shared configuration file to stop the main synapse sending these notifications.
+
+Note this worker cannot be load-balanced: only one instance should be active.
+
+### `synapse.app.federation_reader`
+
+Handles a subset of federation endpoints. In particular, it can handle REST
+endpoints matching the following regular expressions:
+
+    ^/_matrix/federation/v1/event/
+    ^/_matrix/federation/v1/state/
+    ^/_matrix/federation/v1/state_ids/
+    ^/_matrix/federation/v1/backfill/
+    ^/_matrix/federation/v1/get_missing_events/
+    ^/_matrix/federation/v1/publicRooms
+    ^/_matrix/federation/v1/query/
+    ^/_matrix/federation/v1/make_join/
+    ^/_matrix/federation/v1/make_leave/
+    ^/_matrix/federation/v1/send_join/
+    ^/_matrix/federation/v1/send_leave/
+    ^/_matrix/federation/v1/invite/
+    ^/_matrix/federation/v1/query_auth/
+    ^/_matrix/federation/v1/event_auth/
+    ^/_matrix/federation/v1/exchange_third_party_invite/
+    ^/_matrix/federation/v1/send/
+    ^/_matrix/key/v2/query
+
+The above endpoints should all be routed to the federation_reader worker by the
+reverse-proxy configuration.
+
+The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single
+instance.
+
+### `synapse.app.federation_sender`
+
+Handles sending federation traffic to other servers. Doesn't handle any
+REST endpoints itself, but you should set `send_federation: False` in the
+shared configuration file to stop the main synapse sending this traffic.
+
+Note this worker cannot be load-balanced: only one instance should be active.
+
+### `synapse.app.media_repository`
+
+Handles the media repository. It can handle all endpoints starting with:
+
+    /_matrix/media/
+
+And the following regular expressions matching media-specific administration APIs:
+
+    ^/_synapse/admin/v1/purge_media_cache$
+    ^/_synapse/admin/v1/room/.*/media$
+    ^/_synapse/admin/v1/quarantine_media/.*$
+
+You should also set `enable_media_repo: False` in the shared configuration
+file to stop the main synapse running background jobs related to managing the
+media repository.
+
+Note this worker cannot be load-balanced: only one instance should be active.
+
+### `synapse.app.client_reader`
+
+Handles client API endpoints. It can handle REST endpoints matching the
+following regular expressions:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
+    ^/_matrix/client/(api/v1|r0|unstable)/login$
+    ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/query$
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
+    ^/_matrix/client/versions$
+    ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
+
+Additionally, the following REST endpoints can be handled for GET requests:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$
+
+Additionally, the following REST endpoints can be handled, but all requests must
+be routed to the same instance:
+
+    ^/_matrix/client/(r0|unstable)/register$
+
+Pagination requests can also be handled, but all requests with the same path
+room must be routed to the same instance. Additionally, care must be taken to
+ensure that the purge history admin API is not used while pagination requests
+for the room are in flight:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
+
+### `synapse.app.user_dir`
+
+Handles searches in the user directory. It can handle REST endpoints matching
+the following regular expressions:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
+
+### `synapse.app.frontend_proxy`
+
+Proxies some frequently-requested client endpoints to add caching and remove
+load from the main synapse. It can handle REST endpoints matching the following
+regular expressions:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/upload
+
+If `use_presence` is False in the homeserver config, it can also handle REST
+endpoints matching the following regular expressions:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/presence/[^/]+/status
+
+This "stub" presence handler will pass through `GET` request but make the
+`PUT` effectively a no-op.
+
+It will proxy any requests it cannot handle to the main synapse instance. It
+must therefore be configured with the location of the main instance, via
+the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration
+file. For example:
+
+    worker_main_http_uri: http://127.0.0.1:8008
+
+### `synapse.app.event_creator`
+
+Handles some event creation. It can handle REST endpoints matching:
+
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
+    ^/_matrix/client/(api/v1|r0|unstable)/join/
+    ^/_matrix/client/(api/v1|r0|unstable)/profile/
+
+It will create events locally and then send them on to the main synapse
+instance to be persisted and handled.