From 1a8406f42ac3a7f63690c13cb8d8fd5547928632 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Sun, 21 Nov 2021 02:39:02 +0000 Subject: Move documentation files to their place in the hierarchy; update SUMMARY.md --- docs/CAPTCHA_SETUP.md | 37 -- docs/MSC1711_certificates_FAQ.md | 335 ------------ docs/SUMMARY.md | 64 +-- docs/application_services.md | 35 -- docs/auth_chain_diff.dot | 32 -- docs/auth_chain_diff.dot.png | Bin 42427 -> 0 bytes docs/auth_chain_difference_algorithm.md | 108 ---- docs/code_style.md | 177 ------- docs/consent_tracking.md | 197 ------- docs/delegate.md | 102 ---- docs/deprecation_policy.md | 33 -- docs/development/code_style.md | 177 +++++++ .../internal_documentation/media_repository.md | 30 ++ .../room_and_user_statistics.md | 22 + .../state_resolution/auth_chain_diff.dot | 32 ++ .../state_resolution/auth_chain_diff.dot.png | Bin 0 -> 42427 bytes .../auth_chain_difference_algorithm.md | 108 ++++ docs/development/opentracing.md | 93 ++++ .../synapse_architecture/log_contexts.md | 364 +++++++++++++ .../synapse_architecture/replication.md | 37 ++ .../synapse_architecture/tcp_replication.md | 257 +++++++++ docs/federate.md | 66 --- docs/introduction/welcome_and_overview.md | 79 +++ docs/jwt.md | 97 ---- docs/log_contexts.md | 364 ------------- docs/manhole.md | 99 ---- docs/media_repository.md | 30 -- docs/message_retention_policies.md | 205 -------- docs/metrics-howto.md | 283 ---------- docs/openid.md | 572 --------------------- docs/opentracing.md | 93 ---- docs/other/dependency_deprecation_policy.md | 33 ++ docs/password_auth_providers.md | 129 ----- docs/postgres.md | 255 --------- docs/replication.md | 37 -- docs/reverse_proxy.md | 267 ---------- docs/room_and_user_statistics.md | 22 - docs/server_notices.md | 61 --- docs/setup/delegation.md | 102 ++++ docs/setup/postgres.md | 255 +++++++++ docs/setup/reverse_proxy.md | 267 ++++++++++ docs/setup/turn-howto.md | 296 +++++++++++ docs/sso_mapping_providers.md | 197 ------- docs/structured_logging.md | 161 ------ docs/synctl_workers.md | 36 -- docs/tcp_replication.md | 257 --------- docs/templates.md | 239 --------- docs/turn-howto.md | 296 ----------- docs/upgrading/upgrading_from_pre_synapse_1.0.md | 335 ++++++++++++ docs/usage/administration/manhole.md | 99 ++++ docs/usage/administration/monitoring.md | 283 ++++++++++ docs/usage/configuration/application_services.md | 35 ++ docs/usage/configuration/consent_tracking.md | 197 +++++++ docs/usage/configuration/json_web_tokens.md | 97 ++++ .../configuration/message_retention_policies.md | 205 ++++++++ docs/usage/configuration/registration_captcha.md | 37 ++ docs/usage/configuration/server_notices.md | 61 +++ docs/usage/configuration/structured_logging.md | 161 ++++++ docs/usage/configuration/templates.md | 239 +++++++++ .../user_authentication/password_auth_providers.md | 129 +++++ .../user_authentication/single_sign_on/openid.md | 572 +++++++++++++++++++++ .../single_sign_on/sso_mapping_providers.md | 197 +++++++ docs/usage/configuration/user_directory.md | 49 ++ docs/usage/configuration/workers/README.md | 560 ++++++++++++++++++++ docs/usage/configuration/workers/synctl_workers.md | 36 ++ docs/usage/federation/README.md | 66 +++ docs/user_directory.md | 49 -- docs/welcome_and_overview.md | 79 --- docs/workers.md | 560 -------------------- 69 files changed, 5542 insertions(+), 5542 deletions(-) delete mode 100644 docs/CAPTCHA_SETUP.md delete mode 100644 docs/MSC1711_certificates_FAQ.md delete mode 100644 docs/application_services.md delete mode 100644 docs/auth_chain_diff.dot delete mode 100644 docs/auth_chain_diff.dot.png delete mode 100644 docs/auth_chain_difference_algorithm.md delete mode 100644 docs/code_style.md delete mode 100644 docs/consent_tracking.md delete mode 100644 docs/delegate.md delete mode 100644 docs/deprecation_policy.md create mode 100644 docs/development/code_style.md create mode 100644 docs/development/internal_documentation/media_repository.md create mode 100644 docs/development/internal_documentation/room_and_user_statistics.md create mode 100644 docs/development/internal_documentation/state_resolution/auth_chain_diff.dot create mode 100644 docs/development/internal_documentation/state_resolution/auth_chain_diff.dot.png create mode 100644 docs/development/internal_documentation/state_resolution/auth_chain_difference_algorithm.md create mode 100644 docs/development/opentracing.md create mode 100644 docs/development/synapse_architecture/log_contexts.md create mode 100644 docs/development/synapse_architecture/replication.md create mode 100644 docs/development/synapse_architecture/tcp_replication.md delete mode 100644 docs/federate.md create mode 100644 docs/introduction/welcome_and_overview.md delete mode 100644 docs/jwt.md delete mode 100644 docs/log_contexts.md delete mode 100644 docs/manhole.md delete mode 100644 docs/media_repository.md delete mode 100644 docs/message_retention_policies.md delete mode 100644 docs/metrics-howto.md delete mode 100644 docs/openid.md delete mode 100644 docs/opentracing.md create mode 100644 docs/other/dependency_deprecation_policy.md delete mode 100644 docs/password_auth_providers.md delete mode 100644 docs/postgres.md delete mode 100644 docs/replication.md delete mode 100644 docs/reverse_proxy.md delete mode 100644 docs/room_and_user_statistics.md delete mode 100644 docs/server_notices.md create mode 100644 docs/setup/delegation.md create mode 100644 docs/setup/postgres.md create mode 100644 docs/setup/reverse_proxy.md create mode 100644 docs/setup/turn-howto.md delete mode 100644 docs/sso_mapping_providers.md delete mode 100644 docs/structured_logging.md delete mode 100644 docs/synctl_workers.md delete mode 100644 docs/tcp_replication.md delete mode 100644 docs/templates.md delete mode 100644 docs/turn-howto.md create mode 100644 docs/upgrading/upgrading_from_pre_synapse_1.0.md create mode 100644 docs/usage/administration/manhole.md create mode 100644 docs/usage/administration/monitoring.md create mode 100644 docs/usage/configuration/application_services.md create mode 100644 docs/usage/configuration/consent_tracking.md create mode 100644 docs/usage/configuration/json_web_tokens.md create mode 100644 docs/usage/configuration/message_retention_policies.md create mode 100644 docs/usage/configuration/registration_captcha.md create mode 100644 docs/usage/configuration/server_notices.md create mode 100644 docs/usage/configuration/structured_logging.md create mode 100644 docs/usage/configuration/templates.md create mode 100644 docs/usage/configuration/user_authentication/password_auth_providers.md create mode 100644 docs/usage/configuration/user_authentication/single_sign_on/openid.md create mode 100644 docs/usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md create mode 100644 docs/usage/configuration/user_directory.md create mode 100644 docs/usage/configuration/workers/README.md create mode 100644 docs/usage/configuration/workers/synctl_workers.md create mode 100644 docs/usage/federation/README.md delete mode 100644 docs/user_directory.md delete mode 100644 docs/welcome_and_overview.md delete mode 100644 docs/workers.md (limited to 'docs') diff --git a/docs/CAPTCHA_SETUP.md b/docs/CAPTCHA_SETUP.md deleted file mode 100644 index 49419ce8df..0000000000 --- a/docs/CAPTCHA_SETUP.md +++ /dev/null @@ -1,37 +0,0 @@ -# Overview -A captcha can be enabled on your homeserver to help prevent bots from registering -accounts. Synapse currently uses Google's reCAPTCHA service which requires API keys -from Google. - -## Getting API keys - -1. Create a new site at -1. Set the label to anything you want -1. Set the type to reCAPTCHA v2 using the "I'm not a robot" Checkbox option. -This is the only type of captcha that works with Synapse. -1. Add the public hostname for your server, as set in `public_baseurl` -in `homeserver.yaml`, to the list of authorized domains. If you have not set -`public_baseurl`, use `server_name`. -1. Agree to the terms of service and submit. -1. Copy your site key and secret key and add them to your `homeserver.yaml` -configuration file - ```yaml - recaptcha_public_key: YOUR_SITE_KEY - recaptcha_private_key: YOUR_SECRET_KEY - ``` -1. Enable the CAPTCHA for new registrations - ```yaml - enable_registration_captcha: true - ``` -1. Go to the settings page for the CAPTCHA you just created -1. Uncheck the "Verify the origin of reCAPTCHA solutions" checkbox so that the -captcha can be displayed in any client. If you do not disable this option then you -must specify the domains of every client that is allowed to display the CAPTCHA. - -## Configuring IP used for auth - -The reCAPTCHA API requires that the IP address of the user who solved the -CAPTCHA is sent. If the client is connecting through a proxy or load balancer, -it may be required to use the `X-Forwarded-For` (XFF) header instead of the origin -IP address. This can be configured using the `x_forwarded` directive in the -listeners section of the `homeserver.yaml` configuration file. diff --git a/docs/MSC1711_certificates_FAQ.md b/docs/MSC1711_certificates_FAQ.md deleted file mode 100644 index 086899a9d8..0000000000 --- a/docs/MSC1711_certificates_FAQ.md +++ /dev/null @@ -1,335 +0,0 @@ -# MSC1711 Certificates FAQ - -## Historical Note -This document was originally written to guide server admins through the upgrade -path towards Synapse 1.0. Specifically, -[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md) -required that all servers present valid TLS certificates on their federation -API. Admins were encouraged to achieve compliance from version 0.99.0 (released -in February 2019) ahead of version 1.0 (released June 2019) enforcing the -certificate checks. - -Much of what follows is now outdated since most admins will have already -upgraded, however it may be of use to those with old installs returning to the -project. - -If you are setting up a server from scratch you almost certainly should look at -the [installation guide](setup/installation.md) instead. - -## Introduction -The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It -supports the r0.1 release of the server to server specification, but is -compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well -as post-r0.1 behaviour, in order to allow for a smooth upgrade across the -federation. - -The most important thing to know is that Synapse 1.0.0 will require a valid TLS -certificate on federation endpoints. Self signed certificates will not be -sufficient. - -Synapse 0.99.0 makes it easy to configure TLS certificates and will -interoperate with both >= 1.0.0 servers as well as existing servers yet to -upgrade. - -**It is critical that all admins upgrade to 0.99.0 and configure a valid TLS -certificate.** Admins will have 1 month to do so, after which 1.0.0 will be -released and those servers without a valid certificate will not longer be able -to federate with >= 1.0.0 servers. - -Full details on how to carry out this configuration change is given -[below](#configuring-certificates-for-compatibility-with-synapse-100). A -timeline and some frequently asked questions are also given below. - -For more details and context on the release of the r0.1 Server/Server API and -imminent Matrix 1.0 release, you can also see our -[main talk from FOSDEM 2019](https://matrix.org/blog/2019/02/04/matrix-at-fosdem-2019/). - -## Contents -* Timeline -* Configuring certificates for compatibility with Synapse 1.0 -* FAQ - * Synapse 0.99.0 has just been released, what do I need to do right now? - * How do I upgrade? - * What will happen if I do not set up a valid federation certificate - immediately? - * What will happen if I do nothing at all? - * When do I need a SRV record or .well-known URI? - * Can I still use an SRV record? - * I have created a .well-known URI. Do I still need an SRV record? - * It used to work just fine, why are you breaking everything? - * Can I manage my own certificates rather than having Synapse renew - certificates itself? - * Do you still recommend against using a reverse proxy on the federation port? - * Do I still need to give my TLS certificates to Synapse if I am using a - reverse proxy? - * Do I need the same certificate for the client and federation port? - * How do I tell Synapse to reload my keys/certificates after I replace them? - -## Timeline - -**5th Feb 2019 - Synapse 0.99.0 is released.** - -All server admins are encouraged to upgrade. - -0.99.0: - -- provides support for ACME to make setting up Let's Encrypt certs easy, as - well as .well-known support. - -- does not enforce that a valid CA cert is present on the federation API, but - rather makes it easy to set one up. - -- provides support for .well-known - -Admins should upgrade and configure a valid CA cert. Homeservers that require a -.well-known entry (see below), should retain their SRV record and use it -alongside their .well-known record. - -**10th June 2019 - Synapse 1.0.0 is released** - -1.0.0 is scheduled for release on 10th June. In -accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html) -1.0.0 will enforce certificate validity. This means that any homeserver without a -valid certificate after this point will no longer be able to federate with -1.0.0 servers. - -## Configuring certificates for compatibility with Synapse 1.0.0 - -### If you do not currently have an SRV record - -In this case, your `server_name` points to the host where your Synapse is -running. There is no need to create a `.well-known` URI or an SRV record, but -you will need to give Synapse a valid, signed, certificate. - -### If you do have an SRV record currently - -If you are using an SRV record, your matrix domain (`server_name`) may not -point to the same host that your Synapse is running on (the 'target -domain'). (If it does, you can follow the recommendation above; otherwise, read -on.) - -Let's assume that your `server_name` is `example.com`, and your Synapse is -hosted at a target domain of `customer.example.net`. Currently you should have -an SRV record which looks like: - -``` -_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net. -``` - -In this situation, you have three choices for how to proceed: - -#### Option 1: give Synapse a certificate for your matrix domain - -Synapse 1.0 will expect your server to present a TLS certificate for your -`server_name` (`example.com` in the above example). You can achieve this by acquiring a -certificate for the `server_name` yourself (for example, using `certbot`), and giving it -and the key to Synapse via `tls_certificate_path` and `tls_private_key_path`. - -#### Option 2: run Synapse behind a reverse proxy - -If you have an existing reverse proxy set up with correct TLS certificates for -your domain, you can simply route all traffic through the reverse proxy by -updating the SRV record appropriately (or removing it, if the proxy listens on -8448). - -See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a -reverse proxy. - -#### Option 3: add a .well-known file to delegate your matrix traffic - -This will allow you to keep Synapse on a separate domain, without having to -give it a certificate for the matrix domain. - -You can do this with a `.well-known` file as follows: - - 1. Keep the SRV record in place - it is needed for backwards compatibility - with Synapse 0.34 and earlier. - - 2. Give Synapse a certificate corresponding to the target domain - (`customer.example.net` in the above example). You can do this by acquire a - certificate for the target domain and giving it to Synapse via `tls_certificate_path` - and `tls_private_key_path`. - - 3. Restart Synapse to ensure the new certificate is loaded. - - 4. Arrange for a `.well-known` file at - `https:///.well-known/matrix/server` with contents: - - ```json - {"m.server": ""} - ``` - - where the target server name is resolved as usual (i.e. SRV lookup, falling - back to talking to port 8448). - - In the above example, where synapse is listening on port 8000, - `https://example.com/.well-known/matrix/server` should have `m.server` set to one of: - - 1. `customer.example.net` ─ with a SRV record on - `_matrix._tcp.customer.example.com` pointing to port 8000, or: - - 2. `customer.example.net` ─ updating synapse to listen on the default port - 8448, or: - - 3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy - on `customer.example.net:8000` it correctly handles HTTP requests with - Host header set to `customer.example.net:8000`. - -## FAQ - -### Synapse 0.99.0 has just been released, what do I need to do right now? - -Upgrade as soon as you can in preparation for Synapse 1.0.0, and update your -TLS certificates as [above](#configuring-certificates-for-compatibility-with-synapse-100). - -### What will happen if I do not set up a valid federation certificate immediately? - -Nothing initially, but once 1.0.0 is in the wild it will not be possible to -federate with 1.0.0 servers. - -### What will happen if I do nothing at all? - -If the admin takes no action at all, and remains on a Synapse < 0.99.0 then the -homeserver will be unable to federate with those who have implemented -.well-known. Then, as above, once the month upgrade window has expired the -homeserver will not be able to federate with any Synapse >= 1.0.0 - -### When do I need a SRV record or .well-known URI? - -If your homeserver listens on the default federation port (8448), and your -`server_name` points to the host that your homeserver runs on, you do not need an -SRV record or `.well-known/matrix/server` URI. - -For instance, if you registered `example.com` and pointed its DNS A record at a -fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host, -giving it a server_name of `example.com`, and it would automatically generate a -valid TLS certificate for you via Let's Encrypt and no SRV record or -`.well-known` URI would be needed. - -This is the common case, although you can add an SRV record or -`.well-known/matrix/server` URI for completeness if you wish. - -**However**, if your server does not listen on port 8448, or if your `server_name` -does not point to the host that your homeserver runs on, you will need to let -other servers know how to find it. - -In this case, you should see ["If you do have an SRV record -currently"](#if-you-do-have-an-srv-record-currently) above. - -### Can I still use an SRV record? - -Firstly, if you didn't need an SRV record before (because your server is -listening on port 8448 of your server_name), you certainly don't need one now: -the defaults are still the same. - -If you previously had an SRV record, you can keep using it provided you are -able to give Synapse a TLS certificate corresponding to your server name. For -example, suppose you had the following SRV record, which directs matrix traffic -for example.com to matrix.example.com:443: - -``` -_matrix._tcp.example.com. IN SRV 10 5 443 matrix.example.com -``` - -In this case, Synapse must be given a certificate for example.com - or be -configured to acquire one from Let's Encrypt. - -If you are unable to give Synapse a certificate for your server_name, you will -also need to use a .well-known URI instead. However, see also "I have created a -.well-known URI. Do I still need an SRV record?". - -### I have created a .well-known URI. Do I still need an SRV record? - -As of Synapse 0.99, Synapse will first check for the existence of a `.well-known` -URI and follow any delegation it suggests. It will only then check for the -existence of an SRV record. - -That means that the SRV record will often be redundant. However, you should -remember that there may still be older versions of Synapse in the federation -which do not understand `.well-known` URIs, so if you removed your SRV record you -would no longer be able to federate with them. - -It is therefore best to leave the SRV record in place for now. Synapse 0.34 and -earlier will follow the SRV record (and not care about the invalid -certificate). Synapse 0.99 and later will follow the .well-known URI, with the -correct certificate chain. - -### It used to work just fine, why are you breaking everything? - -We have always wanted Matrix servers to be as easy to set up as possible, and -so back when we started federation in 2014 we didn't want admins to have to go -through the cumbersome process of buying a valid TLS certificate to run a -server. This was before Let's Encrypt came along and made getting a free and -valid TLS certificate straightforward. So instead, we adopted a system based on -[Perspectives](https://en.wikipedia.org/wiki/Convergence_(SSL)): an approach -where you check a set of "notary servers" (in practice, homeservers) to vouch -for the validity of a certificate rather than having it signed by a CA. As long -as enough different notaries agree on the certificate's validity, then it is -trusted. - -However, in practice this has never worked properly. Most people only use the -default notary server (matrix.org), leading to inadvertent centralisation which -we want to eliminate. Meanwhile, we never implemented the full consensus -algorithm to query the servers participating in a room to determine consensus -on whether a given certificate is valid. This is fiddly to get right -(especially in face of sybil attacks), and we found ourselves questioning -whether it was worth the effort to finish the work and commit to maintaining a -secure certificate validation system as opposed to focusing on core Matrix -development. - -Meanwhile, Let's Encrypt came along in 2016, and put the final nail in the -coffin of the Perspectives project (which was already pretty dead). So, the -Spec Core Team decided that a better approach would be to mandate valid TLS -certificates for federation alongside the rest of the Web. More details can be -found in -[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach). - -This results in a breaking change, which is disruptive, but absolutely critical -for the security model. However, the existence of Let's Encrypt as a trivial -way to replace the old self-signed certificates with valid CA-signed ones helps -smooth things over massively, especially as Synapse can now automate Let's -Encrypt certificate generation if needed. - -### Can I manage my own certificates rather than having Synapse renew certificates itself? - -Yes, you are welcome to manage your certificates yourself. Synapse will only -attempt to obtain certificates from Let's Encrypt if you configure it to do -so.The only requirement is that there is a valid TLS cert present for -federation end points. - -### Do you still recommend against using a reverse proxy on the federation port? - -We no longer actively recommend against using a reverse proxy. Many admins will -find it easier to direct federation traffic to a reverse proxy and manage their -own TLS certificates, and this is a supported configuration. - -See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a -reverse proxy. - -### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? - -Practically speaking, this is no longer necessary. - -If you are using a reverse proxy for all of your TLS traffic, then you can set -`no_tls: True`. In that case, the only reason Synapse needs the certificate is -to populate a legacy 'tls_fingerprints' field in the federation API. This is -ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will -check it is when attempting to fetch the server keys - and generally this is -delegated via `matrix.org`, which is on 0.99.0. - -However, there is a bug in Synapse 0.99.0 -[4554]() which prevents -Synapse from starting if you do not give it a TLS certificate. To work around -this, you can give it any TLS certificate at all. This will be fixed soon. - -### Do I need the same certificate for the client and federation port? - -No. There is nothing stopping you from using different certificates, -particularly if you are using a reverse proxy. However, Synapse will use the -same certificate on any ports where TLS is configured. - -### How do I tell Synapse to reload my keys/certificates after I replace them? - -Synapse will reload the keys and certificates when it receives a SIGHUP - for -example `kill -HUP $(cat homeserver.pid)`. Alternatively, simply restart -Synapse, though this will result in downtime while it restarts. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index cdedf8bccc..9af7d006c0 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -1,42 +1,42 @@ # Summary # Introduction -- [Welcome and Overview](welcome_and_overview.md) +- [Welcome and Overview](introduction/welcome_and_overview.md) # Setup - [Installation](setup/installation.md) - - [Using Postgres](postgres.md) - - [Configuring a Reverse Proxy](reverse_proxy.md) + - [Using Postgres](setup/postgres.md) + - [Configuring a Reverse Proxy](setup/reverse_proxy.md) - [Configuring a Forward/Outbound Proxy](setup/forward_proxy.md) - - [Configuring a Turn Server](turn-howto.md) - - [Delegation](delegate.md) + - [Configuring a Turn Server](setup/turn-howto.md) + - [Delegation](setup/delegation.md) # Upgrading - [Upgrading between Synapse Versions](upgrade.md) - - [Upgrading from pre-Synapse 1.0](MSC1711_certificates_FAQ.md) + - [Upgrading from pre-Synapse 1.0](upgrading/upgrading_from_pre_synapse_1.0.md) # Usage - - [Federation](federate.md) + - [Federation](usage/federation/README.md) - [Configuration](usage/configuration/README.md) - [Homeserver Sample Config File](usage/configuration/homeserver_sample_config.md) - [Logging Sample Config File](usage/configuration/logging_sample_config.md) - - [Structured Logging](structured_logging.md) - - [Templates](templates.md) + - [Structured Logging](usage/configuration/structured_logging.md) + - [Templates](usage/configuration/templates.md) - [User Authentication](usage/configuration/user_authentication/README.md) - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md) - - [OpenID Connect](openid.md) + - [OpenID Connect](usage/configuration/user_authentication/single_sign_on/openid.md) - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md) - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md) - - [SSO Mapping Providers](sso_mapping_providers.md) - - [Password Auth Providers](password_auth_providers.md) - - [JSON Web Tokens](jwt.md) - - [Registration Captcha](CAPTCHA_SETUP.md) - - [Application Services](application_services.md) - - [Server Notices](server_notices.md) - - [Consent Tracking](consent_tracking.md) + - [SSO Mapping Providers](usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md) + - [Password Auth Providers](usage/configuration/user_authentication/password_auth_providers.md) + - [JSON Web Tokens](usage/configuration/json_web_tokens.md) + - [Registration Captcha](usage/configuration/registration_captcha.md) + - [Application Services](usage/configuration/application_services.md) + - [Server Notices](usage/configuration/server_notices.md) + - [Consent Tracking](usage/configuration/consent_tracking.md) - [URL Previews](development/url_previews.md) - - [User Directory](user_directory.md) - - [Message Retention Policies](message_retention_policies.md) + - [User Directory](usage/configuration/user_directory.md) + - [Message Retention Policies](usage/configuration/message_retention_policies.md) - [Pluggable Modules](modules/index.md) - [Writing a module](modules/writing_a_module.md) - [Spam checker callbacks](modules/spam_checker_callbacks.md) @@ -45,8 +45,8 @@ - [Account validity callbacks](modules/account_validity_callbacks.md) - [Password auth provider callbacks](modules/password_auth_provider_callbacks.md) - [Porting a legacy module to the new interface](modules/porting_legacy_module.md) - - [Workers](workers.md) - - [Using `synctl` with Workers](synctl_workers.md) + - [Workers](usage/configuration/workers/README.md) + - [Using `synctl` with Workers](usage/configuration/workers/synctl_workers.md) - [Systemd](systemd-with-workers/README.md) - [Administration](usage/administration/README.md) - [Admin API](usage/administration/admin_api/README.md) @@ -64,33 +64,33 @@ - [Statistics](admin_api/statistics.md) - [Users](admin_api/user_admin_api.md) - [Server Version](admin_api/version_api.md) - - [Manhole](manhole.md) - - [Monitoring](metrics-howto.md) + - [Manhole](usage/administration/manhole.md) + - [Monitoring](usage/administration/monitoring.md) - [Request log format](usage/administration/request_log.md) - [Scripts]() # Development - [Contributing Guide](development/contributing_guide.md) - - [Code Style](code_style.md) + - [Code Style](development/code_style.md) - [Git Usage](development/git.md) - [Testing]() - - [OpenTracing](opentracing.md) + - [OpenTracing](development/opentracing.md) - [Database Schemas](development/database_schema.md) - [Experimental features](development/experimental_features.md) - [Synapse Architecture]() - - [Log Contexts](log_contexts.md) - - [Replication](replication.md) - - [TCP Replication](tcp_replication.md) + - [Log Contexts](development/synapse_architecture/log_contexts.md) + - [Replication](development/synapse_architecture/replication.md) + - [TCP Replication](development/synapse_architecture/tcp_replication.md) - [Internal Documentation](development/internal_documentation/README.md) - [Single Sign-On]() - [SAML](development/saml.md) - [CAS](development/cas.md) - [Room DAG concepts](development/room-dag-concepts.md) - [State Resolution]() - - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md) - - [Media Repository](media_repository.md) - - [Room and User Statistics](room_and_user_statistics.md) + - [The Auth Chain Difference Algorithm](development/internal_documentation/state_resolution/auth_chain_difference_algorithm.md) + - [Media Repository](development/internal_documentation/media_repository.md) + - [Room and User Statistics](development/internal_documentation/room_and_user_statistics.md) - [Scripts]() # Other - - [Dependency Deprecation Policy](deprecation_policy.md) + - [Dependency Deprecation Policy](other/dependency_deprecation_policy.md) diff --git a/docs/application_services.md b/docs/application_services.md deleted file mode 100644 index e4592010a2..0000000000 --- a/docs/application_services.md +++ /dev/null @@ -1,35 +0,0 @@ -# Registering an Application Service - -The registration of new application services depends on the homeserver used. -In synapse, you need to create a new configuration file for your AS and add it -to the list specified under the `app_service_config_files` config -option in your synapse config. - -For example: - -```yaml -app_service_config_files: -- /home/matrix/.synapse/.yaml -``` - -The format of the AS configuration file is as follows: - -```yaml -url: -as_token: -hs_token: -sender_localpart: -namespaces: - users: # List of users we're interested in - - exclusive: - regex: - group_id: - - ... - aliases: [] # List of aliases we're interested in - rooms: [] # List of room ids we're interested in -``` - -`exclusive`: If enabled, only this application service is allowed to register users in its namespace(s). -`group_id`: All users of this application service are dynamically joined to this group. This is useful for e.g user organisation or flairs. - -See the [spec](https://matrix.org/docs/spec/application_service/unstable.html) for further details on how application services work. diff --git a/docs/auth_chain_diff.dot b/docs/auth_chain_diff.dot deleted file mode 100644 index 978d579ada..0000000000 --- a/docs/auth_chain_diff.dot +++ /dev/null @@ -1,32 +0,0 @@ -digraph auth { - nodesep=0.5; - rankdir="RL"; - - C [label="Create (1,1)"]; - - BJ [label="Bob's Join (2,1)", color=red]; - BJ2 [label="Bob's Join (2,2)", color=red]; - BJ2 -> BJ [color=red, dir=none]; - - subgraph cluster_foo { - A1 [label="Alice's invite (4,1)", color=blue]; - A2 [label="Alice's Join (4,2)", color=blue]; - A3 [label="Alice's Join (4,3)", color=blue]; - A3 -> A2 -> A1 [color=blue, dir=none]; - color=none; - } - - PL1 [label="Power Level (3,1)", color=darkgreen]; - PL2 [label="Power Level (3,2)", color=darkgreen]; - PL2 -> PL1 [color=darkgreen, dir=none]; - - {rank = same; C; BJ; PL1; A1;} - - A1 -> C [color=grey]; - A1 -> BJ [color=grey]; - PL1 -> C [color=grey]; - BJ2 -> PL1 [penwidth=2]; - - A3 -> PL2 [penwidth=2]; - A1 -> PL1 -> BJ -> C [penwidth=2]; -} diff --git a/docs/auth_chain_diff.dot.png b/docs/auth_chain_diff.dot.png deleted file mode 100644 index 771c07308f..0000000000 Binary files a/docs/auth_chain_diff.dot.png and /dev/null differ diff --git a/docs/auth_chain_difference_algorithm.md b/docs/auth_chain_difference_algorithm.md deleted file mode 100644 index 30f72a70da..0000000000 --- a/docs/auth_chain_difference_algorithm.md +++ /dev/null @@ -1,108 +0,0 @@ -# Auth Chain Difference Algorithm - -The auth chain difference algorithm is used by V2 state resolution, where a -naive implementation can be a significant source of CPU and DB usage. - -### Definitions - -A *state set* is a set of state events; e.g. the input of a state resolution -algorithm is a collection of state sets. - -The *auth chain* of a set of events are all the events' auth events and *their* -auth events, recursively (i.e. the events reachable by walking the graph induced -by an event's auth events links). - -The *auth chain difference* of a collection of state sets is the union minus the -intersection of the sets of auth chains corresponding to the state sets, i.e an -event is in the auth chain difference if it is reachable by walking the auth -event graph from at least one of the state sets but not from *all* of the state -sets. - -## Breadth First Walk Algorithm - -A way of calculating the auth chain difference without calculating the full auth -chains for each state set is to do a parallel breadth first walk (ordered by -depth) of each state set's auth chain. By tracking which events are reachable -from each state set we can finish early if every pending event is reachable from -every state set. - -This can work well for state sets that have a small auth chain difference, but -can be very inefficient for larger differences. However, this algorithm is still -used if we don't have a chain cover index for the room (e.g. because we're in -the process of indexing it). - -## Chain Cover Index - -Synapse computes auth chain differences by pre-computing a "chain cover" index -for the auth chain in a room, allowing efficient reachability queries like "is -event A in the auth chain of event B". This is done by assigning every event a -*chain ID* and *sequence number* (e.g. `(5,3)`), and having a map of *links* -between chains (e.g. `(5,3) -> (2,4)`) such that A is reachable by B (i.e. `A` -is in the auth chain of `B`) if and only if either: - -1. A and B have the same chain ID and `A`'s sequence number is less than `B`'s - sequence number; or -2. there is a link `L` between `B`'s chain ID and `A`'s chain ID such that - `L.start_seq_no` <= `B.seq_no` and `A.seq_no` <= `L.end_seq_no`. - -There are actually two potential implementations, one where we store links from -each chain to every other reachable chain (the transitive closure of the links -graph), and one where we remove redundant links (the transitive reduction of the -links graph) e.g. if we have chains `C3 -> C2 -> C1` then the link `C3 -> C1` -would not be stored. Synapse uses the former implementations so that it doesn't -need to recurse to test reachability between chains. - -### Example - -An example auth graph would look like the following, where chains have been -formed based on type/state_key and are denoted by colour and are labelled with -`(chain ID, sequence number)`. Links are denoted by the arrows (links in grey -are those that would be remove in the second implementation described above). - -![Example](auth_chain_diff.dot.png) - -Note that we don't include all links between events and their auth events, as -most of those links would be redundant. For example, all events point to the -create event, but each chain only needs the one link from it's base to the -create event. - -## Using the Index - -This index can be used to calculate the auth chain difference of the state sets -by looking at the chain ID and sequence numbers reachable from each state set: - -1. For every state set lookup the chain ID/sequence numbers of each state event -2. Use the index to find all chains and the maximum sequence number reachable - from each state set. -3. The auth chain difference is then all events in each chain that have sequence - numbers between the maximum sequence number reachable from *any* state set and - the minimum reachable by *all* state sets (if any). - -Note that steps 2 is effectively calculating the auth chain for each state set -(in terms of chain IDs and sequence numbers), and step 3 is calculating the -difference between the union and intersection of the auth chains. - -### Worked Example - -For example, given the above graph, we can calculate the difference between -state sets consisting of: - -1. `S1`: Alice's invite `(4,1)` and Bob's second join `(2,2)`; and -2. `S2`: Alice's second join `(4,3)` and Bob's first join `(2,1)`. - -Using the index we see that the following auth chains are reachable from each -state set: - -1. `S1`: `(1,1)`, `(2,2)`, `(3,1)` & `(4,1)` -2. `S2`: `(1,1)`, `(2,1)`, `(3,2)` & `(4,3)` - -And so, for each the ranges that are in the auth chain difference: -1. Chain 1: None, (since everything can reach the create event). -2. Chain 2: The range `(1, 2]` (i.e. just `2`), as `1` is reachable by all state - sets and the maximum reachable is `2` (corresponding to Bob's second join). -3. Chain 3: Similarly the range `(1, 2]` (corresponding to the second power - level). -4. Chain 4: The range `(1, 3]` (corresponding to both of Alice's joins). - -So the final result is: Bob's second join `(2,2)`, the second power level -`(3,2)` and both of Alice's joins `(4,2)` & `(4,3)`. diff --git a/docs/code_style.md b/docs/code_style.md deleted file mode 100644 index 4d8e7c973d..0000000000 --- a/docs/code_style.md +++ /dev/null @@ -1,177 +0,0 @@ -# Code Style - -## Formatting tools - -The Synapse codebase uses a number of code formatting tools in order to -quickly and automatically check for formatting (and sometimes logical) -errors in code. - -The necessary tools are detailed below. - -First install them with: - -```sh -pip install -e ".[lint,mypy]" -``` - -- **black** - - The Synapse codebase uses [black](https://pypi.org/project/black/) - as an opinionated code formatter, ensuring all comitted code is - properly formatted. - - Have `black` auto-format your code (it shouldn't change any - functionality) with: - - ```sh - black . --exclude="\.tox|build|env" - ``` - -- **flake8** - - `flake8` is a code checking tool. We require code to pass `flake8` - before being merged into the codebase. - - Check all application and test code with: - - ```sh - flake8 synapse tests - ``` - -- **isort** - - `isort` ensures imports are nicely formatted, and can suggest and - auto-fix issues such as double-importing. - - Auto-fix imports with: - - ```sh - isort -rc synapse tests - ``` - - `-rc` means to recursively search the given directories. - -It's worth noting that modern IDEs and text editors can run these tools -automatically on save. It may be worth looking into whether this -functionality is supported in your editor for a more convenient -development workflow. It is not, however, recommended to run `flake8` on -save as it takes a while and is very resource intensive. - -## General rules - -- **Naming**: - - Use camel case for class and type names - - Use underscores for functions and variables. -- **Docstrings**: should follow the [google code - style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings). - See the - [examples](http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) - in the sphinx documentation. -- **Imports**: - - Imports should be sorted by `isort` as described above. - - Prefer to import classes and functions rather than packages or - modules. - - Example: - - ```python - from synapse.types import UserID - ... - user_id = UserID(local, server) - ``` - - is preferred over: - - ```python - from synapse import types - ... - user_id = types.UserID(local, server) - ``` - - (or any other variant). - - This goes against the advice in the Google style guide, but it - means that errors in the name are caught early (at import time). - - - Avoid wildcard imports (`from synapse.types import *`) and - relative imports (`from .types import UserID`). - -## Configuration file format - -The [sample configuration file](./sample_config.yaml) acts as a -reference to Synapse's configuration options for server administrators. -Remember that many readers will be unfamiliar with YAML and server -administration in general, so that it is important that the file be as -easy to understand as possible, which includes following a consistent -format. - -Some guidelines follow: - -- Sections should be separated with a heading consisting of a single - line prefixed and suffixed with `##`. There should be **two** blank - lines before the section header, and **one** after. -- Each option should be listed in the file with the following format: - - A comment describing the setting. Each line of this comment - should be prefixed with a hash (`#`) and a space. - - The comment should describe the default behaviour (ie, what - happens if the setting is omitted), as well as what the effect - will be if the setting is changed. - - Often, the comment end with something like "uncomment the - following to ". - - - A line consisting of only `#`. - - A commented-out example setting, prefixed with only `#`. - - For boolean (on/off) options, convention is that this example - should be the *opposite* to the default (so the comment will end - with "Uncomment the following to enable [or disable] - ." For other options, the example should give some - non-default value which is likely to be useful to the reader. - -- There should be a blank line between each option. -- Where several settings are grouped into a single dict, *avoid* the - convention where the whole block is commented out, resulting in - comment lines starting `# #`, as this is hard to read and confusing - to edit. Instead, leave the top-level config option uncommented, and - follow the conventions above for sub-options. Ensure that your code - correctly handles the top-level option being set to `None` (as it - will be if no sub-options are enabled). -- Lines should be wrapped at 80 characters. -- Use two-space indents. -- `true` and `false` are spelt thus (as opposed to `True`, etc.) -- Use single quotes (`'`) rather than double-quotes (`"`) or backticks - (`` ` ``) to refer to configuration options. - -Example: - -```yaml -## Frobnication ## - -# The frobnicator will ensure that all requests are fully frobnicated. -# To enable it, uncomment the following. -# -#frobnicator_enabled: true - -# By default, the frobnicator will frobnicate with the default frobber. -# The following will make it use an alternative frobber. -# -#frobincator_frobber: special_frobber - -# Settings for the frobber -# -frobber: - # frobbing speed. Defaults to 1. - # - #speed: 10 - - # frobbing distance. Defaults to 1000. - # - #distance: 100 -``` - -Note that the sample configuration is generated from the synapse code -and is maintained by a script, `scripts-dev/generate_sample_config`. -Making sure that the output from this script matches the desired format -is left as an exercise for the reader! diff --git a/docs/consent_tracking.md b/docs/consent_tracking.md deleted file mode 100644 index fb1fec80fe..0000000000 --- a/docs/consent_tracking.md +++ /dev/null @@ -1,197 +0,0 @@ -Support in Synapse for tracking agreement to server terms and conditions -======================================================================== - -Synapse 0.30 introduces support for tracking whether users have agreed to the -terms and conditions set by the administrator of a server - and blocking access -to the server until they have. - -There are several parts to this functionality; each requires some specific -configuration in `homeserver.yaml` to be enabled. - -Note that various parts of the configuation and this document refer to the -"privacy policy": agreement with a privacy policy is one particular use of this -feature, but of course adminstrators can specify other terms and conditions -unrelated to "privacy" per se. - -Collecting policy agreement from a user ---------------------------------------- - -Synapse can be configured to serve the user a simple policy form with an -"accept" button. Clicking "Accept" records the user's acceptance in the -database and shows a success page. - -To enable this, first create templates for the policy and success pages. -These should be stored on the local filesystem. - -These templates use the [Jinja2](http://jinja.pocoo.org) templating language, -and [docs/privacy_policy_templates](https://github.com/matrix-org/synapse/tree/develop/docs/privacy_policy_templates/) -gives examples of the sort of thing that can be done. - -Note that the templates must be stored under a name giving the language of the -template - currently this must always be `en` (for "English"); -internationalisation support is intended for the future. - -The template for the policy itself should be versioned and named according to -the version: for example `1.0.html`. The version of the policy which the user -has agreed to is stored in the database. - -Once the templates are in place, make the following changes to `homeserver.yaml`: - - 1. Add a `user_consent` section, which should look like: - - ```yaml - user_consent: - template_dir: privacy_policy_templates - version: 1.0 - ``` - - `template_dir` points to the directory containing the policy - templates. `version` defines the version of the policy which will be served - to the user. In the example above, Synapse will serve - `privacy_policy_templates/en/1.0.html`. - - - 2. Add a `form_secret` setting at the top level: - - - ```yaml - form_secret: "" - ``` - - This should be set to an arbitrary secret string (try `pwgen -y 30` to - generate suitable secrets). - - More on what this is used for below. - - 3. Add `consent` wherever the `client` resource is currently enabled in the - `listeners` configuration. For example: - - ```yaml - listeners: - - port: 8008 - resources: - - names: - - client - - consent - ``` - - -Finally, ensure that `jinja2` is installed. If you are using a virtualenv, this -should be a matter of `pip install Jinja2`. On debian, try `apt-get install -python-jinja2`. - -Once this is complete, and the server has been restarted, try visiting -`https:///_matrix/consent`. If correctly configured, this should give -an error "Missing string query parameter 'u'". It is now possible to manually -construct URIs where users can give their consent. - -### Enabling consent tracking at registration - -1. Add the following to your configuration: - - ```yaml - user_consent: - require_at_registration: true - policy_name: "Privacy Policy" # or whatever you'd like to call the policy - ``` - -2. In your consent templates, make use of the `public_version` variable to - see if an unauthenticated user is viewing the page. This is typically - wrapped around the form that would be used to actually agree to the document: - - ```html - {% if not public_version %} - -
- - - - -
- {% endif %} - ``` - -3. Restart Synapse to apply the changes. - -Visiting `https:///_matrix/consent` should now give you a view of the privacy -document. This is what users will be able to see when registering for accounts. - -### Constructing the consent URI - -It may be useful to manually construct the "consent URI" for a given user - for -instance, in order to send them an email asking them to consent. To do this, -take the base `https:///_matrix/consent` URL and add the following -query parameters: - - * `u`: the user id of the user. This can either be a full MXID - (`@user:server.com`) or just the localpart (`user`). - - * `h`: hex-encoded HMAC-SHA256 of `u` using the `form_secret` as a key. It is - possible to calculate this on the commandline with something like: - - ```bash - echo -n '' | openssl sha256 -hmac '' - ``` - - This should result in a URI which looks something like: - `https:///_matrix/consent?u=&h=68a152465a4d...`. - - -Note that not providing a `u` parameter will be interpreted as wanting to view -the document from an unauthenticated perspective, such as prior to registration. -Therefore, the `h` parameter is not required in this scenario. To enable this -behaviour, set `require_at_registration` to `true` in your `user_consent` config. - - -Sending users a server notice asking them to agree to the policy ----------------------------------------------------------------- - -It is possible to configure Synapse to send a [server -notice](server_notices.md) to anybody who has not yet agreed to the current -version of the policy. To do so: - - * ensure that the consent resource is configured, as in the previous section - - * ensure that server notices are configured, as in [the server notice documentation](server_notices.md). - - * Add `server_notice_content` under `user_consent` in `homeserver.yaml`. For - example: - - ```yaml - user_consent: - server_notice_content: - msgtype: m.text - body: >- - Please give your consent to the privacy policy at %(consent_uri)s. - ``` - - Synapse automatically replaces the placeholder `%(consent_uri)s` with the - consent uri for that user. - - * ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base - URI that clients use to connect to the server. (It is used to construct - `consent_uri` in the server notice.) - - -Blocking users from using the server until they agree to the policy -------------------------------------------------------------------- - -Synapse can be configured to block any attempts to join rooms or send messages -until the user has given their agreement to the policy. (Joining the server -notices room is exempted from this). - -To enable this, add `block_events_error` under `user_consent`. For example: - -```yaml -user_consent: - block_events_error: >- - You can't send any messages until you consent to the privacy policy at - %(consent_uri)s. -``` - -Synapse automatically replaces the placeholder `%(consent_uri)s` with the -consent uri for that user. - -ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base -URI that clients use to connect to the server. (It is used to construct -`consent_uri` in the error.) diff --git a/docs/delegate.md b/docs/delegate.md deleted file mode 100644 index ee9cbb3b1c..0000000000 --- a/docs/delegate.md +++ /dev/null @@ -1,102 +0,0 @@ -# Delegation of incoming federation traffic - -In the following documentation, we use the term `server_name` to refer to that setting -in your homeserver configuration file. It appears at the ends of user ids, and tells -other homeservers where they can find your server. - -By default, other homeservers will expect to be able to reach yours via -your `server_name`, on port 8448. For example, if you set your `server_name` -to `example.com` (so that your user names look like `@user:example.com`), -other servers will try to connect to yours at `https://example.com:8448/`. - -Delegation is a Matrix feature allowing a homeserver admin to retain a -`server_name` of `example.com` so that user IDs, room aliases, etc continue -to look like `*:example.com`, whilst having federation traffic routed -to a different server and/or port (e.g. `synapse.example.com:443`). - -## .well-known delegation - -To use this method, you need to be able to configure the server at -`https://` to serve a file at -`https:///.well-known/matrix/server`. There are two ways to do this, shown below. - -Note that the `.well-known` file is hosted on the default port for `https` (port 443). - -### External server - -For maximum flexibility, you need to configure an external server such as nginx, Apache -or HAProxy to serve the `https:///.well-known/matrix/server` file. Setting -up such a server is out of the scope of this documentation, but note that it is often -possible to configure your [reverse proxy](reverse_proxy.md) for this. - -The URL `https:///.well-known/matrix/server` should be configured -return a JSON structure containing the key `m.server` like this: - -```json -{ - "m.server": "[:]" -} -``` - -In our example (where we want federation traffic to be routed to -`https://synapse.example.com`, on port 443), this would mean that -`https://example.com/.well-known/matrix/server` should return: - -```json -{ - "m.server": "synapse.example.com:443" -} -``` - -Note, specifying a port is optional. If no port is specified, then it defaults -to 8448. - -### Serving a `.well-known/matrix/server` file with Synapse - -If you are able to set up your domain so that `https://` is routed to -Synapse (i.e., the only change needed is to direct federation traffic to port 443 -instead of port 8448), then it is possible to configure Synapse to serve a suitable -`.well-known/matrix/server` file. To do so, add the following to your `homeserver.yaml` -file: - -```yaml -serve_server_wellknown: true -``` - -**Note**: this *only* works if `https://` is routed to Synapse, so is -generally not suitable if Synapse is hosted at a subdomain such as -`https://synapse.example.com`. - -## SRV DNS record delegation - -It is also possible to do delegation using a SRV DNS record. However, that is generally -not recommended, as it can be difficult to configure the TLS certificates correctly in -this case, and it offers little advantage over `.well-known` delegation. - -However, if you really need it, you can find some documentation on what such a -record should look like and how Synapse will use it in [the Matrix -specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names). - -## Delegation FAQ - -### When do I need delegation? - -If your homeserver's APIs are accessible on the default federation port (8448) -and the domain your `server_name` points to, you do not need any delegation. - -For instance, if you registered `example.com` and pointed its DNS A record at a -fresh server, you could install Synapse on that host, giving it a `server_name` -of `example.com`, and once a reverse proxy has been set up to proxy all requests -sent to the port `8448` and serve TLS certificates for `example.com`, you -wouldn't need any delegation set up. - -**However**, if your homeserver's APIs aren't accessible on port 8448 and on the -domain `server_name` points to, you will need to let other servers know how to -find it using delegation. - -### Should I use a reverse proxy for federation traffic? - -Generally, using a reverse proxy for both the federation and client traffic is a good -idea, since it saves handling TLS traffic in Synapse. See -[the reverse proxy documentation](reverse_proxy.md) for information on setting up a -reverse proxy. diff --git a/docs/deprecation_policy.md b/docs/deprecation_policy.md deleted file mode 100644 index 06ea340559..0000000000 --- a/docs/deprecation_policy.md +++ /dev/null @@ -1,33 +0,0 @@ -Deprecation Policy for Platform Dependencies -============================================ - -Synapse has a number of platform dependencies, including Python and PostgreSQL. -This document outlines the policy towards which versions we support, and when we -drop support for versions in the future. - - -Policy ------- - -Synapse follows the upstream support life cycles for Python and PostgreSQL, -i.e. when a version reaches End of Life Synapse will withdraw support for that -version in future releases. - -Details on the upstream support life cycles for Python and PostgreSQL are -documented at https://endoflife.date/python and -https://endoflife.date/postgresql. - - -Context -------- - -It is important for system admins to have a clear understanding of the platform -requirements of Synapse and its deprecation policies so that they can -effectively plan upgrading their infrastructure ahead of time. This is -especially important in contexts where upgrading the infrastructure requires -auditing and approval from a security team, or where otherwise upgrading is a -long process. - -By following the upstream support life cycles Synapse can ensure that its -dependencies continue to get security patches, while not requiring system admins -to constantly update their platform dependencies to the latest versions. diff --git a/docs/development/code_style.md b/docs/development/code_style.md new file mode 100644 index 0000000000..4d8e7c973d --- /dev/null +++ b/docs/development/code_style.md @@ -0,0 +1,177 @@ +# Code Style + +## Formatting tools + +The Synapse codebase uses a number of code formatting tools in order to +quickly and automatically check for formatting (and sometimes logical) +errors in code. + +The necessary tools are detailed below. + +First install them with: + +```sh +pip install -e ".[lint,mypy]" +``` + +- **black** + + The Synapse codebase uses [black](https://pypi.org/project/black/) + as an opinionated code formatter, ensuring all comitted code is + properly formatted. + + Have `black` auto-format your code (it shouldn't change any + functionality) with: + + ```sh + black . --exclude="\.tox|build|env" + ``` + +- **flake8** + + `flake8` is a code checking tool. We require code to pass `flake8` + before being merged into the codebase. + + Check all application and test code with: + + ```sh + flake8 synapse tests + ``` + +- **isort** + + `isort` ensures imports are nicely formatted, and can suggest and + auto-fix issues such as double-importing. + + Auto-fix imports with: + + ```sh + isort -rc synapse tests + ``` + + `-rc` means to recursively search the given directories. + +It's worth noting that modern IDEs and text editors can run these tools +automatically on save. It may be worth looking into whether this +functionality is supported in your editor for a more convenient +development workflow. It is not, however, recommended to run `flake8` on +save as it takes a while and is very resource intensive. + +## General rules + +- **Naming**: + - Use camel case for class and type names + - Use underscores for functions and variables. +- **Docstrings**: should follow the [google code + style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings). + See the + [examples](http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) + in the sphinx documentation. +- **Imports**: + - Imports should be sorted by `isort` as described above. + - Prefer to import classes and functions rather than packages or + modules. + + Example: + + ```python + from synapse.types import UserID + ... + user_id = UserID(local, server) + ``` + + is preferred over: + + ```python + from synapse import types + ... + user_id = types.UserID(local, server) + ``` + + (or any other variant). + + This goes against the advice in the Google style guide, but it + means that errors in the name are caught early (at import time). + + - Avoid wildcard imports (`from synapse.types import *`) and + relative imports (`from .types import UserID`). + +## Configuration file format + +The [sample configuration file](./sample_config.yaml) acts as a +reference to Synapse's configuration options for server administrators. +Remember that many readers will be unfamiliar with YAML and server +administration in general, so that it is important that the file be as +easy to understand as possible, which includes following a consistent +format. + +Some guidelines follow: + +- Sections should be separated with a heading consisting of a single + line prefixed and suffixed with `##`. There should be **two** blank + lines before the section header, and **one** after. +- Each option should be listed in the file with the following format: + - A comment describing the setting. Each line of this comment + should be prefixed with a hash (`#`) and a space. + + The comment should describe the default behaviour (ie, what + happens if the setting is omitted), as well as what the effect + will be if the setting is changed. + + Often, the comment end with something like "uncomment the + following to ". + + - A line consisting of only `#`. + - A commented-out example setting, prefixed with only `#`. + + For boolean (on/off) options, convention is that this example + should be the *opposite* to the default (so the comment will end + with "Uncomment the following to enable [or disable] + ." For other options, the example should give some + non-default value which is likely to be useful to the reader. + +- There should be a blank line between each option. +- Where several settings are grouped into a single dict, *avoid* the + convention where the whole block is commented out, resulting in + comment lines starting `# #`, as this is hard to read and confusing + to edit. Instead, leave the top-level config option uncommented, and + follow the conventions above for sub-options. Ensure that your code + correctly handles the top-level option being set to `None` (as it + will be if no sub-options are enabled). +- Lines should be wrapped at 80 characters. +- Use two-space indents. +- `true` and `false` are spelt thus (as opposed to `True`, etc.) +- Use single quotes (`'`) rather than double-quotes (`"`) or backticks + (`` ` ``) to refer to configuration options. + +Example: + +```yaml +## Frobnication ## + +# The frobnicator will ensure that all requests are fully frobnicated. +# To enable it, uncomment the following. +# +#frobnicator_enabled: true + +# By default, the frobnicator will frobnicate with the default frobber. +# The following will make it use an alternative frobber. +# +#frobincator_frobber: special_frobber + +# Settings for the frobber +# +frobber: + # frobbing speed. Defaults to 1. + # + #speed: 10 + + # frobbing distance. Defaults to 1000. + # + #distance: 100 +``` + +Note that the sample configuration is generated from the synapse code +and is maintained by a script, `scripts-dev/generate_sample_config`. +Making sure that the output from this script matches the desired format +is left as an exercise for the reader! diff --git a/docs/development/internal_documentation/media_repository.md b/docs/development/internal_documentation/media_repository.md new file mode 100644 index 0000000000..99ee8f1ef7 --- /dev/null +++ b/docs/development/internal_documentation/media_repository.md @@ -0,0 +1,30 @@ +# Media Repository + +*Synapse implementation-specific details for the media repository* + +The media repository is where attachments and avatar photos are stored. +It stores attachment content and thumbnails for media uploaded by local users. +It caches attachment content and thumbnails for media uploaded by remote users. + +## Storage + +Each item of media is assigned a `media_id` when it is uploaded. +The `media_id` is a randomly chosen, URL safe 24 character string. + +Metadata such as the MIME type, upload time and length are stored in the +sqlite3 database indexed by `media_id`. + +Content is stored on the filesystem under a `"local_content"` directory. + +Thumbnails are stored under a `"local_thumbnails"` directory. + +The item with `media_id` `"aabbccccccccdddddddddddd"` is stored under +`"local_content/aa/bb/ccccccccdddddddddddd"`. Its thumbnail with width +`128` and height `96` and type `"image/jpeg"` is stored under +`"local_thumbnails/aa/bb/ccccccccdddddddddddd/128-96-image-jpeg"` + +Remote content is cached under `"remote_content"` directory. Each item of +remote content is assigned a local `"filesystem_id"` to ensure that the +directory structure `"remote_content/server_name/aa/bb/ccccccccdddddddddddd"` +is appropriate. Thumbnails for remote content are stored under +`"remote_thumbnail/server_name/..."` diff --git a/docs/development/internal_documentation/room_and_user_statistics.md b/docs/development/internal_documentation/room_and_user_statistics.md new file mode 100644 index 0000000000..cc38c890bb --- /dev/null +++ b/docs/development/internal_documentation/room_and_user_statistics.md @@ -0,0 +1,22 @@ +Room and User Statistics +======================== + +Synapse maintains room and user statistics in various tables. These can be used +for administrative purposes but are also used when generating the public room +directory. + + +# Synapse Developer Documentation + +## High-Level Concepts + +### Definitions + +* **subject**: Something we are tracking stats about – currently a room or user. +* **current row**: An entry for a subject in the appropriate current statistics + table. Each subject can have only one. + +### Overview + +Stats correspond to the present values. Current rows contain the most up-to-date +statistics for a room. Each subject can only have one entry. diff --git a/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot b/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot new file mode 100644 index 0000000000..978d579ada --- /dev/null +++ b/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot @@ -0,0 +1,32 @@ +digraph auth { + nodesep=0.5; + rankdir="RL"; + + C [label="Create (1,1)"]; + + BJ [label="Bob's Join (2,1)", color=red]; + BJ2 [label="Bob's Join (2,2)", color=red]; + BJ2 -> BJ [color=red, dir=none]; + + subgraph cluster_foo { + A1 [label="Alice's invite (4,1)", color=blue]; + A2 [label="Alice's Join (4,2)", color=blue]; + A3 [label="Alice's Join (4,3)", color=blue]; + A3 -> A2 -> A1 [color=blue, dir=none]; + color=none; + } + + PL1 [label="Power Level (3,1)", color=darkgreen]; + PL2 [label="Power Level (3,2)", color=darkgreen]; + PL2 -> PL1 [color=darkgreen, dir=none]; + + {rank = same; C; BJ; PL1; A1;} + + A1 -> C [color=grey]; + A1 -> BJ [color=grey]; + PL1 -> C [color=grey]; + BJ2 -> PL1 [penwidth=2]; + + A3 -> PL2 [penwidth=2]; + A1 -> PL1 -> BJ -> C [penwidth=2]; +} diff --git a/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot.png b/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot.png new file mode 100644 index 0000000000..771c07308f Binary files /dev/null and b/docs/development/internal_documentation/state_resolution/auth_chain_diff.dot.png differ diff --git a/docs/development/internal_documentation/state_resolution/auth_chain_difference_algorithm.md b/docs/development/internal_documentation/state_resolution/auth_chain_difference_algorithm.md new file mode 100644 index 0000000000..30f72a70da --- /dev/null +++ b/docs/development/internal_documentation/state_resolution/auth_chain_difference_algorithm.md @@ -0,0 +1,108 @@ +# Auth Chain Difference Algorithm + +The auth chain difference algorithm is used by V2 state resolution, where a +naive implementation can be a significant source of CPU and DB usage. + +### Definitions + +A *state set* is a set of state events; e.g. the input of a state resolution +algorithm is a collection of state sets. + +The *auth chain* of a set of events are all the events' auth events and *their* +auth events, recursively (i.e. the events reachable by walking the graph induced +by an event's auth events links). + +The *auth chain difference* of a collection of state sets is the union minus the +intersection of the sets of auth chains corresponding to the state sets, i.e an +event is in the auth chain difference if it is reachable by walking the auth +event graph from at least one of the state sets but not from *all* of the state +sets. + +## Breadth First Walk Algorithm + +A way of calculating the auth chain difference without calculating the full auth +chains for each state set is to do a parallel breadth first walk (ordered by +depth) of each state set's auth chain. By tracking which events are reachable +from each state set we can finish early if every pending event is reachable from +every state set. + +This can work well for state sets that have a small auth chain difference, but +can be very inefficient for larger differences. However, this algorithm is still +used if we don't have a chain cover index for the room (e.g. because we're in +the process of indexing it). + +## Chain Cover Index + +Synapse computes auth chain differences by pre-computing a "chain cover" index +for the auth chain in a room, allowing efficient reachability queries like "is +event A in the auth chain of event B". This is done by assigning every event a +*chain ID* and *sequence number* (e.g. `(5,3)`), and having a map of *links* +between chains (e.g. `(5,3) -> (2,4)`) such that A is reachable by B (i.e. `A` +is in the auth chain of `B`) if and only if either: + +1. A and B have the same chain ID and `A`'s sequence number is less than `B`'s + sequence number; or +2. there is a link `L` between `B`'s chain ID and `A`'s chain ID such that + `L.start_seq_no` <= `B.seq_no` and `A.seq_no` <= `L.end_seq_no`. + +There are actually two potential implementations, one where we store links from +each chain to every other reachable chain (the transitive closure of the links +graph), and one where we remove redundant links (the transitive reduction of the +links graph) e.g. if we have chains `C3 -> C2 -> C1` then the link `C3 -> C1` +would not be stored. Synapse uses the former implementations so that it doesn't +need to recurse to test reachability between chains. + +### Example + +An example auth graph would look like the following, where chains have been +formed based on type/state_key and are denoted by colour and are labelled with +`(chain ID, sequence number)`. Links are denoted by the arrows (links in grey +are those that would be remove in the second implementation described above). + +![Example](auth_chain_diff.dot.png) + +Note that we don't include all links between events and their auth events, as +most of those links would be redundant. For example, all events point to the +create event, but each chain only needs the one link from it's base to the +create event. + +## Using the Index + +This index can be used to calculate the auth chain difference of the state sets +by looking at the chain ID and sequence numbers reachable from each state set: + +1. For every state set lookup the chain ID/sequence numbers of each state event +2. Use the index to find all chains and the maximum sequence number reachable + from each state set. +3. The auth chain difference is then all events in each chain that have sequence + numbers between the maximum sequence number reachable from *any* state set and + the minimum reachable by *all* state sets (if any). + +Note that steps 2 is effectively calculating the auth chain for each state set +(in terms of chain IDs and sequence numbers), and step 3 is calculating the +difference between the union and intersection of the auth chains. + +### Worked Example + +For example, given the above graph, we can calculate the difference between +state sets consisting of: + +1. `S1`: Alice's invite `(4,1)` and Bob's second join `(2,2)`; and +2. `S2`: Alice's second join `(4,3)` and Bob's first join `(2,1)`. + +Using the index we see that the following auth chains are reachable from each +state set: + +1. `S1`: `(1,1)`, `(2,2)`, `(3,1)` & `(4,1)` +2. `S2`: `(1,1)`, `(2,1)`, `(3,2)` & `(4,3)` + +And so, for each the ranges that are in the auth chain difference: +1. Chain 1: None, (since everything can reach the create event). +2. Chain 2: The range `(1, 2]` (i.e. just `2`), as `1` is reachable by all state + sets and the maximum reachable is `2` (corresponding to Bob's second join). +3. Chain 3: Similarly the range `(1, 2]` (corresponding to the second power + level). +4. Chain 4: The range `(1, 3]` (corresponding to both of Alice's joins). + +So the final result is: Bob's second join `(2,2)`, the second power level +`(3,2)` and both of Alice's joins `(4,2)` & `(4,3)`. diff --git a/docs/development/opentracing.md b/docs/development/opentracing.md new file mode 100644 index 0000000000..f91362f112 --- /dev/null +++ b/docs/development/opentracing.md @@ -0,0 +1,93 @@ +# OpenTracing + +## Background + +OpenTracing is a semi-standard being adopted by a number of distributed +tracing platforms. It is a common api for facilitating vendor-agnostic +tracing instrumentation. That is, we can use the OpenTracing api and +select one of a number of tracer implementations to do the heavy lifting +in the background. Our current selected implementation is Jaeger. + +OpenTracing is a tool which gives an insight into the causal +relationship of work done in and between servers. The servers each track +events and report them to a centralised server - in Synapse's case: +Jaeger. The basic unit used to represent events is the span. The span +roughly represents a single piece of work that was done and the time at +which it occurred. A span can have child spans, meaning that the work of +the child had to be completed for the parent span to complete, or it can +have follow-on spans which represent work that is undertaken as a result +of the parent but is not depended on by the parent to in order to +finish. + +Since this is undertaken in a distributed environment a request to +another server, such as an RPC or a simple GET, can be considered a span +(a unit or work) for the local server. This causal link is what +OpenTracing aims to capture and visualise. In order to do this metadata +about the local server's span, i.e the 'span context', needs to be +included with the request to the remote. + +It is up to the remote server to decide what it does with the spans it +creates. This is called the sampling policy and it can be configured +through Jaeger's settings. + +For OpenTracing concepts see +. + +For more information about Jaeger's implementation see + + +## Setting up OpenTracing + +To receive OpenTracing spans, start up a Jaeger server. This can be done +using docker like so: + +```sh +docker run -d --name jaeger \ + -p 6831:6831/udp \ + -p 6832:6832/udp \ + -p 5778:5778 \ + -p 16686:16686 \ + -p 14268:14268 \ + jaegertracing/all-in-one:1 +``` + +Latest documentation is probably at +https://www.jaegertracing.io/docs/latest/getting-started. + +## Enable OpenTracing in Synapse + +OpenTracing is not enabled by default. It must be enabled in the +homeserver config by uncommenting the config options under `opentracing` +as shown in the [sample config](./sample_config.yaml). For example: + +```yaml +opentracing: + enabled: true + homeserver_whitelist: + - "mytrustedhomeserver.org" + - "*.myotherhomeservers.com" +``` + +## Homeserver whitelisting + +The homeserver whitelist is configured using regular expressions. A list +of regular expressions can be given and their union will be compared +when propagating any spans contexts to another homeserver. + +Though it's mostly safe to send and receive span contexts to and from +untrusted users since span contexts are usually opaque ids it can lead +to two problems, namely: + +- If the span context is marked as sampled by the sending homeserver + the receiver will sample it. Therefore two homeservers with wildly + different sampling policies could incur higher sampling counts than + intended. +- Sending servers can attach arbitrary data to spans, known as + 'baggage'. For safety this has been disabled in Synapse but that + doesn't prevent another server sending you baggage which will be + logged to OpenTracing's logs. + +## Configuring Jaeger + +Sampling strategies can be set as in this document: +. diff --git a/docs/development/synapse_architecture/log_contexts.md b/docs/development/synapse_architecture/log_contexts.md new file mode 100644 index 0000000000..cb15dbe158 --- /dev/null +++ b/docs/development/synapse_architecture/log_contexts.md @@ -0,0 +1,364 @@ +# Log Contexts + +To help track the processing of individual requests, synapse uses a +'`log context`' to track which request it is handling at any given +moment. This is done via a thread-local variable; a `logging.Filter` is +then used to fish the information back out of the thread-local variable +and add it to each log record. + +Logcontexts are also used for CPU and database accounting, so that we +can track which requests were responsible for high CPU use or database +activity. + +The `synapse.logging.context` module provides facilities for managing +the current log context (as well as providing the `LoggingContextFilter` +class). + +Asynchronous functions make the whole thing complicated, so this document describes +how it all works, and how to write code which follows the rules. + +In this document, "awaitable" refers to any object which can be `await`ed. In the context of +Synapse, that normally means either a coroutine or a Twisted +[`Deferred`](https://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferred.html). + +## Logcontexts without asynchronous code + +In the absence of any asynchronous voodoo, things are simple enough. As with +any code of this nature, the rule is that our function should leave +things as it found them: + +```python +from synapse.logging import context # omitted from future snippets + +def handle_request(request_id): + request_context = context.LoggingContext() + + calling_context = context.set_current_context(request_context) + try: + request_context.request = request_id + do_request_handling() + logger.debug("finished") + finally: + context.set_current_context(calling_context) + +def do_request_handling(): + logger.debug("phew") # this will be logged against request_id +``` + +LoggingContext implements the context management methods, so the above +can be written much more succinctly as: + +```python +def handle_request(request_id): + with context.LoggingContext() as request_context: + request_context.request = request_id + do_request_handling() + logger.debug("finished") + +def do_request_handling(): + logger.debug("phew") +``` + +## Using logcontexts with awaitables + +Awaitables break the linear flow of code so that there is no longer a single entry point +where we should set the logcontext and a single exit point where we should remove it. + +Consider the example above, where `do_request_handling` needs to do some +blocking operation, and returns an awaitable: + +```python +async def handle_request(request_id): + with context.LoggingContext() as request_context: + request_context.request = request_id + await do_request_handling() + logger.debug("finished") +``` + +In the above flow: + +- The logcontext is set +- `do_request_handling` is called, and returns an awaitable +- `handle_request` awaits the awaitable +- Execution of `handle_request` is suspended + +So we have stopped processing the request (and will probably go on to +start processing the next), without clearing the logcontext. + +To circumvent this problem, synapse code assumes that, wherever you have +an awaitable, you will want to `await` it. To that end, whereever +functions return awaitables, we adopt the following conventions: + +**Rules for functions returning awaitables:** + +> - If the awaitable is already complete, the function returns with the +> same logcontext it started with. +> - If the awaitable is incomplete, the function clears the logcontext +> before returning; when the awaitable completes, it restores the +> logcontext before running any callbacks. + +That sounds complicated, but actually it means a lot of code (including +the example above) "just works". There are two cases: + +- If `do_request_handling` returns a completed awaitable, then the + logcontext will still be in place. In this case, execution will + continue immediately after the `await`; the "finished" line will + be logged against the right context, and the `with` block restores + the original context before we return to the caller. +- If the returned awaitable is incomplete, `do_request_handling` clears + the logcontext before returning. The logcontext is therefore clear + when `handle_request` `await`s the awaitable. + + Once `do_request_handling`'s awaitable completes, it will reinstate + the logcontext, before running the second half of `handle_request`, + so again the "finished" line will be logged against the right context, + and the `with` block restores the original context. + +As an aside, it's worth noting that `handle_request` follows our rules +- though that only matters if the caller has its own logcontext which it +cares about. + +The following sections describe pitfalls and helpful patterns when +implementing these rules. + +Always await your awaitables +---------------------------- + +Whenever you get an awaitable back from a function, you should `await` on +it as soon as possible. Do not pass go; do not do any logging; do not +call any other functions. + +```python +async def fun(): + logger.debug("starting") + await do_some_stuff() # just like this + + coro = more_stuff() + result = await coro # also fine, of course + + return result +``` + +Provided this pattern is followed all the way back up to the callchain +to where the logcontext was set, this will make things work out ok: +provided `do_some_stuff` and `more_stuff` follow the rules above, then +so will `fun`. + +It's all too easy to forget to `await`: for instance if we forgot that +`do_some_stuff` returned an awaitable, we might plough on regardless. This +leads to a mess; it will probably work itself out eventually, but not +before a load of stuff has been logged against the wrong context. +(Normally, other things will break, more obviously, if you forget to +`await`, so this tends not to be a major problem in practice.) + +Of course sometimes you need to do something a bit fancier with your +awaitable - not all code follows the linear A-then-B-then-C pattern. +Notes on implementing more complex patterns are in later sections. + +## Where you create a new awaitable, make it follow the rules + +Most of the time, an awaitable comes from another synapse function. +Sometimes, though, we need to make up a new awaitable, or we get an awaitable +back from external code. We need to make it follow our rules. + +The easy way to do it is by using `context.make_deferred_yieldable`. Suppose we want to implement +`sleep`, which returns a deferred which will run its callbacks after a +given number of seconds. That might look like: + +```python +# not a logcontext-rules-compliant function +def get_sleep_deferred(seconds): + d = defer.Deferred() + reactor.callLater(seconds, d.callback, None) + return d +``` + +That doesn't follow the rules, but we can fix it by calling it through +`context.make_deferred_yieldable`: + +```python +async def sleep(seconds): + return await context.make_deferred_yieldable(get_sleep_deferred(seconds)) +``` + +## Fire-and-forget + +Sometimes you want to fire off a chain of execution, but not wait for +its result. That might look a bit like this: + +```python +async def do_request_handling(): + await foreground_operation() + + # *don't* do this + background_operation() + + logger.debug("Request handling complete") + +async def background_operation(): + await first_background_step() + logger.debug("Completed first step") + await second_background_step() + logger.debug("Completed second step") +``` + +The above code does a couple of steps in the background after +`do_request_handling` has finished. The log lines are still logged +against the `request_context` logcontext, which may or may not be +desirable. There are two big problems with the above, however. The first +problem is that, if `background_operation` returns an incomplete +awaitable, it will expect its caller to `await` immediately, so will have +cleared the logcontext. In this example, that means that 'Request +handling complete' will be logged without any context. + +The second problem, which is potentially even worse, is that when the +awaitable returned by `background_operation` completes, it will restore +the original logcontext. There is nothing waiting on that awaitable, so +the logcontext will leak into the reactor and possibly get attached to +some arbitrary future operation. + +There are two potential solutions to this. + +One option is to surround the call to `background_operation` with a +`PreserveLoggingContext` call. That will reset the logcontext before +starting `background_operation` (so the context restored when the +deferred completes will be the empty logcontext), and will restore the +current logcontext before continuing the foreground process: + +```python +async def do_request_handling(): + await foreground_operation() + + # start background_operation off in the empty logcontext, to + # avoid leaking the current context into the reactor. + with PreserveLoggingContext(): + background_operation() + + # this will now be logged against the request context + logger.debug("Request handling complete") +``` + +Obviously that option means that the operations done in +`background_operation` would be not be logged against a logcontext +(though that might be fixed by setting a different logcontext via a +`with LoggingContext(...)` in `background_operation`). + +The second option is to use `context.run_in_background`, which wraps a +function so that it doesn't reset the logcontext even when it returns +an incomplete awaitable, and adds a callback to the returned awaitable to +reset the logcontext. In other words, it turns a function that follows +the Synapse rules about logcontexts and awaitables into one which behaves +more like an external function --- the opposite operation to that +described in the previous section. It can be used like this: + +```python +async def do_request_handling(): + await foreground_operation() + + context.run_in_background(background_operation) + + # this will now be logged against the request context + logger.debug("Request handling complete") +``` + +## Passing synapse deferreds into third-party functions + +A typical example of this is where we want to collect together two or +more awaitables via `defer.gatherResults`: + +```python +a1 = operation1() +a2 = operation2() +a3 = defer.gatherResults([a1, a2]) +``` + +This is really a variation of the fire-and-forget problem above, in that +we are firing off `a1` and `a2` without awaiting on them. The difference +is that we now have third-party code attached to their callbacks. Anyway +either technique given in the [Fire-and-forget](#fire-and-forget) +section will work. + +Of course, the new awaitable returned by `gather` needs to be +wrapped in order to make it follow the logcontext rules before we can +yield it, as described in [Where you create a new awaitable, make it +follow the +rules](#where-you-create-a-new-awaitable-make-it-follow-the-rules). + +So, option one: reset the logcontext before starting the operations to +be gathered: + +```python +async def do_request_handling(): + with PreserveLoggingContext(): + a1 = operation1() + a2 = operation2() + result = await defer.gatherResults([a1, a2]) +``` + +In this case particularly, though, option two, of using +`context.run_in_background` almost certainly makes more sense, so that +`operation1` and `operation2` are both logged against the original +logcontext. This looks like: + +```python +async def do_request_handling(): + a1 = context.run_in_background(operation1) + a2 = context.run_in_background(operation2) + + result = await make_deferred_yieldable(defer.gatherResults([a1, a2])) +``` + +## A note on garbage-collection of awaitable chains + +It turns out that our logcontext rules do not play nicely with awaitable +chains which get orphaned and garbage-collected. + +Imagine we have some code that looks like this: + +```python +listener_queue = [] + +def on_something_interesting(): + for d in listener_queue: + d.callback("foo") + +async def await_something_interesting(): + new_awaitable = defer.Deferred() + listener_queue.append(new_awaitable) + + with PreserveLoggingContext(): + await new_awaitable +``` + +Obviously, the idea here is that we have a bunch of things which are +waiting for an event. (It's just an example of the problem here, but a +relatively common one.) + +Now let's imagine two further things happen. First of all, whatever was +waiting for the interesting thing goes away. (Perhaps the request times +out, or something *even more* interesting happens.) + +Secondly, let's suppose that we decide that the interesting thing is +never going to happen, and we reset the listener queue: + +```python +def reset_listener_queue(): + listener_queue.clear() +``` + +So, both ends of the awaitable chain have now dropped their references, +and the awaitable chain is now orphaned, and will be garbage-collected at +some point. Note that `await_something_interesting` is a coroutine, +which Python implements as a generator function. When Python +garbage-collects generator functions, it gives them a chance to +clean up by making the `await` (or `yield`) raise a `GeneratorExit` +exception. In our case, that means that the `__exit__` handler of +`PreserveLoggingContext` will carefully restore the request context, but +there is now nothing waiting for its return, so the request context is +never cleared. + +To reiterate, this problem only arises when *both* ends of a awaitable +chain are dropped. Dropping the the reference to an awaitable you're +supposed to be awaiting is bad practice, so this doesn't +actually happen too much. Unfortunately, when it does happen, it will +lead to leaked logcontexts which are incredibly hard to track down. diff --git a/docs/development/synapse_architecture/replication.md b/docs/development/synapse_architecture/replication.md new file mode 100644 index 0000000000..e82df0de8a --- /dev/null +++ b/docs/development/synapse_architecture/replication.md @@ -0,0 +1,37 @@ +# Replication Architecture + +## Motivation + +We'd like to be able to split some of the work that synapse does into +multiple python processes. In theory multiple synapse processes could +share a single postgresql database and we\'d scale up by running more +synapse processes. However much of synapse assumes that only one process +is interacting with the database, both for assigning unique identifiers +when inserting into tables, notifying components about new updates, and +for invalidating its caches. + +So running multiple copies of the current code isn't an option. One way +to run multiple processes would be to have a single writer process and +multiple reader processes connected to the same database. In order to do +this we'd need a way for the reader process to invalidate its in-memory +caches when an update happens on the writer. One way to do this is for +the writer to present an append-only log of updates which the readers +can consume to invalidate their caches and to push updates to listening +clients or pushers. + +Synapse already stores much of its data as an append-only log so that it +can correctly respond to `/sync` requests so the amount of code changes +needed to expose the append-only log to the readers should be fairly +minimal. + +## Architecture + +### The Replication Protocol + +See [the TCP replication documentation](tcp_replication.md). + +### The Slaved DataStore + +There are read-only version of the synapse storage layer in +`synapse/replication/slave/storage` that use the response of the +replication API to invalidate their caches. diff --git a/docs/development/synapse_architecture/tcp_replication.md b/docs/development/synapse_architecture/tcp_replication.md new file mode 100644 index 0000000000..15df949deb --- /dev/null +++ b/docs/development/synapse_architecture/tcp_replication.md @@ -0,0 +1,257 @@ +# TCP Replication + +## Motivation + +Previously the workers used an HTTP long poll mechanism to get updates +from the master, which had the problem of causing a lot of duplicate +work on the server. This TCP protocol replaces those APIs with the aim +of increased efficiency. + +## Overview + +The protocol is based on fire and forget, line based commands. An +example flow would be (where '>' indicates master to worker and +'<' worker to master flows): + + > SERVER example.com + < REPLICATE + > POSITION events master 53 53 + > RDATA events master 54 ["$foo1:bar.com", ...] + > RDATA events master 55 ["$foo4:bar.com", ...] + +The example shows the server accepting a new connection and sending its identity +with the `SERVER` command, followed by the client server to respond with the +position of all streams. The server then periodically sends `RDATA` commands +which have the format `RDATA `, where +the format of `` is defined by the individual streams. The +`` is the name of the Synapse process that generated the data +(usually "master"). + +Error reporting happens by either the client or server sending an ERROR +command, and usually the connection will be closed. + +Since the protocol is a simple line based, its possible to manually +connect to the server using a tool like netcat. A few things should be +noted when manually using the protocol: + +- The federation stream is only available if federation sending has + been disabled on the main process. +- The server will only time connections out that have sent a `PING` + command. If a ping is sent then the connection will be closed if no + further commands are receieved within 15s. Both the client and + server protocol implementations will send an initial PING on + connection and ensure at least one command every 5s is sent (not + necessarily `PING`). +- `RDATA` commands *usually* include a numeric token, however if the + stream has multiple rows to replicate per token the server will send + multiple `RDATA` commands, with all but the last having a token of + `batch`. See the documentation on `commands.RdataCommand` for + further details. + +## Architecture + +The basic structure of the protocol is line based, where the initial +word of each line specifies the command. The rest of the line is parsed +based on the command. For example, the RDATA command is defined as: + + RDATA + +(Note that may contains spaces, but cannot contain +newlines.) + +Blank lines are ignored. + +### Keep alives + +Both sides are expected to send at least one command every 5s or so, and +should send a `PING` command if necessary. If either side do not receive +a command within e.g. 15s then the connection should be closed. + +Because the server may be connected to manually using e.g. netcat, the +timeouts aren't enabled until an initial `PING` command is seen. Both +the client and server implementations below send a `PING` command +immediately on connection to ensure the timeouts are enabled. + +This ensures that both sides can quickly realize if the tcp connection +has gone and handle the situation appropriately. + +### Start up + +When a new connection is made, the server: + +- Sends a `SERVER` command, which includes the identity of the server, + allowing the client to detect if its connected to the expected + server +- Sends a `PING` command as above, to enable the client to time out + connections promptly. + +The client: + +- Sends a `NAME` command, allowing the server to associate a human + friendly name with the connection. This is optional. +- Sends a `PING` as above +- Sends a `REPLICATE` to get the current position of all streams. +- On receipt of a `SERVER` command, checks that the server name + matches the expected server name. + +### Error handling + +If either side detects an error it can send an `ERROR` command and close +the connection. + +If the client side loses the connection to the server it should +reconnect, following the steps above. + +### Congestion + +If the server sends messages faster than the client can consume them the +server will first buffer a (fairly large) number of commands and then +disconnect the client. This ensures that we don't queue up an unbounded +number of commands in memory and gives us a potential oppurtunity to +squawk loudly. When/if the client recovers it can reconnect to the +server and ask for missed messages. + +### Reliability + +In general the replication stream should be considered an unreliable +transport since e.g. commands are not resent if the connection +disappears. + +The exception to that are the replication streams, i.e. RDATA commands, +since these include tokens which can be used to restart the stream on +connection errors. + +The client should keep track of the token in the last RDATA command +received for each stream so that on reconneciton it can start streaming +from the correct place. Note: not all RDATA have valid tokens due to +batching. See `RdataCommand` for more details. + +### Example + +An example iteraction is shown below. Each line is prefixed with '>' +or '<' to indicate which side is sending, these are *not* included on +the wire: + + * connection established * + > SERVER localhost:8823 + > PING 1490197665618 + < NAME synapse.app.appservice + < PING 1490197665618 + < REPLICATE + > POSITION events master 1 1 + > POSITION backfill master 1 1 + > POSITION caches master 1 1 + > RDATA caches master 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513] + > RDATA events master 14 ["$149019767112vOHxz:localhost:8823", + "!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null] + < PING 1490197675618 + > ERROR server stopping + * connection closed by server * + +The `POSITION` command sent by the server is used to set the clients +position without needing to send data with the `RDATA` command. + +An example of a batched set of `RDATA` is: + + > RDATA caches master batch ["get_user_by_id",["@test:localhost:8823"],1490197670513] + > RDATA caches master batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513] + > RDATA caches master batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513] + > RDATA caches master 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513] + +In this case the client shouldn't advance their caches token until it +sees the the last `RDATA`. + +### List of commands + +The list of valid commands, with which side can send it: server (S) or +client (C): + +#### SERVER (S) + + Sent at the start to identify which server the client is talking to + +#### RDATA (S) + + A single update in a stream + +#### POSITION (S) + + On receipt of a POSITION command clients should check if they have missed any + updates, and if so then fetch them out of band. Sent in response to a + REPLICATE command (but can happen at any time). + + The POSITION command includes the source of the stream. Currently all streams + are written by a single process (usually "master"). If fetching missing + updates via HTTP API, rather than via the DB, then processes should make the + request to the appropriate process. + + Two positions are included, the "new" position and the last position sent respectively. + This allows servers to tell instances that the positions have advanced but no + data has been written, without clients needlessly checking to see if they + have missed any updates. + +#### ERROR (S, C) + + There was an error + +#### PING (S, C) + + Sent periodically to ensure the connection is still alive + +#### NAME (C) + + Sent at the start by client to inform the server who they are + +#### REPLICATE (C) + +Asks the server for the current position of all streams. + +#### USER_SYNC (C) + + A user has started or stopped syncing on this process. + +#### CLEAR_USER_SYNC (C) + + The server should clear all associated user sync data from the worker. + + This is used when a worker is shutting down. + +#### FEDERATION_ACK (C) + + Acknowledge receipt of some federation data + +### REMOTE_SERVER_UP (S, C) + + Inform other processes that a remote server may have come back online. + +See `synapse/replication/tcp/commands.py` for a detailed description and +the format of each command. + +### Cache Invalidation Stream + +The cache invalidation stream is used to inform workers when they need +to invalidate any of their caches in the data store. This is done by +streaming all cache invalidations done on master down to the workers, +assuming that any caches on the workers also exist on the master. + +Each individual cache invalidation results in a row being sent down +replication, which includes the cache name (the name of the function) +and they key to invalidate. For example: + + > RDATA caches master 550953771 ["get_user_by_id", ["@bob:example.com"], 1550574873251] + +Alternatively, an entire cache can be invalidated by sending down a `null` +instead of the key. For example: + + > RDATA caches master 550953772 ["get_user_by_id", null, 1550574873252] + +However, there are times when a number of caches need to be invalidated +at the same time with the same key. To reduce traffic we batch those +invalidations into a single poke by defining a special cache name that +workers understand to mean to expand to invalidate the correct caches. + +Currently the special cache names are declared in +`synapse/storage/_base.py` and are: + +1. `cs_cache_fake` ─ invalidates caches that depend on the current + state diff --git a/docs/federate.md b/docs/federate.md deleted file mode 100644 index 5107f995be..0000000000 --- a/docs/federate.md +++ /dev/null @@ -1,66 +0,0 @@ -Setting up federation -===================== - -Federation is the process by which users on different servers can participate -in the same room. For this to work, those other servers must be able to contact -yours to send messages. - -The `server_name` configured in the Synapse configuration file (often -`homeserver.yaml`) defines how resources (users, rooms, etc.) will be -identified (eg: `@user:example.com`, `#room:example.com`). By default, -it is also the domain that other servers will use to try to reach your -server (via port 8448). This is easy to set up and will work provided -you set the `server_name` to match your machine's public DNS hostname. - -For this default configuration to work, you will need to listen for TLS -connections on port 8448. The preferred way to do that is by using a -reverse proxy: see [the reverse proxy documentation](reverse_proxy.md) for instructions -on how to correctly set one up. - -In some cases you might not want to run Synapse on the machine that has -the `server_name` as its public DNS hostname, or you might want federation -traffic to use a different port than 8448. For example, you might want to -have your user names look like `@user:example.com`, but you want to run -Synapse on `synapse.example.com` on port 443. This can be done using -delegation, which allows an admin to control where federation traffic should -be sent. See [the delegation documentation](delegate.md) for instructions on how to set this up. - -Once federation has been configured, you should be able to join a room over -federation. A good place to start is `#synapse:matrix.org` - a room for -Synapse admins. - -## Troubleshooting - -You can use the [federation tester](https://matrix.org/federationtester) -to check if your homeserver is configured correctly. Alternatively try the -[JSON API used by the federation tester](https://matrix.org/federationtester/api/report?server_name=DOMAIN). -Note that you'll have to modify this URL to replace `DOMAIN` with your -`server_name`. Hitting the API directly provides extra detail. - -The typical failure mode for federation is that when the server tries to join -a room, it is rejected with "401: Unauthorized". Generally this means that other -servers in the room could not access yours. (Joining a room over federation is -a complicated dance which requires connections in both directions). - -Another common problem is that people on other servers can't join rooms that -you invite them to. This can be caused by an incorrectly-configured reverse -proxy: see [the reverse proxy documentation](reverse_proxy.md) for instructions on how -to correctly configure a reverse proxy. - -### Known issues - -**HTTP `308 Permanent Redirect` redirects are not followed**: Due to missing features -in the HTTP library used by Synapse, 308 redirects are currently not followed by -federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This -may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`), -and especially users of the Kubernetes *Nginx Ingress* module, which uses 308 redirect -codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871) -might be helpful. For other users, switching to a `301 Moved Permanently` code may be -an option. 308 redirect codes will be supported properly in a future -release of Synapse. - -## Running a demo federation of Synapses - -If you want to get up and running quickly with a trio of homeservers in a -private federation, there is a script in the `demo` directory. This is mainly -useful just for development purposes. See [demo/README](https://github.com/matrix-org/synapse/tree/develop/demo/). diff --git a/docs/introduction/welcome_and_overview.md b/docs/introduction/welcome_and_overview.md new file mode 100644 index 0000000000..aab2d6b4f0 --- /dev/null +++ b/docs/introduction/welcome_and_overview.md @@ -0,0 +1,79 @@ +# Introduction + +Welcome to the documentation repository for Synapse, a +[Matrix](https://matrix.org) homeserver implementation developed by the matrix.org core +team. + +## Installing and using Synapse + +This documentation covers topics for **installation**, **configuration** and +**maintainence** of your Synapse process: + +* Learn how to [install](setup/installation.md) and + [configure](usage/configuration/index.html) your own instance, perhaps with [Single + Sign-On](usage/configuration/user_authentication/index.html). + +* See how to [upgrade](upgrade.md) between Synapse versions. + +* Administer your instance using the [Admin + API](usage/administration/admin_api/index.html), installing [pluggable + modules](modules/index.html), or by accessing the [manhole](manhole.md). + +* Learn how to [read log lines](usage/administration/request_log.md), configure + [logging](usage/configuration/logging_sample_config.md) or set up [structured + logging](structured_logging.md). + +* Scale Synapse through additional [worker processes](workers.md). + +* Set up [monitoring and metrics](metrics-howto.md) to keep an eye on your + Synapse instance's performance. + +## Developing on Synapse + +Contributions are welcome! Synapse is primarily written in +[Python](https://python.org). As a developer, you may be interested in the +following documentation: + +* Read the [Contributing Guide](development/contributing_guide.md). It is meant + to walk new contributors through the process of developing and submitting a + change to the Synapse codebase (which is [hosted on + GitHub](https://github.com/matrix-org/synapse)). + +* Set up your [development + environment](development/contributing_guide.md#2-what-do-i-need), then learn + how to [lint](development/contributing_guide.md#run-the-linters) and + [test](development/contributing_guide.md#8-test-test-test) your code. + +* Look at [the issue tracker](https://github.com/matrix-org/synapse/issues) for + bugs to fix or features to add. If you're new, it may be best to start with + those labeled [good first + issue](https://github.com/matrix-org/synapse/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). + +* Understand [how Synapse is + built](development/internal_documentation/index.html), how to [migrate + database schemas](development/database_schema.md), learn about + [federation](federate.md) and how to [set up a local + federation](federate.md#running-a-demo-federation-of-synapses) for development. + +* We like to keep our `git` history clean. [Learn](development/git.md) how to + do so! + +* And finally, contribute to this documentation! The source for which is + [located here](https://github.com/matrix-org/synapse/tree/develop/docs). + +## Donating to Synapse development + +Want to help keep Synapse going but don't know how to code? Synapse is a +[Matrix.org Foundation](https://matrix.org) project. Consider becoming a +supportor on [Liberapay](https://liberapay.com/matrixdotorg), +[Patreon](https://patreon.com/matrixdotorg) or through +[PayPal](https://paypal.me/matrixdotorg) via a one-time donation. + +If you are an organisation or enterprise and would like to sponsor development, +reach out to us over email at: support (at) matrix.org + +## Reporting a security vulnerability + +If you've found a security issue in Synapse or any other Matrix.org Foundation +project, please report it to us in accordance with our [Security Disclosure +Policy](https://www.matrix.org/security-disclosure-policy/). Thank you! diff --git a/docs/jwt.md b/docs/jwt.md deleted file mode 100644 index 5be9fd26e3..0000000000 --- a/docs/jwt.md +++ /dev/null @@ -1,97 +0,0 @@ -# JWT Login Type - -Synapse comes with a non-standard login type to support -[JSON Web Tokens](https://en.wikipedia.org/wiki/JSON_Web_Token). In general the -documentation for -[the login endpoint](https://matrix.org/docs/spec/client_server/r0.6.1#login) -is still valid (and the mechanism works similarly to the -[token based login](https://matrix.org/docs/spec/client_server/r0.6.1#token-based)). - -To log in using a JSON Web Token, clients should submit a `/login` request as -follows: - -```json -{ - "type": "org.matrix.login.jwt", - "token": "" -} -``` - -Note that the login type of `m.login.jwt` is supported, but is deprecated. This -will be removed in a future version of Synapse. - -The `token` field should include the JSON web token with the following claims: - -* The `sub` (subject) claim is required and should encode the local part of the - user ID. -* The expiration time (`exp`), not before time (`nbf`), and issued at (`iat`) - claims are optional, but validated if present. -* The issuer (`iss`) claim is optional, but required and validated if configured. -* The audience (`aud`) claim is optional, but required and validated if configured. - Providing the audience claim when not configured will cause validation to fail. - -In the case that the token is not valid, the homeserver must respond with -`403 Forbidden` and an error code of `M_FORBIDDEN`. - -As with other login types, there are additional fields (e.g. `device_id` and -`initial_device_display_name`) which can be included in the above request. - -## Preparing Synapse - -The JSON Web Token integration in Synapse uses the -[`PyJWT`](https://pypi.org/project/pyjwt/) library, which must be installed -as follows: - - * The relevant libraries are included in the Docker images and Debian packages - provided by `matrix.org` so no further action is needed. - - * If you installed Synapse into a virtualenv, run `/path/to/env/bin/pip - install synapse[pyjwt]` to install the necessary dependencies. - - * For other installation mechanisms, see the documentation provided by the - maintainer. - -To enable the JSON web token integration, you should then add an `jwt_config` section -to your configuration file (or uncomment the `enabled: true` line in the -existing section). See [sample_config.yaml](./sample_config.yaml) for some -sample settings. - -## How to test JWT as a developer - -Although JSON Web Tokens are typically generated from an external server, the -examples below use [PyJWT](https://pyjwt.readthedocs.io/en/latest/) directly. - -1. Configure Synapse with JWT logins, note that this example uses a pre-shared - secret and an algorithm of HS256: - - ```yaml - jwt_config: - enabled: true - secret: "my-secret-token" - algorithm: "HS256" - ``` -2. Generate a JSON web token: - - ```bash - $ pyjwt --key=my-secret-token --alg=HS256 encode sub=test-user - eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc - ``` -3. Query for the login types and ensure `org.matrix.login.jwt` is there: - - ```bash - curl http://localhost:8080/_matrix/client/r0/login - ``` -4. Login used the generated JSON web token from above: - - ```bash - $ curl http://localhost:8082/_matrix/client/r0/login -X POST \ - --data '{"type":"org.matrix.login.jwt","token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc"}' - { - "access_token": "", - "device_id": "ACBDEFGHI", - "home_server": "localhost:8080", - "user_id": "@test-user:localhost:8480" - } - ``` - -You should now be able to use the returned access token to query the client API. diff --git a/docs/log_contexts.md b/docs/log_contexts.md deleted file mode 100644 index cb15dbe158..0000000000 --- a/docs/log_contexts.md +++ /dev/null @@ -1,364 +0,0 @@ -# Log Contexts - -To help track the processing of individual requests, synapse uses a -'`log context`' to track which request it is handling at any given -moment. This is done via a thread-local variable; a `logging.Filter` is -then used to fish the information back out of the thread-local variable -and add it to each log record. - -Logcontexts are also used for CPU and database accounting, so that we -can track which requests were responsible for high CPU use or database -activity. - -The `synapse.logging.context` module provides facilities for managing -the current log context (as well as providing the `LoggingContextFilter` -class). - -Asynchronous functions make the whole thing complicated, so this document describes -how it all works, and how to write code which follows the rules. - -In this document, "awaitable" refers to any object which can be `await`ed. In the context of -Synapse, that normally means either a coroutine or a Twisted -[`Deferred`](https://twistedmatrix.com/documents/current/api/twisted.internet.defer.Deferred.html). - -## Logcontexts without asynchronous code - -In the absence of any asynchronous voodoo, things are simple enough. As with -any code of this nature, the rule is that our function should leave -things as it found them: - -```python -from synapse.logging import context # omitted from future snippets - -def handle_request(request_id): - request_context = context.LoggingContext() - - calling_context = context.set_current_context(request_context) - try: - request_context.request = request_id - do_request_handling() - logger.debug("finished") - finally: - context.set_current_context(calling_context) - -def do_request_handling(): - logger.debug("phew") # this will be logged against request_id -``` - -LoggingContext implements the context management methods, so the above -can be written much more succinctly as: - -```python -def handle_request(request_id): - with context.LoggingContext() as request_context: - request_context.request = request_id - do_request_handling() - logger.debug("finished") - -def do_request_handling(): - logger.debug("phew") -``` - -## Using logcontexts with awaitables - -Awaitables break the linear flow of code so that there is no longer a single entry point -where we should set the logcontext and a single exit point where we should remove it. - -Consider the example above, where `do_request_handling` needs to do some -blocking operation, and returns an awaitable: - -```python -async def handle_request(request_id): - with context.LoggingContext() as request_context: - request_context.request = request_id - await do_request_handling() - logger.debug("finished") -``` - -In the above flow: - -- The logcontext is set -- `do_request_handling` is called, and returns an awaitable -- `handle_request` awaits the awaitable -- Execution of `handle_request` is suspended - -So we have stopped processing the request (and will probably go on to -start processing the next), without clearing the logcontext. - -To circumvent this problem, synapse code assumes that, wherever you have -an awaitable, you will want to `await` it. To that end, whereever -functions return awaitables, we adopt the following conventions: - -**Rules for functions returning awaitables:** - -> - If the awaitable is already complete, the function returns with the -> same logcontext it started with. -> - If the awaitable is incomplete, the function clears the logcontext -> before returning; when the awaitable completes, it restores the -> logcontext before running any callbacks. - -That sounds complicated, but actually it means a lot of code (including -the example above) "just works". There are two cases: - -- If `do_request_handling` returns a completed awaitable, then the - logcontext will still be in place. In this case, execution will - continue immediately after the `await`; the "finished" line will - be logged against the right context, and the `with` block restores - the original context before we return to the caller. -- If the returned awaitable is incomplete, `do_request_handling` clears - the logcontext before returning. The logcontext is therefore clear - when `handle_request` `await`s the awaitable. - - Once `do_request_handling`'s awaitable completes, it will reinstate - the logcontext, before running the second half of `handle_request`, - so again the "finished" line will be logged against the right context, - and the `with` block restores the original context. - -As an aside, it's worth noting that `handle_request` follows our rules -- though that only matters if the caller has its own logcontext which it -cares about. - -The following sections describe pitfalls and helpful patterns when -implementing these rules. - -Always await your awaitables ----------------------------- - -Whenever you get an awaitable back from a function, you should `await` on -it as soon as possible. Do not pass go; do not do any logging; do not -call any other functions. - -```python -async def fun(): - logger.debug("starting") - await do_some_stuff() # just like this - - coro = more_stuff() - result = await coro # also fine, of course - - return result -``` - -Provided this pattern is followed all the way back up to the callchain -to where the logcontext was set, this will make things work out ok: -provided `do_some_stuff` and `more_stuff` follow the rules above, then -so will `fun`. - -It's all too easy to forget to `await`: for instance if we forgot that -`do_some_stuff` returned an awaitable, we might plough on regardless. This -leads to a mess; it will probably work itself out eventually, but not -before a load of stuff has been logged against the wrong context. -(Normally, other things will break, more obviously, if you forget to -`await`, so this tends not to be a major problem in practice.) - -Of course sometimes you need to do something a bit fancier with your -awaitable - not all code follows the linear A-then-B-then-C pattern. -Notes on implementing more complex patterns are in later sections. - -## Where you create a new awaitable, make it follow the rules - -Most of the time, an awaitable comes from another synapse function. -Sometimes, though, we need to make up a new awaitable, or we get an awaitable -back from external code. We need to make it follow our rules. - -The easy way to do it is by using `context.make_deferred_yieldable`. Suppose we want to implement -`sleep`, which returns a deferred which will run its callbacks after a -given number of seconds. That might look like: - -```python -# not a logcontext-rules-compliant function -def get_sleep_deferred(seconds): - d = defer.Deferred() - reactor.callLater(seconds, d.callback, None) - return d -``` - -That doesn't follow the rules, but we can fix it by calling it through -`context.make_deferred_yieldable`: - -```python -async def sleep(seconds): - return await context.make_deferred_yieldable(get_sleep_deferred(seconds)) -``` - -## Fire-and-forget - -Sometimes you want to fire off a chain of execution, but not wait for -its result. That might look a bit like this: - -```python -async def do_request_handling(): - await foreground_operation() - - # *don't* do this - background_operation() - - logger.debug("Request handling complete") - -async def background_operation(): - await first_background_step() - logger.debug("Completed first step") - await second_background_step() - logger.debug("Completed second step") -``` - -The above code does a couple of steps in the background after -`do_request_handling` has finished. The log lines are still logged -against the `request_context` logcontext, which may or may not be -desirable. There are two big problems with the above, however. The first -problem is that, if `background_operation` returns an incomplete -awaitable, it will expect its caller to `await` immediately, so will have -cleared the logcontext. In this example, that means that 'Request -handling complete' will be logged without any context. - -The second problem, which is potentially even worse, is that when the -awaitable returned by `background_operation` completes, it will restore -the original logcontext. There is nothing waiting on that awaitable, so -the logcontext will leak into the reactor and possibly get attached to -some arbitrary future operation. - -There are two potential solutions to this. - -One option is to surround the call to `background_operation` with a -`PreserveLoggingContext` call. That will reset the logcontext before -starting `background_operation` (so the context restored when the -deferred completes will be the empty logcontext), and will restore the -current logcontext before continuing the foreground process: - -```python -async def do_request_handling(): - await foreground_operation() - - # start background_operation off in the empty logcontext, to - # avoid leaking the current context into the reactor. - with PreserveLoggingContext(): - background_operation() - - # this will now be logged against the request context - logger.debug("Request handling complete") -``` - -Obviously that option means that the operations done in -`background_operation` would be not be logged against a logcontext -(though that might be fixed by setting a different logcontext via a -`with LoggingContext(...)` in `background_operation`). - -The second option is to use `context.run_in_background`, which wraps a -function so that it doesn't reset the logcontext even when it returns -an incomplete awaitable, and adds a callback to the returned awaitable to -reset the logcontext. In other words, it turns a function that follows -the Synapse rules about logcontexts and awaitables into one which behaves -more like an external function --- the opposite operation to that -described in the previous section. It can be used like this: - -```python -async def do_request_handling(): - await foreground_operation() - - context.run_in_background(background_operation) - - # this will now be logged against the request context - logger.debug("Request handling complete") -``` - -## Passing synapse deferreds into third-party functions - -A typical example of this is where we want to collect together two or -more awaitables via `defer.gatherResults`: - -```python -a1 = operation1() -a2 = operation2() -a3 = defer.gatherResults([a1, a2]) -``` - -This is really a variation of the fire-and-forget problem above, in that -we are firing off `a1` and `a2` without awaiting on them. The difference -is that we now have third-party code attached to their callbacks. Anyway -either technique given in the [Fire-and-forget](#fire-and-forget) -section will work. - -Of course, the new awaitable returned by `gather` needs to be -wrapped in order to make it follow the logcontext rules before we can -yield it, as described in [Where you create a new awaitable, make it -follow the -rules](#where-you-create-a-new-awaitable-make-it-follow-the-rules). - -So, option one: reset the logcontext before starting the operations to -be gathered: - -```python -async def do_request_handling(): - with PreserveLoggingContext(): - a1 = operation1() - a2 = operation2() - result = await defer.gatherResults([a1, a2]) -``` - -In this case particularly, though, option two, of using -`context.run_in_background` almost certainly makes more sense, so that -`operation1` and `operation2` are both logged against the original -logcontext. This looks like: - -```python -async def do_request_handling(): - a1 = context.run_in_background(operation1) - a2 = context.run_in_background(operation2) - - result = await make_deferred_yieldable(defer.gatherResults([a1, a2])) -``` - -## A note on garbage-collection of awaitable chains - -It turns out that our logcontext rules do not play nicely with awaitable -chains which get orphaned and garbage-collected. - -Imagine we have some code that looks like this: - -```python -listener_queue = [] - -def on_something_interesting(): - for d in listener_queue: - d.callback("foo") - -async def await_something_interesting(): - new_awaitable = defer.Deferred() - listener_queue.append(new_awaitable) - - with PreserveLoggingContext(): - await new_awaitable -``` - -Obviously, the idea here is that we have a bunch of things which are -waiting for an event. (It's just an example of the problem here, but a -relatively common one.) - -Now let's imagine two further things happen. First of all, whatever was -waiting for the interesting thing goes away. (Perhaps the request times -out, or something *even more* interesting happens.) - -Secondly, let's suppose that we decide that the interesting thing is -never going to happen, and we reset the listener queue: - -```python -def reset_listener_queue(): - listener_queue.clear() -``` - -So, both ends of the awaitable chain have now dropped their references, -and the awaitable chain is now orphaned, and will be garbage-collected at -some point. Note that `await_something_interesting` is a coroutine, -which Python implements as a generator function. When Python -garbage-collects generator functions, it gives them a chance to -clean up by making the `await` (or `yield`) raise a `GeneratorExit` -exception. In our case, that means that the `__exit__` handler of -`PreserveLoggingContext` will carefully restore the request context, but -there is now nothing waiting for its return, so the request context is -never cleared. - -To reiterate, this problem only arises when *both* ends of a awaitable -chain are dropped. Dropping the the reference to an awaitable you're -supposed to be awaiting is bad practice, so this doesn't -actually happen too much. Unfortunately, when it does happen, it will -lead to leaked logcontexts which are incredibly hard to track down. diff --git a/docs/manhole.md b/docs/manhole.md deleted file mode 100644 index 715ed840f2..0000000000 --- a/docs/manhole.md +++ /dev/null @@ -1,99 +0,0 @@ -Using the synapse manhole -========================= - -The "manhole" allows server administrators to access a Python shell on a running -Synapse installation. This is a very powerful mechanism for administration and -debugging. - -**_Security Warning_** - -Note that this will give administrative access to synapse to **all users** with -shell access to the server. It should therefore **not** be enabled in -environments where untrusted users have shell access. - -## Configuring the manhole - -To enable it, first uncomment the `manhole` listener configuration in -`homeserver.yaml`. The configuration is slightly different if you're using docker. - -#### Docker config - -If you are using Docker, set `bind_addresses` to `['0.0.0.0']` as shown: - -```yaml -listeners: - - port: 9000 - bind_addresses: ['0.0.0.0'] - type: manhole -``` - -When using `docker run` to start the server, you will then need to change the command to the following to include the -`manhole` port forwarding. The `-p 127.0.0.1:9000:9000` below is important: it -ensures that access to the `manhole` is only possible for local users. - -```bash -docker run -d --name synapse \ - --mount type=volume,src=synapse-data,dst=/data \ - -p 8008:8008 \ - -p 127.0.0.1:9000:9000 \ - matrixdotorg/synapse:latest -``` - -#### Native config - -If you are not using docker, set `bind_addresses` to `['::1', '127.0.0.1']` as shown. -The `bind_addresses` in the example below is important: it ensures that access to the -`manhole` is only possible for local users). - -```yaml -listeners: - - port: 9000 - bind_addresses: ['::1', '127.0.0.1'] - type: manhole -``` - -### Security settings - -The following config options are available: - -- `username` - The username for the manhole (defaults to `matrix`) -- `password` - The password for the manhole (defaults to `rabbithole`) -- `ssh_priv_key` - The path to a private SSH key (defaults to a hardcoded value) -- `ssh_pub_key` - The path to a public SSH key (defaults to a hardcoded value) - -For example: - -```yaml -manhole_settings: - username: manhole - password: mypassword - ssh_priv_key: "/home/synapse/manhole_keys/id_rsa" - ssh_pub_key: "/home/synapse/manhole_keys/id_rsa.pub" -``` - - -## Accessing synapse manhole - -Then restart synapse, and point an ssh client at port 9000 on localhost, using -the username and password configured in `homeserver.yaml` - with the default -configuration, this would be: - -```bash -ssh -p9000 matrix@localhost -``` - -Then enter the password when prompted (the default is `rabbithole`). - -This gives a Python REPL in which `hs` gives access to the -`synapse.server.HomeServer` object - which in turn gives access to many other -parts of the process. - -Note that, prior to Synapse 1.41, any call which returns a coroutine will need to be wrapped in `ensureDeferred`. - -As a simple example, retrieving an event from the database: - -```pycon ->>> from twisted.internet import defer ->>> defer.ensureDeferred(hs.get_datastore().get_event('$1416420717069yeQaw:matrix.org')) -> -``` diff --git a/docs/media_repository.md b/docs/media_repository.md deleted file mode 100644 index 99ee8f1ef7..0000000000 --- a/docs/media_repository.md +++ /dev/null @@ -1,30 +0,0 @@ -# Media Repository - -*Synapse implementation-specific details for the media repository* - -The media repository is where attachments and avatar photos are stored. -It stores attachment content and thumbnails for media uploaded by local users. -It caches attachment content and thumbnails for media uploaded by remote users. - -## Storage - -Each item of media is assigned a `media_id` when it is uploaded. -The `media_id` is a randomly chosen, URL safe 24 character string. - -Metadata such as the MIME type, upload time and length are stored in the -sqlite3 database indexed by `media_id`. - -Content is stored on the filesystem under a `"local_content"` directory. - -Thumbnails are stored under a `"local_thumbnails"` directory. - -The item with `media_id` `"aabbccccccccdddddddddddd"` is stored under -`"local_content/aa/bb/ccccccccdddddddddddd"`. Its thumbnail with width -`128` and height `96` and type `"image/jpeg"` is stored under -`"local_thumbnails/aa/bb/ccccccccdddddddddddd/128-96-image-jpeg"` - -Remote content is cached under `"remote_content"` directory. Each item of -remote content is assigned a local `"filesystem_id"` to ensure that the -directory structure `"remote_content/server_name/aa/bb/ccccccccdddddddddddd"` -is appropriate. Thumbnails for remote content are stored under -`"remote_thumbnail/server_name/..."` diff --git a/docs/message_retention_policies.md b/docs/message_retention_policies.md deleted file mode 100644 index 9214d6d7e9..0000000000 --- a/docs/message_retention_policies.md +++ /dev/null @@ -1,205 +0,0 @@ -# Message retention policies - -Synapse admins can enable support for message retention policies on -their homeserver. Message retention policies exist at a room level, -follow the semantics described in -[MSC1763](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1763/proposals/1763-configurable-retention-periods.md), -and allow server and room admins to configure how long messages should -be kept in a homeserver's database before being purged from it. -**Please note that, as this feature isn't part of the Matrix -specification yet, this implementation is to be considered as -experimental.** - -A message retention policy is mainly defined by its `max_lifetime` -parameter, which defines how long a message can be kept around after -it was sent to the room. If a room doesn't have a message retention -policy, and there's no default one for a given server, then no message -sent in that room is ever purged on that server. - -MSC1763 also specifies semantics for a `min_lifetime` parameter which -defines the amount of time after which an event _can_ get purged (after -it was sent to the room), but Synapse doesn't currently support it -beyond registering it. - -Both `max_lifetime` and `min_lifetime` are optional parameters. - -Note that message retention policies don't apply to state events. - -Once an event reaches its expiry date (defined as the time it was sent -plus the value for `max_lifetime` in the room), two things happen: - -* Synapse stops serving the event to clients via any endpoint. -* The message gets picked up by the next purge job (see the "Purge jobs" - section) and is removed from Synapse's database. - -Since purge jobs don't run continuously, this means that an event might -stay in a server's database for longer than the value for `max_lifetime` -in the room would allow, though hidden from clients. - -Similarly, if a server (with support for message retention policies -enabled) receives from another server an event that should have been -purged according to its room's policy, then the receiving server will -process and store that event until it's picked up by the next purge job, -though it will always hide it from clients. - -Synapse requires at least one message in each room, so it will never -delete the last message in a room. It will, however, hide it from -clients. - - -## Server configuration - -Support for this feature can be enabled and configured in the -`retention` section of the Synapse configuration file (see the -[sample file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518)). - -To enable support for message retention policies, set the setting -`enabled` in this section to `true`. - - -### Default policy - -A default message retention policy is a policy defined in Synapse's -configuration that is used by Synapse for every room that doesn't have a -message retention policy configured in its state. This allows server -admins to ensure that messages are never kept indefinitely in a server's -database. - -A default policy can be defined as such, in the `retention` section of -the configuration file: - -```yaml -default_policy: - min_lifetime: 1d - max_lifetime: 1y -``` - -Here, `min_lifetime` and `max_lifetime` have the same meaning and level -of support as previously described. They can be expressed either as a -duration (using the units `s` (seconds), `m` (minutes), `h` (hours), -`d` (days), `w` (weeks) and `y` (years)) or as a number of milliseconds. - - -### Purge jobs - -Purge jobs are the jobs that Synapse runs in the background to purge -expired events from the database. They are only run if support for -message retention policies is enabled in the server's configuration. If -no configuration for purge jobs is configured by the server admin, -Synapse will use a default configuration, which is described in the -[sample configuration file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518). - -Some server admins might want a finer control on when events are removed -depending on an event's room's policy. This can be done by setting the -`purge_jobs` sub-section in the `retention` section of the configuration -file. An example of such configuration could be: - -```yaml -purge_jobs: - - longest_max_lifetime: 3d - interval: 12h - - shortest_max_lifetime: 3d - longest_max_lifetime: 1w - interval: 1d - - shortest_max_lifetime: 1w - interval: 2d -``` - -In this example, we define three jobs: - -* one that runs twice a day (every 12 hours) and purges events in rooms - which policy's `max_lifetime` is lower or equal to 3 days. -* one that runs once a day and purges events in rooms which policy's - `max_lifetime` is between 3 days and a week. -* one that runs once every 2 days and purges events in rooms which - policy's `max_lifetime` is greater than a week. - -Note that this example is tailored to show different configurations and -features slightly more jobs than it's probably necessary (in practice, a -server admin would probably consider it better to replace the two last -jobs with one that runs once a day and handles rooms which which -policy's `max_lifetime` is greater than 3 days). - -Keep in mind, when configuring these jobs, that a purge job can become -quite heavy on the server if it targets many rooms, therefore prefer -having jobs with a low interval that target a limited set of rooms. Also -make sure to include a job with no minimum and one with no maximum to -make sure your configuration handles every policy. - -As previously mentioned in this documentation, while a purge job that -runs e.g. every day means that an expired event might stay in the -database for up to a day after its expiry, Synapse hides expired events -from clients as soon as they expire, so the event is not visible to -local users between its expiry date and the moment it gets purged from -the server's database. - - -### Lifetime limits - -Server admins can set limits on the values of `max_lifetime` to use when -purging old events in a room. These limits can be defined as such in the -`retention` section of the configuration file: - -```yaml -allowed_lifetime_min: 1d -allowed_lifetime_max: 1y -``` - -The limits are considered when running purge jobs. If necessary, the -effective value of `max_lifetime` will be brought between -`allowed_lifetime_min` and `allowed_lifetime_max` (inclusive). -This means that, if the value of `max_lifetime` defined in the room's state -is lower than `allowed_lifetime_min`, the value of `allowed_lifetime_min` -will be used instead. Likewise, if the value of `max_lifetime` is higher -than `allowed_lifetime_max`, the value of `allowed_lifetime_max` will be -used instead. - -In the example above, we ensure Synapse never deletes events that are less -than one day old, and that it always deletes events that are over a year -old. - -If a default policy is set, and its `max_lifetime` value is lower than -`allowed_lifetime_min` or higher than `allowed_lifetime_max`, the same -process applies. - -Both parameters are optional; if one is omitted Synapse won't use it to -adjust the effective value of `max_lifetime`. - -Like other settings in this section, these parameters can be expressed -either as a duration or as a number of milliseconds. - - -## Room configuration - -To configure a room's message retention policy, a room's admin or -moderator needs to send a state event in that room with the type -`m.room.retention` and the following content: - -```json -{ - "max_lifetime": ... -} -``` - -In this event's content, the `max_lifetime` parameter has the same -meaning as previously described, and needs to be expressed in -milliseconds. The event's content can also include a `min_lifetime` -parameter, which has the same meaning and limited support as previously -described. - -Note that over every server in the room, only the ones with support for -message retention policies will actually remove expired events. This -support is currently not enabled by default in Synapse. - - -## Note on reclaiming disk space - -While purge jobs actually delete data from the database, the disk space -used by the database might not decrease immediately on the database's -host. However, even though the database engine won't free up the disk -space, it will start writing new data into where the purged data was. - -If you want to reclaim the freed disk space anyway and return it to the -operating system, the server admin needs to run `VACUUM FULL;` (or -`VACUUM;` for SQLite databases) on Synapse's database (see the related -[PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-vacuum.html)). diff --git a/docs/metrics-howto.md b/docs/metrics-howto.md deleted file mode 100644 index 4a77d5604c..0000000000 --- a/docs/metrics-howto.md +++ /dev/null @@ -1,283 +0,0 @@ -# How to monitor Synapse metrics using Prometheus - -1. Install Prometheus: - - Follow instructions at - - -1. Enable Synapse metrics: - - There are two methods of enabling metrics in Synapse. - - The first serves the metrics as a part of the usual web server and - can be enabled by adding the \"metrics\" resource to the existing - listener as such: - - ```yaml - resources: - - names: - - client - - metrics - ``` - - This provides a simple way of adding metrics to your Synapse - installation, and serves under `/_synapse/metrics`. If you do not - wish your metrics be publicly exposed, you will need to either - filter it out at your load balancer, or use the second method. - - The second method runs the metrics server on a different port, in a - different thread to Synapse. This can make it more resilient to - heavy load meaning metrics cannot be retrieved, and can be exposed - to just internal networks easier. The served metrics are available - over HTTP only, and will be available at `/_synapse/metrics`. - - Add a new listener to homeserver.yaml: - - ```yaml - listeners: - - type: metrics - port: 9000 - bind_addresses: - - '0.0.0.0' - ``` - - For both options, you will need to ensure that `enable_metrics` is - set to `True`. - -1. Restart Synapse. - -1. Add a Prometheus target for Synapse. - - It needs to set the `metrics_path` to a non-default value (under - `scrape_configs`): - - ```yaml - - job_name: "synapse" - scrape_interval: 15s - metrics_path: "/_synapse/metrics" - static_configs: - - targets: ["my.server.here:port"] - ``` - - where `my.server.here` is the IP address of Synapse, and `port` is - the listener port configured with the `metrics` resource. - - If your prometheus is older than 1.5.2, you will need to replace - `static_configs` in the above with `target_groups`. - -1. Restart Prometheus. - -1. Consider using the [grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana/) - and required [recording rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus/) - -## Monitoring workers - -To monitor a Synapse installation using [workers](workers.md), -every worker needs to be monitored independently, in addition to -the main homeserver process. This is because workers don't send -their metrics to the main homeserver process, but expose them -directly (if they are configured to do so). - -To allow collecting metrics from a worker, you need to add a -`metrics` listener to its configuration, by adding the following -under `worker_listeners`: - -```yaml - - type: metrics - bind_address: '' - port: 9101 -``` - -The `bind_address` and `port` parameters should be set so that -the resulting listener can be reached by prometheus, and they -don't clash with an existing worker. -With this example, the worker's metrics would then be available -on `http://127.0.0.1:9101`. - -Example Prometheus target for Synapse with workers: - -```yaml - - job_name: "synapse" - scrape_interval: 15s - metrics_path: "/_synapse/metrics" - static_configs: - - targets: ["my.server.here:port"] - labels: - instance: "my.server" - job: "master" - index: 1 - - targets: ["my.workerserver.here:port"] - labels: - instance: "my.server" - job: "generic_worker" - index: 1 - - targets: ["my.workerserver.here:port"] - labels: - instance: "my.server" - job: "generic_worker" - index: 2 - - targets: ["my.workerserver.here:port"] - labels: - instance: "my.server" - job: "media_repository" - index: 1 -``` - -Labels (`instance`, `job`, `index`) can be defined as anything. -The labels are used to group graphs in grafana. - -## Renaming of metrics & deprecation of old names in 1.2 - -Synapse 1.2 updates the Prometheus metrics to match the naming -convention of the upstream `prometheus_client`. The old names are -considered deprecated and will be removed in a future version of -Synapse. - -| New Name | Old Name | -| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -| python_gc_objects_collected_total | python_gc_objects_collected | -| python_gc_objects_uncollectable_total | python_gc_objects_uncollectable | -| python_gc_collections_total | python_gc_collections | -| process_cpu_seconds_total | process_cpu_seconds | -| synapse_federation_client_sent_transactions_total | synapse_federation_client_sent_transactions | -| synapse_federation_client_events_processed_total | synapse_federation_client_events_processed | -| synapse_event_processing_loop_count_total | synapse_event_processing_loop_count | -| synapse_event_processing_loop_room_count_total | synapse_event_processing_loop_room_count | -| synapse_util_metrics_block_count_total | synapse_util_metrics_block_count | -| synapse_util_metrics_block_time_seconds_total | synapse_util_metrics_block_time_seconds | -| synapse_util_metrics_block_ru_utime_seconds_total | synapse_util_metrics_block_ru_utime_seconds | -| synapse_util_metrics_block_ru_stime_seconds_total | synapse_util_metrics_block_ru_stime_seconds | -| synapse_util_metrics_block_db_txn_count_total | synapse_util_metrics_block_db_txn_count | -| synapse_util_metrics_block_db_txn_duration_seconds_total | synapse_util_metrics_block_db_txn_duration_seconds | -| synapse_util_metrics_block_db_sched_duration_seconds_total | synapse_util_metrics_block_db_sched_duration_seconds | -| synapse_background_process_start_count_total | synapse_background_process_start_count | -| synapse_background_process_ru_utime_seconds_total | synapse_background_process_ru_utime_seconds | -| synapse_background_process_ru_stime_seconds_total | synapse_background_process_ru_stime_seconds | -| synapse_background_process_db_txn_count_total | synapse_background_process_db_txn_count | -| synapse_background_process_db_txn_duration_seconds_total | synapse_background_process_db_txn_duration_seconds | -| synapse_background_process_db_sched_duration_seconds_total | synapse_background_process_db_sched_duration_seconds | -| synapse_storage_events_persisted_events_total | synapse_storage_events_persisted_events | -| synapse_storage_events_persisted_events_sep_total | synapse_storage_events_persisted_events_sep | -| synapse_storage_events_state_delta_total | synapse_storage_events_state_delta | -| synapse_storage_events_state_delta_single_event_total | synapse_storage_events_state_delta_single_event | -| synapse_storage_events_state_delta_reuse_delta_total | synapse_storage_events_state_delta_reuse_delta | -| synapse_federation_server_received_pdus_total | synapse_federation_server_received_pdus | -| synapse_federation_server_received_edus_total | synapse_federation_server_received_edus | -| synapse_handler_presence_notified_presence_total | synapse_handler_presence_notified_presence | -| synapse_handler_presence_federation_presence_out_total | synapse_handler_presence_federation_presence_out | -| synapse_handler_presence_presence_updates_total | synapse_handler_presence_presence_updates | -| synapse_handler_presence_timers_fired_total | synapse_handler_presence_timers_fired | -| synapse_handler_presence_federation_presence_total | synapse_handler_presence_federation_presence | -| synapse_handler_presence_bump_active_time_total | synapse_handler_presence_bump_active_time | -| synapse_federation_client_sent_edus_total | synapse_federation_client_sent_edus | -| synapse_federation_client_sent_pdu_destinations_count_total | synapse_federation_client_sent_pdu_destinations:count | -| synapse_federation_client_sent_pdu_destinations_total | synapse_federation_client_sent_pdu_destinations:total | -| synapse_handlers_appservice_events_processed_total | synapse_handlers_appservice_events_processed | -| synapse_notifier_notified_events_total | synapse_notifier_notified_events | -| synapse_push_bulk_push_rule_evaluator_push_rules_invalidation_counter_total | synapse_push_bulk_push_rule_evaluator_push_rules_invalidation_counter | -| synapse_push_bulk_push_rule_evaluator_push_rules_state_size_counter_total | synapse_push_bulk_push_rule_evaluator_push_rules_state_size_counter | -| synapse_http_httppusher_http_pushes_processed_total | synapse_http_httppusher_http_pushes_processed | -| synapse_http_httppusher_http_pushes_failed_total | synapse_http_httppusher_http_pushes_failed | -| synapse_http_httppusher_badge_updates_processed_total | synapse_http_httppusher_badge_updates_processed | -| synapse_http_httppusher_badge_updates_failed_total | synapse_http_httppusher_badge_updates_failed | - -Removal of deprecated metrics & time based counters becoming histograms in 0.31.0 ---------------------------------------------------------------------------------- - -The duplicated metrics deprecated in Synapse 0.27.0 have been removed. - -All time duration-based metrics have been changed to be seconds. This -affects: - -| msec -> sec metrics | -| -------------------------------------- | -| python_gc_time | -| python_twisted_reactor_tick_time | -| synapse_storage_query_time | -| synapse_storage_schedule_time | -| synapse_storage_transaction_time | - -Several metrics have been changed to be histograms, which sort entries -into buckets and allow better analysis. The following metrics are now -histograms: - -| Altered metrics | -| ------------------------------------------------ | -| python_gc_time | -| python_twisted_reactor_pending_calls | -| python_twisted_reactor_tick_time | -| synapse_http_server_response_time_seconds | -| synapse_storage_query_time | -| synapse_storage_schedule_time | -| synapse_storage_transaction_time | - -Block and response metrics renamed for 0.27.0 ---------------------------------------------- - -Synapse 0.27.0 begins the process of rationalising the duplicate -`*:count` metrics reported for the resource tracking for code blocks and -HTTP requests. - -At the same time, the corresponding `*:total` metrics are being renamed, -as the `:total` suffix no longer makes sense in the absence of a -corresponding `:count` metric. - -To enable a graceful migration path, this release just adds new names -for the metrics being renamed. A future release will remove the old -ones. - -The following table shows the new metrics, and the old metrics which -they are replacing. - -| New name | Old name | -| ------------------------------------------------------------- | ---------------------------------------------------------- | -| synapse_util_metrics_block_count | synapse_util_metrics_block_timer:count | -| synapse_util_metrics_block_count | synapse_util_metrics_block_ru_utime:count | -| synapse_util_metrics_block_count | synapse_util_metrics_block_ru_stime:count | -| synapse_util_metrics_block_count | synapse_util_metrics_block_db_txn_count:count | -| synapse_util_metrics_block_count | synapse_util_metrics_block_db_txn_duration:count | -| synapse_util_metrics_block_time_seconds | synapse_util_metrics_block_timer:total | -| synapse_util_metrics_block_ru_utime_seconds | synapse_util_metrics_block_ru_utime:total | -| synapse_util_metrics_block_ru_stime_seconds | synapse_util_metrics_block_ru_stime:total | -| synapse_util_metrics_block_db_txn_count | synapse_util_metrics_block_db_txn_count:total | -| synapse_util_metrics_block_db_txn_duration_seconds | synapse_util_metrics_block_db_txn_duration:total | -| synapse_http_server_response_count | synapse_http_server_requests | -| synapse_http_server_response_count | synapse_http_server_response_time:count | -| synapse_http_server_response_count | synapse_http_server_response_ru_utime:count | -| synapse_http_server_response_count | synapse_http_server_response_ru_stime:count | -| synapse_http_server_response_count | synapse_http_server_response_db_txn_count:count | -| synapse_http_server_response_count | synapse_http_server_response_db_txn_duration:count | -| synapse_http_server_response_time_seconds | synapse_http_server_response_time:total | -| synapse_http_server_response_ru_utime_seconds | synapse_http_server_response_ru_utime:total | -| synapse_http_server_response_ru_stime_seconds | synapse_http_server_response_ru_stime:total | -| synapse_http_server_response_db_txn_count | synapse_http_server_response_db_txn_count:total | -| synapse_http_server_response_db_txn_duration_seconds | synapse_http_server_response_db_txn_duration:total | - -Standard Metric Names ---------------------- - -As of synapse version 0.18.2, the format of the process-wide metrics has -been changed to fit prometheus standard naming conventions. Additionally -the units have been changed to seconds, from miliseconds. - -| New name | Old name | -| ---------------------------------------- | --------------------------------- | -| process_cpu_user_seconds_total | process_resource_utime / 1000 | -| process_cpu_system_seconds_total | process_resource_stime / 1000 | -| process_open_fds (no \'type\' label) | process_fds | - -The python-specific counts of garbage collector performance have been -renamed. - -| New name | Old name | -| -------------------------------- | -------------------------- | -| python_gc_time | reactor_gc_time | -| python_gc_unreachable_total | reactor_gc_unreachable | -| python_gc_counts | reactor_gc_counts | - -The twisted-specific reactor metrics have been renamed. - -| New name | Old name | -| -------------------------------------- | ----------------------- | -| python_twisted_reactor_pending_calls | reactor_pending_calls | -| python_twisted_reactor_tick_time | reactor_tick_time | diff --git a/docs/openid.md b/docs/openid.md deleted file mode 100644 index c74e8bda60..0000000000 --- a/docs/openid.md +++ /dev/null @@ -1,572 +0,0 @@ -# Configuring Synapse to authenticate against an OpenID Connect provider - -Synapse can be configured to use an OpenID Connect Provider (OP) for -authentication, instead of its own local password database. - -Any OP should work with Synapse, as long as it supports the authorization code -flow. There are a few options for that: - - - start a local OP. Synapse has been tested with [Hydra][hydra] and - [Dex][dex-idp]. Note that for an OP to work, it should be served under a - secure (HTTPS) origin. A certificate signed with a self-signed, locally - trusted CA should work. In that case, start Synapse with a `SSL_CERT_FILE` - environment variable set to the path of the CA. - - - set up a SaaS OP, like [Google][google-idp], [Auth0][auth0] or - [Okta][okta]. Synapse has been tested with Auth0 and Google. - -It may also be possible to use other OAuth2 providers which provide the -[authorization code grant type](https://tools.ietf.org/html/rfc6749#section-4.1), -such as [Github][github-idp]. - -[google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect -[auth0]: https://auth0.com/ -[authentik]: https://goauthentik.io/ -[lemonldap]: https://lemonldap-ng.org/ -[okta]: https://www.okta.com/ -[dex-idp]: https://github.com/dexidp/dex -[keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols -[hydra]: https://www.ory.sh/docs/hydra/ -[github-idp]: https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps - -## Preparing Synapse - -The OpenID integration in Synapse uses the -[`authlib`](https://pypi.org/project/Authlib/) library, which must be installed -as follows: - - * The relevant libraries are included in the Docker images and Debian packages - provided by `matrix.org` so no further action is needed. - - * If you installed Synapse into a virtualenv, run `/path/to/env/bin/pip - install matrix-synapse[oidc]` to install the necessary dependencies. - - * For other installation mechanisms, see the documentation provided by the - maintainer. - -To enable the OpenID integration, you should then add a section to the `oidc_providers` -setting in your configuration file (or uncomment one of the existing examples). -See [sample_config.yaml](./sample_config.yaml) for some sample settings, as well as -the text below for example configurations for specific providers. - -## Sample configs - -Here are a few configs for providers that should work with Synapse. - -### Microsoft Azure Active Directory -Azure AD can act as an OpenID Connect Provider. Register a new application under -*App registrations* in the Azure AD management console. The RedirectURI for your -application should point to your matrix server: -`[synapse public baseurl]/_synapse/client/oidc/callback` - -Go to *Certificates & secrets* and register a new client secret. Make note of your -Directory (tenant) ID as it will be used in the Azure links. -Edit your Synapse config file and change the `oidc_config` section: - -```yaml -oidc_providers: - - idp_id: microsoft - idp_name: Microsoft - issuer: "https://login.microsoftonline.com//v2.0" - client_id: "" - client_secret: "" - scopes: ["openid", "profile"] - authorization_endpoint: "https://login.microsoftonline.com//oauth2/v2.0/authorize" - token_endpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" - userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo" - - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username.split('@')[0] }}" - display_name_template: "{{ user.name }}" -``` - -### Dex - -[Dex][dex-idp] is a simple, open-source, certified OpenID Connect Provider. -Although it is designed to help building a full-blown provider with an -external database, it can be configured with static passwords in a config file. - -Follow the [Getting Started guide](https://dexidp.io/docs/getting-started/) -to install Dex. - -Edit `examples/config-dev.yaml` config file from the Dex repo to add a client: - -```yaml -staticClients: -- id: synapse - secret: secret - redirectURIs: - - '[synapse public baseurl]/_synapse/client/oidc/callback' - name: 'Synapse' -``` - -Run with `dex serve examples/config-dev.yaml`. - -Synapse config: - -```yaml -oidc_providers: - - idp_id: dex - idp_name: "My Dex server" - skip_verification: true # This is needed as Dex is served on an insecure endpoint - issuer: "http://127.0.0.1:5556/dex" - client_id: "synapse" - client_secret: "secret" - scopes: ["openid", "profile"] - user_mapping_provider: - config: - localpart_template: "{{ user.name }}" - display_name_template: "{{ user.name|capitalize }}" -``` -### Keycloak - -[Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat. - -Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to install Keycloak and set up a realm. - -1. Click `Clients` in the sidebar and click `Create` - -2. Fill in the fields as below: - -| Field | Value | -|-----------|-----------| -| Client ID | `synapse` | -| Client Protocol | `openid-connect` | - -3. Click `Save` -4. Fill in the fields as below: - -| Field | Value | -|-----------|-----------| -| Client ID | `synapse` | -| Enabled | `On` | -| Client Protocol | `openid-connect` | -| Access Type | `confidential` | -| Valid Redirect URIs | `[synapse public baseurl]/_synapse/client/oidc/callback` | - -5. Click `Save` -6. On the Credentials tab, update the fields: - -| Field | Value | -|-------|-------| -| Client Authenticator | `Client ID and Secret` | - -7. Click `Regenerate Secret` -8. Copy Secret - -```yaml -oidc_providers: - - idp_id: keycloak - idp_name: "My KeyCloak server" - issuer: "https://127.0.0.1:8443/auth/realms/{realm_name}" - client_id: "synapse" - client_secret: "copy secret generated from above" - scopes: ["openid", "profile"] - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}" - display_name_template: "{{ user.name }}" -``` -### Auth0 - -[Auth0][auth0] is a hosted SaaS IdP solution. - -1. Create a regular web application for Synapse -2. Set the Allowed Callback URLs to `[synapse public baseurl]/_synapse/client/oidc/callback` -3. Add a rule to add the `preferred_username` claim. -
- Code sample - - ```js - function addPersistenceAttribute(user, context, callback) { - user.user_metadata = user.user_metadata || {}; - user.user_metadata.preferred_username = user.user_metadata.preferred_username || user.user_id; - context.idToken.preferred_username = user.user_metadata.preferred_username; - - auth0.users.updateUserMetadata(user.user_id, user.user_metadata) - .then(function(){ - callback(null, user, context); - }) - .catch(function(err){ - callback(err); - }); - } - ``` -
- -Synapse config: - -```yaml -oidc_providers: - - idp_id: auth0 - idp_name: Auth0 - issuer: "https://your-tier.eu.auth0.com/" # TO BE FILLED - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - scopes: ["openid", "profile"] - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}" - display_name_template: "{{ user.name }}" -``` - -### Authentik - -[Authentik][authentik] is an open-source IdP solution. - -1. Create a provider in Authentik, with type OAuth2/OpenID. -2. The parameters are: -- Client Type: Confidential -- JWT Algorithm: RS256 -- Scopes: OpenID, Email and Profile -- RSA Key: Select any available key -- Redirect URIs: `[synapse public baseurl]/_synapse/client/oidc/callback` -3. Create an application for synapse in Authentik and link it to the provider. -4. Note the slug of your application, Client ID and Client Secret. - -Synapse config: -```yaml -oidc_providers: - - idp_id: authentik - idp_name: authentik - discover: true - issuer: "https://your.authentik.example.org/application/o/your-app-slug/" # TO BE FILLED: domain and slug - client_id: "your client id" # TO BE FILLED - client_secret: "your client secret" # TO BE FILLED - scopes: - - "openid" - - "profile" - - "email" - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}}" - display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize. -``` - -### LemonLDAP - -[LemonLDAP::NG][lemonldap] is an open-source IdP solution. - -1. Create an OpenID Connect Relying Parties in LemonLDAP::NG -2. The parameters are: -- Client ID under the basic menu of the new Relying Parties (`Options > Basic > - Client ID`) -- Client secret (`Options > Basic > Client secret`) -- JWT Algorithm: RS256 within the security menu of the new Relying Parties - (`Options > Security > ID Token signature algorithm` and `Options > Security > - Access Token signature algorithm`) -- Scopes: OpenID, Email and Profile -- Allowed redirection addresses for login (`Options > Basic > Allowed - redirection addresses for login` ) : - `[synapse public baseurl]/_synapse/client/oidc/callback` - -Synapse config: -```yaml -oidc_providers: - - idp_id: lemonldap - idp_name: lemonldap - discover: true - issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain - client_id: "your client id" # TO BE FILLED - client_secret: "your client secret" # TO BE FILLED - scopes: - - "openid" - - "profile" - - "email" - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}}" - # TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter. - display_name_template: "{{ user.preferred_username|capitalize }}" -``` - -### GitHub - -[GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but -just a regular OAuth2 provider. - -The [`/user` API endpoint](https://developer.github.com/v3/users/#get-the-authenticated-user) -can be used to retrieve information on the authenticated user. As the Synapse -login mechanism needs an attribute to uniquely identify users, and that endpoint -does not return a `sub` property, an alternative `subject_claim` has to be set. - -1. Create a new OAuth application: https://github.com/settings/applications/new. -2. Set the callback URL to `[synapse public baseurl]/_synapse/client/oidc/callback`. - -Synapse config: - -```yaml -oidc_providers: - - idp_id: github - idp_name: Github - idp_brand: "github" # optional: styling hint for clients - discover: false - issuer: "https://github.com/" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - authorization_endpoint: "https://github.com/login/oauth/authorize" - token_endpoint: "https://github.com/login/oauth/access_token" - userinfo_endpoint: "https://api.github.com/user" - scopes: ["read:user"] - user_mapping_provider: - config: - subject_claim: "id" - localpart_template: "{{ user.login }}" - display_name_template: "{{ user.name }}" -``` - -### Google - -[Google][google-idp] is an OpenID certified authentication and authorisation provider. - -1. Set up a project in the Google API Console (see - https://developers.google.com/identity/protocols/oauth2/openid-connect#appsetup). -2. Add an "OAuth Client ID" for a Web Application under "Credentials". -3. Copy the Client ID and Client Secret, and add the following to your synapse config: - ```yaml - oidc_providers: - - idp_id: google - idp_name: Google - idp_brand: "google" # optional: styling hint for clients - issuer: "https://accounts.google.com/" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - scopes: ["openid", "profile"] - user_mapping_provider: - config: - localpart_template: "{{ user.given_name|lower }}" - display_name_template: "{{ user.name }}" - ``` -4. Back in the Google console, add this Authorized redirect URI: `[synapse - public baseurl]/_synapse/client/oidc/callback`. - -### Twitch - -1. Setup a developer account on [Twitch](https://dev.twitch.tv/) -2. Obtain the OAuth 2.0 credentials by [creating an app](https://dev.twitch.tv/console/apps/) -3. Add this OAuth Redirect URL: `[synapse public baseurl]/_synapse/client/oidc/callback` - -Synapse config: - -```yaml -oidc_providers: - - idp_id: twitch - idp_name: Twitch - issuer: "https://id.twitch.tv/oauth2/" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - client_auth_method: "client_secret_post" - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}" - display_name_template: "{{ user.name }}" -``` - -### GitLab - -1. Create a [new application](https://gitlab.com/profile/applications). -2. Add the `read_user` and `openid` scopes. -3. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback` - -Synapse config: - -```yaml -oidc_providers: - - idp_id: gitlab - idp_name: Gitlab - idp_brand: "gitlab" # optional: styling hint for clients - issuer: "https://gitlab.com/" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - client_auth_method: "client_secret_post" - scopes: ["openid", "read_user"] - user_profile_method: "userinfo_endpoint" - user_mapping_provider: - config: - localpart_template: '{{ user.nickname }}' - display_name_template: '{{ user.name }}' -``` - -### Facebook - -Like Github, Facebook provide a custom OAuth2 API rather than an OIDC-compliant -one so requires a little more configuration. - -0. You will need a Facebook developer account. You can register for one - [here](https://developers.facebook.com/async/registration/). -1. On the [apps](https://developers.facebook.com/apps/) page of the developer - console, "Create App", and choose "Build Connected Experiences". -2. Once the app is created, add "Facebook Login" and choose "Web". You don't - need to go through the whole form here. -3. In the left-hand menu, open "Products"/"Facebook Login"/"Settings". - * Add `[synapse public baseurl]/_synapse/client/oidc/callback` as an OAuth Redirect - URL. -4. In the left-hand menu, open "Settings/Basic". Here you can copy the "App ID" - and "App Secret" for use below. - -Synapse config: - -```yaml - - idp_id: facebook - idp_name: Facebook - idp_brand: "facebook" # optional: styling hint for clients - discover: false - issuer: "https://facebook.com" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - scopes: ["openid", "email"] - authorization_endpoint: https://facebook.com/dialog/oauth - token_endpoint: https://graph.facebook.com/v9.0/oauth/access_token - user_profile_method: "userinfo_endpoint" - userinfo_endpoint: "https://graph.facebook.com/v9.0/me?fields=id,name,email,picture" - user_mapping_provider: - config: - subject_claim: "id" - display_name_template: "{{ user.name }}" -``` - -Relevant documents: - * https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow - * Using Facebook's Graph API: https://developers.facebook.com/docs/graph-api/using-graph-api/ - * Reference to the User endpoint: https://developers.facebook.com/docs/graph-api/reference/user - -### Gitea - -Gitea is, like Github, not an OpenID provider, but just an OAuth2 provider. - -The [`/user` API endpoint](https://try.gitea.io/api/swagger#/user/userGetCurrent) -can be used to retrieve information on the authenticated user. As the Synapse -login mechanism needs an attribute to uniquely identify users, and that endpoint -does not return a `sub` property, an alternative `subject_claim` has to be set. - -1. Create a new application. -2. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback` - -Synapse config: - -```yaml -oidc_providers: - - idp_id: gitea - idp_name: Gitea - discover: false - issuer: "https://your-gitea.com/" - client_id: "your-client-id" # TO BE FILLED - client_secret: "your-client-secret" # TO BE FILLED - client_auth_method: client_secret_post - scopes: [] # Gitea doesn't support Scopes - authorization_endpoint: "https://your-gitea.com/login/oauth/authorize" - token_endpoint: "https://your-gitea.com/login/oauth/access_token" - userinfo_endpoint: "https://your-gitea.com/api/v1/user" - user_mapping_provider: - config: - subject_claim: "id" - localpart_template: "{{ user.login }}" - display_name_template: "{{ user.full_name }}" -``` - -### XWiki - -Install [OpenID Connect Provider](https://extensions.xwiki.org/xwiki/bin/view/Extension/OpenID%20Connect/OpenID%20Connect%20Provider/) extension in your [XWiki](https://www.xwiki.org) instance. - -Synapse config: - -```yaml -oidc_providers: - - idp_id: xwiki - idp_name: "XWiki" - issuer: "https://myxwikihost/xwiki/oidc/" - client_id: "your-client-id" # TO BE FILLED - client_auth_method: none - scopes: ["openid", "profile"] - user_profile_method: "userinfo_endpoint" - user_mapping_provider: - config: - localpart_template: "{{ user.preferred_username }}" - display_name_template: "{{ user.name }}" -``` - -### Apple - -Configuring "Sign in with Apple" (SiWA) requires an Apple Developer account. - -You will need to create a new "Services ID" for SiWA, and create and download a -private key with "SiWA" enabled. - -As well as the private key file, you will need: - * Client ID: the "identifier" you gave the "Services ID" - * Team ID: a 10-character ID associated with your developer account. - * Key ID: the 10-character identifier for the key. - -https://help.apple.com/developer-account/?lang=en#/dev77c875b7e has more -documentation on setting up SiWA. - -The synapse config will look like this: - -```yaml - - idp_id: apple - idp_name: Apple - issuer: "https://appleid.apple.com" - client_id: "your-client-id" # Set to the "identifier" for your "ServicesID" - client_auth_method: "client_secret_post" - client_secret_jwt_key: - key_file: "/path/to/AuthKey_KEYIDCODE.p8" # point to your key file - jwt_header: - alg: ES256 - kid: "KEYIDCODE" # Set to the 10-char Key ID - jwt_payload: - iss: TEAMIDCODE # Set to the 10-char Team ID - scopes: ["name", "email", "openid"] - authorization_endpoint: https://appleid.apple.com/auth/authorize?response_mode=form_post - user_mapping_provider: - config: - email_template: "{{ user.email }}" -``` - -## Django OAuth Toolkit - -[django-oauth-toolkit](https://github.com/jazzband/django-oauth-toolkit) is a -Django application providing out of the box all the endpoints, data and logic -needed to add OAuth2 capabilities to your Django projects. It supports -[OpenID Connect too](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html). - -Configuration on Django's side: - -1. Add an application: https://example.com/admin/oauth2_provider/application/add/ and choose parameters like this: -* `Redirect uris`: https://synapse.example.com/_synapse/client/oidc/callback -* `Client type`: `Confidential` -* `Authorization grant type`: `Authorization code` -* `Algorithm`: `HMAC with SHA-2 256` -2. You can [customize the claims](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html#customizing-the-oidc-responses) Django gives to synapse (optional): -
- Code sample - - ```python - class CustomOAuth2Validator(OAuth2Validator): - - def get_additional_claims(self, request): - return { - "sub": request.user.email, - "email": request.user.email, - "first_name": request.user.first_name, - "last_name": request.user.last_name, - } - ``` -
-Your synapse config is then: - -```yaml -oidc_providers: - - idp_id: django_example - idp_name: "Django Example" - issuer: "https://example.com/o/" - client_id: "your-client-id" # CHANGE ME - client_secret: "your-client-secret" # CHANGE ME - scopes: ["openid"] - user_profile_method: "userinfo_endpoint" # needed because oauth-toolkit does not include user information in the authorization response - user_mapping_provider: - config: - localpart_template: "{{ user.email.split('@')[0] }}" - display_name_template: "{{ user.first_name }} {{ user.last_name }}" - email_template: "{{ user.email }}" -``` diff --git a/docs/opentracing.md b/docs/opentracing.md deleted file mode 100644 index f91362f112..0000000000 --- a/docs/opentracing.md +++ /dev/null @@ -1,93 +0,0 @@ -# OpenTracing - -## Background - -OpenTracing is a semi-standard being adopted by a number of distributed -tracing platforms. It is a common api for facilitating vendor-agnostic -tracing instrumentation. That is, we can use the OpenTracing api and -select one of a number of tracer implementations to do the heavy lifting -in the background. Our current selected implementation is Jaeger. - -OpenTracing is a tool which gives an insight into the causal -relationship of work done in and between servers. The servers each track -events and report them to a centralised server - in Synapse's case: -Jaeger. The basic unit used to represent events is the span. The span -roughly represents a single piece of work that was done and the time at -which it occurred. A span can have child spans, meaning that the work of -the child had to be completed for the parent span to complete, or it can -have follow-on spans which represent work that is undertaken as a result -of the parent but is not depended on by the parent to in order to -finish. - -Since this is undertaken in a distributed environment a request to -another server, such as an RPC or a simple GET, can be considered a span -(a unit or work) for the local server. This causal link is what -OpenTracing aims to capture and visualise. In order to do this metadata -about the local server's span, i.e the 'span context', needs to be -included with the request to the remote. - -It is up to the remote server to decide what it does with the spans it -creates. This is called the sampling policy and it can be configured -through Jaeger's settings. - -For OpenTracing concepts see -. - -For more information about Jaeger's implementation see - - -## Setting up OpenTracing - -To receive OpenTracing spans, start up a Jaeger server. This can be done -using docker like so: - -```sh -docker run -d --name jaeger \ - -p 6831:6831/udp \ - -p 6832:6832/udp \ - -p 5778:5778 \ - -p 16686:16686 \ - -p 14268:14268 \ - jaegertracing/all-in-one:1 -``` - -Latest documentation is probably at -https://www.jaegertracing.io/docs/latest/getting-started. - -## Enable OpenTracing in Synapse - -OpenTracing is not enabled by default. It must be enabled in the -homeserver config by uncommenting the config options under `opentracing` -as shown in the [sample config](./sample_config.yaml). For example: - -```yaml -opentracing: - enabled: true - homeserver_whitelist: - - "mytrustedhomeserver.org" - - "*.myotherhomeservers.com" -``` - -## Homeserver whitelisting - -The homeserver whitelist is configured using regular expressions. A list -of regular expressions can be given and their union will be compared -when propagating any spans contexts to another homeserver. - -Though it's mostly safe to send and receive span contexts to and from -untrusted users since span contexts are usually opaque ids it can lead -to two problems, namely: - -- If the span context is marked as sampled by the sending homeserver - the receiver will sample it. Therefore two homeservers with wildly - different sampling policies could incur higher sampling counts than - intended. -- Sending servers can attach arbitrary data to spans, known as - 'baggage'. For safety this has been disabled in Synapse but that - doesn't prevent another server sending you baggage which will be - logged to OpenTracing's logs. - -## Configuring Jaeger - -Sampling strategies can be set as in this document: -. diff --git a/docs/other/dependency_deprecation_policy.md b/docs/other/dependency_deprecation_policy.md new file mode 100644 index 0000000000..06ea340559 --- /dev/null +++ b/docs/other/dependency_deprecation_policy.md @@ -0,0 +1,33 @@ +Deprecation Policy for Platform Dependencies +============================================ + +Synapse has a number of platform dependencies, including Python and PostgreSQL. +This document outlines the policy towards which versions we support, and when we +drop support for versions in the future. + + +Policy +------ + +Synapse follows the upstream support life cycles for Python and PostgreSQL, +i.e. when a version reaches End of Life Synapse will withdraw support for that +version in future releases. + +Details on the upstream support life cycles for Python and PostgreSQL are +documented at https://endoflife.date/python and +https://endoflife.date/postgresql. + + +Context +------- + +It is important for system admins to have a clear understanding of the platform +requirements of Synapse and its deprecation policies so that they can +effectively plan upgrading their infrastructure ahead of time. This is +especially important in contexts where upgrading the infrastructure requires +auditing and approval from a security team, or where otherwise upgrading is a +long process. + +By following the upstream support life cycles Synapse can ensure that its +dependencies continue to get security patches, while not requiring system admins +to constantly update their platform dependencies to the latest versions. diff --git a/docs/password_auth_providers.md b/docs/password_auth_providers.md deleted file mode 100644 index dc0dfffa21..0000000000 --- a/docs/password_auth_providers.md +++ /dev/null @@ -1,129 +0,0 @@ -

-This page of the Synapse documentation is now deprecated. For up to date -documentation on setting up or writing a password auth provider module, please see -this page. -

- -# Password auth provider modules - -Password auth providers offer a way for server administrators to -integrate their Synapse installation with an existing authentication -system. - -A password auth provider is a Python class which is dynamically loaded -into Synapse, and provides a number of methods by which it can integrate -with the authentication system. - -This document serves as a reference for those looking to implement their -own password auth providers. Additionally, here is a list of known -password auth provider module implementations: - -* [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/) -* [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth) -* [matrix-synapse-rest-password-provider](https://github.com/ma1uta/matrix-synapse-rest-password-provider) - -## Required methods - -Password auth provider classes must provide the following methods: - -* `parse_config(config)` - This method is passed the `config` object for this module from the - homeserver configuration file. - - It should perform any appropriate sanity checks on the provided - configuration, and return an object which is then passed into - `__init__`. - - This method should have the `@staticmethod` decoration. - -* `__init__(self, config, account_handler)` - - The constructor is passed the config object returned by - `parse_config`, and a `synapse.module_api.ModuleApi` object which - allows the password provider to check if accounts exist and/or create - new ones. - -## Optional methods - -Password auth provider classes may optionally provide the following methods: - -* `get_db_schema_files(self)` - - This method, if implemented, should return an Iterable of - `(name, stream)` pairs of database schema files. Each file is applied - in turn at initialisation, and a record is then made in the database - so that it is not re-applied on the next start. - -* `get_supported_login_types(self)` - - This method, if implemented, should return a `dict` mapping from a - login type identifier (such as `m.login.password`) to an iterable - giving the fields which must be provided by the user in the submission - to [the `/login` API](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login). - These fields are passed in the `login_dict` dictionary to `check_auth`. - - For example, if a password auth provider wants to implement a custom - login type of `com.example.custom_login`, where the client is expected - to pass the fields `secret1` and `secret2`, the provider should - implement this method and return the following dict: - - ```python - {"com.example.custom_login": ("secret1", "secret2")} - ``` - -* `check_auth(self, username, login_type, login_dict)` - - This method does the real work. If implemented, it - will be called for each login attempt where the login type matches one - of the keys returned by `get_supported_login_types`. - - It is passed the (possibly unqualified) `user` field provided by the client, - the login type, and a dictionary of login secrets passed by the - client. - - The method should return an `Awaitable` object, which resolves - to the canonical `@localpart:domain` user ID if authentication is - successful, and `None` if not. - - Alternatively, the `Awaitable` can resolve to a `(str, func)` tuple, in - which case the second field is a callback which will be called with - the result from the `/login` call (including `access_token`, - `device_id`, etc.) - -* `check_3pid_auth(self, medium, address, password)` - - This method, if implemented, is called when a user attempts to - register or log in with a third party identifier, such as email. It is - passed the medium (ex. "email"), an address (ex. - "") and the user's password. - - The method should return an `Awaitable` object, which resolves - to a `str` containing the user's (canonical) User id if - authentication was successful, and `None` if not. - - As with `check_auth`, the `Awaitable` may alternatively resolve to a - `(user_id, callback)` tuple. - -* `check_password(self, user_id, password)` - - This method provides a simpler interface than - `get_supported_login_types` and `check_auth` for password auth - providers that just want to provide a mechanism for validating - `m.login.password` logins. - - If implemented, it will be called to check logins with an - `m.login.password` login type. It is passed a qualified - `@localpart:domain` user id, and the password provided by the user. - - The method should return an `Awaitable` object, which resolves - to `True` if authentication is successful, and `False` if not. - -* `on_logged_out(self, user_id, device_id, access_token)` - - This method, if implemented, is called when a user logs out. It is - passed the qualified user ID, the ID of the deactivated device (if - any: access tokens are occasionally created without an associated - device ID), and the (now deactivated) access token. - - It may return an `Awaitable` object; the logout request will - wait for the `Awaitable` to complete, but the result is ignored. diff --git a/docs/postgres.md b/docs/postgres.md deleted file mode 100644 index 083b0aaff0..0000000000 --- a/docs/postgres.md +++ /dev/null @@ -1,255 +0,0 @@ -# Using Postgres - -Synapse supports PostgreSQL versions 9.6 or later. - -## Install postgres client libraries - -Synapse will require the python postgres client library in order to -connect to a postgres database. - -- If you are using the [matrix.org debian/ubuntu - packages](setup/installation.md#matrixorg-packages), the necessary python - library will already be installed, but you will need to ensure the - low-level postgres library is installed, which you can do with - `apt install libpq5`. -- For other pre-built packages, please consult the documentation from - the relevant package. -- If you installed synapse [in a - virtualenv](setup/installation.md#installing-from-source), you can install - the library with: - - ~/synapse/env/bin/pip install "matrix-synapse[postgres]" - - (substituting the path to your virtualenv for `~/synapse/env`, if - you used a different path). You will require the postgres - development files. These are in the `libpq-dev` package on - Debian-derived distributions. - -## Set up database - -Assuming your PostgreSQL database user is called `postgres`, first authenticate as the database user with: - -```sh -su - postgres -# Or, if your system uses sudo to get administrative rights -sudo -u postgres bash -``` - -Then, create a postgres user and a database with: - -```sh -# this will prompt for a password for the new user -createuser --pwprompt synapse_user - -createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse -``` - -The above will create a user called `synapse_user`, and a database called -`synapse`. - -Note that the PostgreSQL database *must* have the correct encoding set -(as shown above), otherwise it will not be able to store UTF8 strings. - -You may need to enable password authentication so `synapse_user` can -connect to the database. See -. - -## Synapse config - -When you are ready to start using PostgreSQL, edit the `database` -section in your config file to match the following lines: - -```yaml -database: - name: psycopg2 - args: - user: - password: - database: - host: - cp_min: 5 - cp_max: 10 -``` - -All key, values in `args` are passed to the `psycopg2.connect(..)` -function, except keys beginning with `cp_`, which are consumed by the -twisted adbapi connection pool. See the [libpq -documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) -for a list of options which can be passed. - -You should consider tuning the `args.keepalives_*` options if there is any danger of -the connection between your homeserver and database dropping, otherwise Synapse -may block for an extended period while it waits for a response from the -database server. Example values might be: - -```yaml -database: - args: - # ... as above - - # seconds of inactivity after which TCP should send a keepalive message to the server - keepalives_idle: 10 - - # the number of seconds after which a TCP keepalive message that is not - # acknowledged by the server should be retransmitted - keepalives_interval: 10 - - # the number of TCP keepalives that can be lost before the client's connection - # to the server is considered dead - keepalives_count: 3 -``` - -## Tuning Postgres - -The default settings should be fine for most deployments. For larger -scale deployments tuning some of the settings is recommended, details of -which can be found at -. - -In particular, we've found tuning the following values helpful for -performance: - -- `shared_buffers` -- `effective_cache_size` -- `work_mem` -- `maintenance_work_mem` -- `autovacuum_work_mem` - -Note that the appropriate values for those fields depend on the amount -of free memory the database host has available. - - -## Porting from SQLite - -### Overview - -The script `synapse_port_db` allows porting an existing synapse server -backed by SQLite to using PostgreSQL. This is done in as a two phase -process: - -1. Copy the existing SQLite database to a separate location and run - the port script against that offline database. -2. Shut down the server. Rerun the port script to port any data that - has come in since taking the first snapshot. Restart server against - the PostgreSQL database. - -The port script is designed to be run repeatedly against newer snapshots -of the SQLite database file. This makes it safe to repeat step 1 if -there was a delay between taking the previous snapshot and being ready -to do step 2. - -It is safe to at any time kill the port script and restart it. - -Note that the database may take up significantly more (25% - 100% more) -space on disk after porting to Postgres. - -### Using the port script - -Firstly, shut down the currently running synapse server and copy its -database file (typically `homeserver.db`) to another location. Once the -copy is complete, restart synapse. For instance: - -```sh -./synctl stop -cp homeserver.db homeserver.db.snapshot -./synctl start -``` - -Copy the old config file into a new config file: - -```sh -cp homeserver.yaml homeserver-postgres.yaml -``` - -Edit the database section as described in the section *Synapse config* -above and with the SQLite snapshot located at `homeserver.db.snapshot` -simply run: - -```sh -synapse_port_db --sqlite-database homeserver.db.snapshot \ - --postgres-config homeserver-postgres.yaml -``` - -The flag `--curses` displays a coloured curses progress UI. - -If the script took a long time to complete, or time has otherwise passed -since the original snapshot was taken, repeat the previous steps with a -newer snapshot. - -To complete the conversion shut down the synapse server and run the port -script one last time, e.g. if the SQLite database is at `homeserver.db` -run: - -```sh -synapse_port_db --sqlite-database homeserver.db \ - --postgres-config homeserver-postgres.yaml -``` - -Once that has completed, change the synapse config to point at the -PostgreSQL database configuration file `homeserver-postgres.yaml`: - -```sh -./synctl stop -mv homeserver.yaml homeserver-old-sqlite.yaml -mv homeserver-postgres.yaml homeserver.yaml -./synctl start -``` - -Synapse should now be running against PostgreSQL. - - -## Troubleshooting - -### Alternative auth methods - -If you get an error along the lines of `FATAL: Ident authentication failed for -user "synapse_user"`, you may need to use an authentication method other than -`ident`: - -* If the `synapse_user` user has a password, add the password to the `database:` - section of `homeserver.yaml`. Then add the following to `pg_hba.conf`: - - ``` - host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that - ``` - -* If the `synapse_user` user does not have a password, then a password doesn't - have to be added to `homeserver.yaml`. But the following does need to be added - to `pg_hba.conf`: - - ``` - host synapse synapse_user ::1/128 trust - ``` - -Note that line order matters in `pg_hba.conf`, so make sure that if you do add a -new line, it is inserted before: - -``` -host all all ::1/128 ident -``` - -### Fixing incorrect `COLLATE` or `CTYPE` - -Synapse will refuse to set up a new database if it has the wrong values of -`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using -different locales can cause issues if the locale library is updated from -underneath the database, or if a different version of the locale is used on any -replicas. - -The safest way to fix the issue is to dump the database and recreate it with -the correct locale parameter (as shown above). It is also possible to change the -parameters on a live database and run a `REINDEX` on the entire database, -however extreme care must be taken to avoid database corruption. - -Note that the above may fail with an error about duplicate rows if corruption -has already occurred, and such duplicate rows will need to be manually removed. - -### Fixing inconsistent sequences error - -Synapse uses Postgres sequences to generate IDs for various tables. A sequence -and associated table can get out of sync if, for example, Synapse has been -downgraded and then upgraded again. - -To fix the issue shut down Synapse (including any and all workers) and run the -SQL command included in the error message. Once done Synapse should start -successfully. diff --git a/docs/replication.md b/docs/replication.md deleted file mode 100644 index e82df0de8a..0000000000 --- a/docs/replication.md +++ /dev/null @@ -1,37 +0,0 @@ -# Replication Architecture - -## Motivation - -We'd like to be able to split some of the work that synapse does into -multiple python processes. In theory multiple synapse processes could -share a single postgresql database and we\'d scale up by running more -synapse processes. However much of synapse assumes that only one process -is interacting with the database, both for assigning unique identifiers -when inserting into tables, notifying components about new updates, and -for invalidating its caches. - -So running multiple copies of the current code isn't an option. One way -to run multiple processes would be to have a single writer process and -multiple reader processes connected to the same database. In order to do -this we'd need a way for the reader process to invalidate its in-memory -caches when an update happens on the writer. One way to do this is for -the writer to present an append-only log of updates which the readers -can consume to invalidate their caches and to push updates to listening -clients or pushers. - -Synapse already stores much of its data as an append-only log so that it -can correctly respond to `/sync` requests so the amount of code changes -needed to expose the append-only log to the readers should be fairly -minimal. - -## Architecture - -### The Replication Protocol - -See [the TCP replication documentation](tcp_replication.md). - -### The Slaved DataStore - -There are read-only version of the synapse storage layer in -`synapse/replication/slave/storage` that use the response of the -replication API to invalidate their caches. diff --git a/docs/reverse_proxy.md b/docs/reverse_proxy.md deleted file mode 100644 index f3b3aea732..0000000000 --- a/docs/reverse_proxy.md +++ /dev/null @@ -1,267 +0,0 @@ -# Using a reverse proxy with Synapse - -It is recommended to put a reverse proxy such as -[nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), -[Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html), -[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy), -[HAProxy](https://www.haproxy.org/) or -[relayd](https://man.openbsd.org/relayd.8) in front of Synapse. One advantage -of doing so is that it means that you can expose the default https port -(443) to Matrix clients without needing to run Synapse with root -privileges. - -You should configure your reverse proxy to forward requests to `/_matrix` or -`/_synapse/client` to Synapse, and have it set the `X-Forwarded-For` and -`X-Forwarded-Proto` request headers. - -You should remember that Matrix clients and other Matrix servers do not -necessarily need to connect to your server via the same server name or -port. Indeed, clients will use port 443 by default, whereas servers default to -port 8448. Where these are different, we refer to the 'client port' and the -'federation port'. See [the Matrix -specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names) -for more details of the algorithm used for federation connections, and -[Delegation](delegate.md) for instructions on setting up delegation. - -**NOTE**: Your reverse proxy must not `canonicalise` or `normalise` -the requested URI in any way (for example, by decoding `%xx` escapes). -Beware that Apache *will* canonicalise URIs unless you specify -`nocanon`. - -Let's assume that we expect clients to connect to our server at -`https://matrix.example.com`, and other servers to connect at -`https://example.com:8448`. The following sections detail the configuration of -the reverse proxy and the homeserver. - - -## Homeserver Configuration - -The HTTP configuration will need to be updated for Synapse to correctly record -client IP addresses and generate redirect URLs while behind a reverse proxy. - -In `homeserver.yaml` set `x_forwarded: true` in the port 8008 section and -consider setting `bind_addresses: ['127.0.0.1']` so that the server only -listens to traffic on localhost. (Do not change `bind_addresses` to `127.0.0.1` -when using a containerized Synapse, as that will prevent it from responding -to proxied traffic.) - - -## Reverse-proxy configuration examples - -**NOTE**: You only need one of these. - -### nginx - -```nginx -server { - listen 443 ssl http2; - listen [::]:443 ssl http2; - - # For the federation port - listen 8448 ssl http2 default_server; - listen [::]:8448 ssl http2 default_server; - - server_name matrix.example.com; - - location ~* ^(\/_matrix|\/_synapse\/client) { - # note: do not add a path (even a single /) after the port in `proxy_pass`, - # otherwise nginx will canonicalise the URI and cause signature verification - # errors. - proxy_pass http://localhost:8008; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header Host $host; - - # Nginx by default only allows file uploads up to 1M in size - # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml - client_max_body_size 50M; - } -} -``` - -### Caddy v1 - -``` -matrix.example.com { - proxy /_matrix http://localhost:8008 { - transparent - } - - proxy /_synapse/client http://localhost:8008 { - transparent - } -} - -example.com:8448 { - proxy / http://localhost:8008 { - transparent - } -} -``` - -### Caddy v2 - -``` -matrix.example.com { - reverse_proxy /_matrix/* http://localhost:8008 - reverse_proxy /_synapse/client/* http://localhost:8008 -} - -example.com:8448 { - reverse_proxy http://localhost:8008 -} -``` -[Delegation](delegate.md) example: -``` -(matrix-well-known-header) { - # Headers - header Access-Control-Allow-Origin "*" - header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" - header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization" - header Content-Type "application/json" -} - -example.com { - handle /.well-known/matrix/server { - import matrix-well-known-header - respond `{"m.server":"matrix.example.com:443"}` - } - - handle /.well-known/matrix/client { - import matrix-well-known-header - respond `{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}` - } -} - -matrix.example.com { - reverse_proxy /_matrix/* http://localhost:8008 - reverse_proxy /_synapse/client/* http://localhost:8008 -} -``` - -### Apache - -```apache - - SSLEngine on - ServerName matrix.example.com - - RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME} - AllowEncodedSlashes NoDecode - ProxyPreserveHost on - ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon - ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix - ProxyPass /_synapse/client http://127.0.0.1:8008/_synapse/client nocanon - ProxyPassReverse /_synapse/client http://127.0.0.1:8008/_synapse/client - - - - SSLEngine on - ServerName example.com - - RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME} - AllowEncodedSlashes NoDecode - ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon - ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix - -``` - -**NOTE**: ensure the `nocanon` options are included. - -**NOTE 2**: It appears that Synapse is currently incompatible with the ModSecurity module for Apache (`mod_security2`). If you need it enabled for other services on your web server, you can disable it for Synapse's two VirtualHosts by including the following lines before each of the two `` above: - -```apache - - SecRuleEngine off - -``` - -**NOTE 3**: Missing `ProxyPreserveHost on` can lead to a redirect loop. - -### HAProxy - -``` -frontend https - bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1 - http-request set-header X-Forwarded-Proto https if { ssl_fc } - http-request set-header X-Forwarded-Proto http if !{ ssl_fc } - http-request set-header X-Forwarded-For %[src] - - # Matrix client traffic - acl matrix-host hdr(host) -i matrix.example.com matrix.example.com:443 - acl matrix-path path_beg /_matrix - acl matrix-path path_beg /_synapse/client - - use_backend matrix if matrix-host matrix-path - -frontend matrix-federation - bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1 - http-request set-header X-Forwarded-Proto https if { ssl_fc } - http-request set-header X-Forwarded-Proto http if !{ ssl_fc } - http-request set-header X-Forwarded-For %[src] - - default_backend matrix - -backend matrix - server matrix 127.0.0.1:8008 -``` - -### Relayd - -``` -table { 127.0.0.1 } -table { 127.0.0.1 } - -http protocol "https" { - tls { no tlsv1.0, ciphers "HIGH" } - tls keypair "example.com" - match header set "X-Forwarded-For" value "$REMOTE_ADDR" - match header set "X-Forwarded-Proto" value "https" - - # set CORS header for .well-known/matrix/server, .well-known/matrix/client - # httpd does not support setting headers, so do it here - match request path "/.well-known/matrix/*" tag "matrix-cors" - match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*" - - pass quick path "/_matrix/*" forward to - pass quick path "/_synapse/client/*" forward to - - # pass on non-matrix traffic to webserver - pass forward to -} - -relay "https_traffic" { - listen on egress port 443 tls - protocol "https" - forward to port 8008 check tcp - forward to port 8080 check tcp -} - -http protocol "matrix" { - tls { no tlsv1.0, ciphers "HIGH" } - tls keypair "example.com" - block - pass quick path "/_matrix/*" forward to - pass quick path "/_synapse/client/*" forward to -} - -relay "matrix_federation" { - listen on egress port 8448 tls - protocol "matrix" - forward to port 8008 check tcp -} -``` - - -## Health check endpoint - -Synapse exposes a health check endpoint for use by reverse proxies. -Each configured HTTP listener has a `/health` endpoint which always returns -200 OK (and doesn't get logged). - -## Synapse administration endpoints - -Endpoints for administering your Synapse instance are placed under -`/_synapse/admin`. These require authentication through an access token of an -admin user. However as access to these endpoints grants the caller a lot of power, -we do not recommend exposing them to the public internet without good reason. diff --git a/docs/room_and_user_statistics.md b/docs/room_and_user_statistics.md deleted file mode 100644 index cc38c890bb..0000000000 --- a/docs/room_and_user_statistics.md +++ /dev/null @@ -1,22 +0,0 @@ -Room and User Statistics -======================== - -Synapse maintains room and user statistics in various tables. These can be used -for administrative purposes but are also used when generating the public room -directory. - - -# Synapse Developer Documentation - -## High-Level Concepts - -### Definitions - -* **subject**: Something we are tracking stats about – currently a room or user. -* **current row**: An entry for a subject in the appropriate current statistics - table. Each subject can have only one. - -### Overview - -Stats correspond to the present values. Current rows contain the most up-to-date -statistics for a room. Each subject can only have one entry. diff --git a/docs/server_notices.md b/docs/server_notices.md deleted file mode 100644 index 339d10a0ab..0000000000 --- a/docs/server_notices.md +++ /dev/null @@ -1,61 +0,0 @@ -# Server Notices - -'Server Notices' are a new feature introduced in Synapse 0.30. They provide a -channel whereby server administrators can send messages to users on the server. - -They are used as part of communication of the server polices (see -[Consent Tracking](consent_tracking.md)), however the intention is that -they may also find a use for features such as "Message of the day". - -This is a feature specific to Synapse, but it uses standard Matrix -communication mechanisms, so should work with any Matrix client. - -## User experience - -When the user is first sent a server notice, they will get an invitation to a -room (typically called 'Server Notices', though this is configurable in -`homeserver.yaml`). They will be **unable to reject** this invitation - -attempts to do so will receive an error. - -Once they accept the invitation, they will see the notice message in the room -history; it will appear to have come from the 'server notices user' (see -below). - -The user is prevented from sending any messages in this room by the power -levels. - -Having joined the room, the user can leave the room if they want. Subsequent -server notices will then cause a new room to be created. - -## Synapse configuration - -Server notices come from a specific user id on the server. Server -administrators are free to choose the user id - something like `server` is -suggested, meaning the notices will come from -`@server:`. Once the Server Notices user is configured, that -user id becomes a special, privileged user, so administrators should ensure -that **it is not already allocated**. - -In order to support server notices, it is necessary to add some configuration -to the `homeserver.yaml` file. In particular, you should add a `server_notices` -section, which should look like this: - -```yaml -server_notices: - system_mxid_localpart: server - system_mxid_display_name: "Server Notices" - system_mxid_avatar_url: "mxc://server.com/oumMVlgDnLYFaPVkExemNVVZ" - room_name: "Server Notices" -``` - -The only compulsory setting is `system_mxid_localpart`, which defines the user -id of the Server Notices user, as above. `room_name` defines the name of the -room which will be created. - -`system_mxid_display_name` and `system_mxid_avatar_url` can be used to set the -displayname and avatar of the Server Notices user. - -## Sending notices - -To send server notices to users you can use the -[admin_api](admin_api/server_notices.md). diff --git a/docs/setup/delegation.md b/docs/setup/delegation.md new file mode 100644 index 0000000000..ee9cbb3b1c --- /dev/null +++ b/docs/setup/delegation.md @@ -0,0 +1,102 @@ +# Delegation of incoming federation traffic + +In the following documentation, we use the term `server_name` to refer to that setting +in your homeserver configuration file. It appears at the ends of user ids, and tells +other homeservers where they can find your server. + +By default, other homeservers will expect to be able to reach yours via +your `server_name`, on port 8448. For example, if you set your `server_name` +to `example.com` (so that your user names look like `@user:example.com`), +other servers will try to connect to yours at `https://example.com:8448/`. + +Delegation is a Matrix feature allowing a homeserver admin to retain a +`server_name` of `example.com` so that user IDs, room aliases, etc continue +to look like `*:example.com`, whilst having federation traffic routed +to a different server and/or port (e.g. `synapse.example.com:443`). + +## .well-known delegation + +To use this method, you need to be able to configure the server at +`https://` to serve a file at +`https:///.well-known/matrix/server`. There are two ways to do this, shown below. + +Note that the `.well-known` file is hosted on the default port for `https` (port 443). + +### External server + +For maximum flexibility, you need to configure an external server such as nginx, Apache +or HAProxy to serve the `https:///.well-known/matrix/server` file. Setting +up such a server is out of the scope of this documentation, but note that it is often +possible to configure your [reverse proxy](reverse_proxy.md) for this. + +The URL `https:///.well-known/matrix/server` should be configured +return a JSON structure containing the key `m.server` like this: + +```json +{ + "m.server": "[:]" +} +``` + +In our example (where we want federation traffic to be routed to +`https://synapse.example.com`, on port 443), this would mean that +`https://example.com/.well-known/matrix/server` should return: + +```json +{ + "m.server": "synapse.example.com:443" +} +``` + +Note, specifying a port is optional. If no port is specified, then it defaults +to 8448. + +### Serving a `.well-known/matrix/server` file with Synapse + +If you are able to set up your domain so that `https://` is routed to +Synapse (i.e., the only change needed is to direct federation traffic to port 443 +instead of port 8448), then it is possible to configure Synapse to serve a suitable +`.well-known/matrix/server` file. To do so, add the following to your `homeserver.yaml` +file: + +```yaml +serve_server_wellknown: true +``` + +**Note**: this *only* works if `https://` is routed to Synapse, so is +generally not suitable if Synapse is hosted at a subdomain such as +`https://synapse.example.com`. + +## SRV DNS record delegation + +It is also possible to do delegation using a SRV DNS record. However, that is generally +not recommended, as it can be difficult to configure the TLS certificates correctly in +this case, and it offers little advantage over `.well-known` delegation. + +However, if you really need it, you can find some documentation on what such a +record should look like and how Synapse will use it in [the Matrix +specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names). + +## Delegation FAQ + +### When do I need delegation? + +If your homeserver's APIs are accessible on the default federation port (8448) +and the domain your `server_name` points to, you do not need any delegation. + +For instance, if you registered `example.com` and pointed its DNS A record at a +fresh server, you could install Synapse on that host, giving it a `server_name` +of `example.com`, and once a reverse proxy has been set up to proxy all requests +sent to the port `8448` and serve TLS certificates for `example.com`, you +wouldn't need any delegation set up. + +**However**, if your homeserver's APIs aren't accessible on port 8448 and on the +domain `server_name` points to, you will need to let other servers know how to +find it using delegation. + +### Should I use a reverse proxy for federation traffic? + +Generally, using a reverse proxy for both the federation and client traffic is a good +idea, since it saves handling TLS traffic in Synapse. See +[the reverse proxy documentation](reverse_proxy.md) for information on setting up a +reverse proxy. diff --git a/docs/setup/postgres.md b/docs/setup/postgres.md new file mode 100644 index 0000000000..083b0aaff0 --- /dev/null +++ b/docs/setup/postgres.md @@ -0,0 +1,255 @@ +# Using Postgres + +Synapse supports PostgreSQL versions 9.6 or later. + +## Install postgres client libraries + +Synapse will require the python postgres client library in order to +connect to a postgres database. + +- If you are using the [matrix.org debian/ubuntu + packages](setup/installation.md#matrixorg-packages), the necessary python + library will already be installed, but you will need to ensure the + low-level postgres library is installed, which you can do with + `apt install libpq5`. +- For other pre-built packages, please consult the documentation from + the relevant package. +- If you installed synapse [in a + virtualenv](setup/installation.md#installing-from-source), you can install + the library with: + + ~/synapse/env/bin/pip install "matrix-synapse[postgres]" + + (substituting the path to your virtualenv for `~/synapse/env`, if + you used a different path). You will require the postgres + development files. These are in the `libpq-dev` package on + Debian-derived distributions. + +## Set up database + +Assuming your PostgreSQL database user is called `postgres`, first authenticate as the database user with: + +```sh +su - postgres +# Or, if your system uses sudo to get administrative rights +sudo -u postgres bash +``` + +Then, create a postgres user and a database with: + +```sh +# this will prompt for a password for the new user +createuser --pwprompt synapse_user + +createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse +``` + +The above will create a user called `synapse_user`, and a database called +`synapse`. + +Note that the PostgreSQL database *must* have the correct encoding set +(as shown above), otherwise it will not be able to store UTF8 strings. + +You may need to enable password authentication so `synapse_user` can +connect to the database. See +. + +## Synapse config + +When you are ready to start using PostgreSQL, edit the `database` +section in your config file to match the following lines: + +```yaml +database: + name: psycopg2 + args: + user: + password: + database: + host: + cp_min: 5 + cp_max: 10 +``` + +All key, values in `args` are passed to the `psycopg2.connect(..)` +function, except keys beginning with `cp_`, which are consumed by the +twisted adbapi connection pool. See the [libpq +documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) +for a list of options which can be passed. + +You should consider tuning the `args.keepalives_*` options if there is any danger of +the connection between your homeserver and database dropping, otherwise Synapse +may block for an extended period while it waits for a response from the +database server. Example values might be: + +```yaml +database: + args: + # ... as above + + # seconds of inactivity after which TCP should send a keepalive message to the server + keepalives_idle: 10 + + # the number of seconds after which a TCP keepalive message that is not + # acknowledged by the server should be retransmitted + keepalives_interval: 10 + + # the number of TCP keepalives that can be lost before the client's connection + # to the server is considered dead + keepalives_count: 3 +``` + +## Tuning Postgres + +The default settings should be fine for most deployments. For larger +scale deployments tuning some of the settings is recommended, details of +which can be found at +. + +In particular, we've found tuning the following values helpful for +performance: + +- `shared_buffers` +- `effective_cache_size` +- `work_mem` +- `maintenance_work_mem` +- `autovacuum_work_mem` + +Note that the appropriate values for those fields depend on the amount +of free memory the database host has available. + + +## Porting from SQLite + +### Overview + +The script `synapse_port_db` allows porting an existing synapse server +backed by SQLite to using PostgreSQL. This is done in as a two phase +process: + +1. Copy the existing SQLite database to a separate location and run + the port script against that offline database. +2. Shut down the server. Rerun the port script to port any data that + has come in since taking the first snapshot. Restart server against + the PostgreSQL database. + +The port script is designed to be run repeatedly against newer snapshots +of the SQLite database file. This makes it safe to repeat step 1 if +there was a delay between taking the previous snapshot and being ready +to do step 2. + +It is safe to at any time kill the port script and restart it. + +Note that the database may take up significantly more (25% - 100% more) +space on disk after porting to Postgres. + +### Using the port script + +Firstly, shut down the currently running synapse server and copy its +database file (typically `homeserver.db`) to another location. Once the +copy is complete, restart synapse. For instance: + +```sh +./synctl stop +cp homeserver.db homeserver.db.snapshot +./synctl start +``` + +Copy the old config file into a new config file: + +```sh +cp homeserver.yaml homeserver-postgres.yaml +``` + +Edit the database section as described in the section *Synapse config* +above and with the SQLite snapshot located at `homeserver.db.snapshot` +simply run: + +```sh +synapse_port_db --sqlite-database homeserver.db.snapshot \ + --postgres-config homeserver-postgres.yaml +``` + +The flag `--curses` displays a coloured curses progress UI. + +If the script took a long time to complete, or time has otherwise passed +since the original snapshot was taken, repeat the previous steps with a +newer snapshot. + +To complete the conversion shut down the synapse server and run the port +script one last time, e.g. if the SQLite database is at `homeserver.db` +run: + +```sh +synapse_port_db --sqlite-database homeserver.db \ + --postgres-config homeserver-postgres.yaml +``` + +Once that has completed, change the synapse config to point at the +PostgreSQL database configuration file `homeserver-postgres.yaml`: + +```sh +./synctl stop +mv homeserver.yaml homeserver-old-sqlite.yaml +mv homeserver-postgres.yaml homeserver.yaml +./synctl start +``` + +Synapse should now be running against PostgreSQL. + + +## Troubleshooting + +### Alternative auth methods + +If you get an error along the lines of `FATAL: Ident authentication failed for +user "synapse_user"`, you may need to use an authentication method other than +`ident`: + +* If the `synapse_user` user has a password, add the password to the `database:` + section of `homeserver.yaml`. Then add the following to `pg_hba.conf`: + + ``` + host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that + ``` + +* If the `synapse_user` user does not have a password, then a password doesn't + have to be added to `homeserver.yaml`. But the following does need to be added + to `pg_hba.conf`: + + ``` + host synapse synapse_user ::1/128 trust + ``` + +Note that line order matters in `pg_hba.conf`, so make sure that if you do add a +new line, it is inserted before: + +``` +host all all ::1/128 ident +``` + +### Fixing incorrect `COLLATE` or `CTYPE` + +Synapse will refuse to set up a new database if it has the wrong values of +`COLLATE` and `CTYPE` set, and will log warnings on existing databases. Using +different locales can cause issues if the locale library is updated from +underneath the database, or if a different version of the locale is used on any +replicas. + +The safest way to fix the issue is to dump the database and recreate it with +the correct locale parameter (as shown above). It is also possible to change the +parameters on a live database and run a `REINDEX` on the entire database, +however extreme care must be taken to avoid database corruption. + +Note that the above may fail with an error about duplicate rows if corruption +has already occurred, and such duplicate rows will need to be manually removed. + +### Fixing inconsistent sequences error + +Synapse uses Postgres sequences to generate IDs for various tables. A sequence +and associated table can get out of sync if, for example, Synapse has been +downgraded and then upgraded again. + +To fix the issue shut down Synapse (including any and all workers) and run the +SQL command included in the error message. Once done Synapse should start +successfully. diff --git a/docs/setup/reverse_proxy.md b/docs/setup/reverse_proxy.md new file mode 100644 index 0000000000..f3b3aea732 --- /dev/null +++ b/docs/setup/reverse_proxy.md @@ -0,0 +1,267 @@ +# Using a reverse proxy with Synapse + +It is recommended to put a reverse proxy such as +[nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html), +[Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html), +[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy), +[HAProxy](https://www.haproxy.org/) or +[relayd](https://man.openbsd.org/relayd.8) in front of Synapse. One advantage +of doing so is that it means that you can expose the default https port +(443) to Matrix clients without needing to run Synapse with root +privileges. + +You should configure your reverse proxy to forward requests to `/_matrix` or +`/_synapse/client` to Synapse, and have it set the `X-Forwarded-For` and +`X-Forwarded-Proto` request headers. + +You should remember that Matrix clients and other Matrix servers do not +necessarily need to connect to your server via the same server name or +port. Indeed, clients will use port 443 by default, whereas servers default to +port 8448. Where these are different, we refer to the 'client port' and the +'federation port'. See [the Matrix +specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names) +for more details of the algorithm used for federation connections, and +[Delegation](delegate.md) for instructions on setting up delegation. + +**NOTE**: Your reverse proxy must not `canonicalise` or `normalise` +the requested URI in any way (for example, by decoding `%xx` escapes). +Beware that Apache *will* canonicalise URIs unless you specify +`nocanon`. + +Let's assume that we expect clients to connect to our server at +`https://matrix.example.com`, and other servers to connect at +`https://example.com:8448`. The following sections detail the configuration of +the reverse proxy and the homeserver. + + +## Homeserver Configuration + +The HTTP configuration will need to be updated for Synapse to correctly record +client IP addresses and generate redirect URLs while behind a reverse proxy. + +In `homeserver.yaml` set `x_forwarded: true` in the port 8008 section and +consider setting `bind_addresses: ['127.0.0.1']` so that the server only +listens to traffic on localhost. (Do not change `bind_addresses` to `127.0.0.1` +when using a containerized Synapse, as that will prevent it from responding +to proxied traffic.) + + +## Reverse-proxy configuration examples + +**NOTE**: You only need one of these. + +### nginx + +```nginx +server { + listen 443 ssl http2; + listen [::]:443 ssl http2; + + # For the federation port + listen 8448 ssl http2 default_server; + listen [::]:8448 ssl http2 default_server; + + server_name matrix.example.com; + + location ~* ^(\/_matrix|\/_synapse\/client) { + # note: do not add a path (even a single /) after the port in `proxy_pass`, + # otherwise nginx will canonicalise the URI and cause signature verification + # errors. + proxy_pass http://localhost:8008; + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Host $host; + + # Nginx by default only allows file uploads up to 1M in size + # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml + client_max_body_size 50M; + } +} +``` + +### Caddy v1 + +``` +matrix.example.com { + proxy /_matrix http://localhost:8008 { + transparent + } + + proxy /_synapse/client http://localhost:8008 { + transparent + } +} + +example.com:8448 { + proxy / http://localhost:8008 { + transparent + } +} +``` + +### Caddy v2 + +``` +matrix.example.com { + reverse_proxy /_matrix/* http://localhost:8008 + reverse_proxy /_synapse/client/* http://localhost:8008 +} + +example.com:8448 { + reverse_proxy http://localhost:8008 +} +``` +[Delegation](delegate.md) example: +``` +(matrix-well-known-header) { + # Headers + header Access-Control-Allow-Origin "*" + header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS" + header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization" + header Content-Type "application/json" +} + +example.com { + handle /.well-known/matrix/server { + import matrix-well-known-header + respond `{"m.server":"matrix.example.com:443"}` + } + + handle /.well-known/matrix/client { + import matrix-well-known-header + respond `{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}` + } +} + +matrix.example.com { + reverse_proxy /_matrix/* http://localhost:8008 + reverse_proxy /_synapse/client/* http://localhost:8008 +} +``` + +### Apache + +```apache + + SSLEngine on + ServerName matrix.example.com + + RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME} + AllowEncodedSlashes NoDecode + ProxyPreserveHost on + ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon + ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix + ProxyPass /_synapse/client http://127.0.0.1:8008/_synapse/client nocanon + ProxyPassReverse /_synapse/client http://127.0.0.1:8008/_synapse/client + + + + SSLEngine on + ServerName example.com + + RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME} + AllowEncodedSlashes NoDecode + ProxyPass /_matrix http://127.0.0.1:8008/_matrix nocanon + ProxyPassReverse /_matrix http://127.0.0.1:8008/_matrix + +``` + +**NOTE**: ensure the `nocanon` options are included. + +**NOTE 2**: It appears that Synapse is currently incompatible with the ModSecurity module for Apache (`mod_security2`). If you need it enabled for other services on your web server, you can disable it for Synapse's two VirtualHosts by including the following lines before each of the two `` above: + +```apache + + SecRuleEngine off + +``` + +**NOTE 3**: Missing `ProxyPreserveHost on` can lead to a redirect loop. + +### HAProxy + +``` +frontend https + bind :::443 v4v6 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1 + http-request set-header X-Forwarded-Proto https if { ssl_fc } + http-request set-header X-Forwarded-Proto http if !{ ssl_fc } + http-request set-header X-Forwarded-For %[src] + + # Matrix client traffic + acl matrix-host hdr(host) -i matrix.example.com matrix.example.com:443 + acl matrix-path path_beg /_matrix + acl matrix-path path_beg /_synapse/client + + use_backend matrix if matrix-host matrix-path + +frontend matrix-federation + bind :::8448 v4v6 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1 + http-request set-header X-Forwarded-Proto https if { ssl_fc } + http-request set-header X-Forwarded-Proto http if !{ ssl_fc } + http-request set-header X-Forwarded-For %[src] + + default_backend matrix + +backend matrix + server matrix 127.0.0.1:8008 +``` + +### Relayd + +``` +table { 127.0.0.1 } +table { 127.0.0.1 } + +http protocol "https" { + tls { no tlsv1.0, ciphers "HIGH" } + tls keypair "example.com" + match header set "X-Forwarded-For" value "$REMOTE_ADDR" + match header set "X-Forwarded-Proto" value "https" + + # set CORS header for .well-known/matrix/server, .well-known/matrix/client + # httpd does not support setting headers, so do it here + match request path "/.well-known/matrix/*" tag "matrix-cors" + match response tagged "matrix-cors" header set "Access-Control-Allow-Origin" value "*" + + pass quick path "/_matrix/*" forward to + pass quick path "/_synapse/client/*" forward to + + # pass on non-matrix traffic to webserver + pass forward to +} + +relay "https_traffic" { + listen on egress port 443 tls + protocol "https" + forward to port 8008 check tcp + forward to port 8080 check tcp +} + +http protocol "matrix" { + tls { no tlsv1.0, ciphers "HIGH" } + tls keypair "example.com" + block + pass quick path "/_matrix/*" forward to + pass quick path "/_synapse/client/*" forward to +} + +relay "matrix_federation" { + listen on egress port 8448 tls + protocol "matrix" + forward to port 8008 check tcp +} +``` + + +## Health check endpoint + +Synapse exposes a health check endpoint for use by reverse proxies. +Each configured HTTP listener has a `/health` endpoint which always returns +200 OK (and doesn't get logged). + +## Synapse administration endpoints + +Endpoints for administering your Synapse instance are placed under +`/_synapse/admin`. These require authentication through an access token of an +admin user. However as access to these endpoints grants the caller a lot of power, +we do not recommend exposing them to the public internet without good reason. diff --git a/docs/setup/turn-howto.md b/docs/setup/turn-howto.md new file mode 100644 index 0000000000..e6812de69e --- /dev/null +++ b/docs/setup/turn-howto.md @@ -0,0 +1,296 @@ +# Overview + +This document explains how to enable VoIP relaying on your homeserver with +TURN. + +The synapse Matrix homeserver supports integration with TURN server via the +[TURN server REST API](). This +allows the homeserver to generate credentials that are valid for use on the +TURN server through the use of a secret shared between the homeserver and the +TURN server. + +The following sections describe how to install [coturn]() (which implements the TURN REST API) and integrate it with synapse. + +## Requirements + +For TURN relaying with `coturn` to work, it must be hosted on a server/endpoint with a public IP. + +Hosting TURN behind a NAT (even with appropriate port forwarding) is known to cause issues +and to often not work. + +## `coturn` setup + +### Initial installation + +The TURN daemon `coturn` is available from a variety of sources such as native package managers, or installation from source. + +#### Debian installation + +Just install the debian package: + +```sh +apt install coturn +``` + +This will install and start a systemd service called `coturn`. + +#### Source installation + +1. Download the [latest release](https://github.com/coturn/coturn/releases/latest) from github. Unpack it and `cd` into the directory. + +1. Configure it: + + ```sh + ./configure + ``` + + You may need to install `libevent2`: if so, you should do so in + the way recommended by your operating system. You can ignore + warnings about lack of database support: a database is unnecessary + for this purpose. + +1. Build and install it: + + ```sh + make + make install + ``` + +### Configuration + +1. Create or edit the config file in `/etc/turnserver.conf`. The relevant + lines, with example values, are: + + ``` + use-auth-secret + static-auth-secret=[your secret key here] + realm=turn.myserver.org + ``` + + See `turnserver.conf` for explanations of the options. One way to generate + the `static-auth-secret` is with `pwgen`: + + ```sh + pwgen -s 64 1 + ``` + + A `realm` must be specified, but its value is somewhat arbitrary. (It is + sent to clients as part of the authentication flow.) It is conventional to + set it to be your server name. + +1. You will most likely want to configure coturn to write logs somewhere. The + easiest way is normally to send them to the syslog: + + ```sh + syslog + ``` + + (in which case, the logs will be available via `journalctl -u coturn` on a + systemd system). Alternatively, coturn can be configured to write to a + logfile - check the example config file supplied with coturn. + +1. Consider your security settings. TURN lets users request a relay which will + connect to arbitrary IP addresses and ports. The following configuration is + suggested as a minimum starting point: + + ``` + # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay. + no-tcp-relay + + # don't let the relay ever try to connect to private IP address ranges within your network (if any) + # given the turn server is likely behind your firewall, remember to include any privileged public IPs too. + denied-peer-ip=10.0.0.0-10.255.255.255 + denied-peer-ip=192.168.0.0-192.168.255.255 + denied-peer-ip=172.16.0.0-172.31.255.255 + + # special case the turn server itself so that client->TURN->TURN->client flows work + allowed-peer-ip=10.0.0.1 + + # consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS. + user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user. + total-quota=1200 + ``` + +1. Also consider supporting TLS/DTLS. To do this, add the following settings + to `turnserver.conf`: + + ``` + # TLS certificates, including intermediate certs. + # For Let's Encrypt certificates, use `fullchain.pem` here. + cert=/path/to/fullchain.pem + + # TLS private key file + pkey=/path/to/privkey.pem + ``` + + In this case, replace the `turn:` schemes in the `turn_uri` settings below + with `turns:`. + + We recommend that you only try to set up TLS/DTLS once you have set up a + basic installation and got it working. + +1. Ensure your firewall allows traffic into the TURN server on the ports + you've configured it to listen on (By default: 3478 and 5349 for TURN + traffic (remember to allow both TCP and UDP traffic), and ports 49152-65535 + for the UDP relay.) + +1. We do not recommend running a TURN server behind NAT, and are not aware of + anyone doing so successfully. + + If you want to try it anyway, you will at least need to tell coturn its + external IP address: + + ``` + external-ip=192.88.99.1 + ``` + + ... and your NAT gateway must forward all of the relayed ports directly + (eg, port 56789 on the external IP must be always be forwarded to port + 56789 on the internal IP). + + If you get this working, let us know! + +1. (Re)start the turn server: + + * If you used the Debian package (or have set up a systemd unit yourself): + ```sh + systemctl restart coturn + ``` + + * If you installed from source: + + ```sh + bin/turnserver -o + ``` + +## Synapse setup + +Your homeserver configuration file needs the following extra keys: + +1. "`turn_uris`": This needs to be a yaml list of public-facing URIs + for your TURN server to be given out to your clients. Add separate + entries for each transport your TURN server supports. +2. "`turn_shared_secret`": This is the secret shared between your + homeserver and your TURN server, so you should set it to the same + string you used in turnserver.conf. +3. "`turn_user_lifetime`": This is the amount of time credentials + generated by your homeserver are valid for (in milliseconds). + Shorter times offer less potential for abuse at the expense of + increased traffic between web clients and your homeserver to + refresh credentials. The TURN REST API specification recommends + one day (86400000). +4. "`turn_allow_guests`": Whether to allow guest users to use the + TURN server. This is enabled by default, as otherwise VoIP will + not work reliably for guests. However, it does introduce a + security risk as it lets guests connect to arbitrary endpoints + without having gone through a CAPTCHA or similar to register a + real account. + +As an example, here is the relevant section of the config file for `matrix.org`. The +`turn_uris` are appropriate for TURN servers listening on the default ports, with no TLS. + + turn_uris: [ "turn:turn.matrix.org?transport=udp", "turn:turn.matrix.org?transport=tcp" ] + turn_shared_secret: "n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons" + turn_user_lifetime: 86400000 + turn_allow_guests: True + +After updating the homeserver configuration, you must restart synapse: + + * If you use synctl: + ```sh + cd /where/you/run/synapse + ./synctl restart + ``` + * If you use systemd: + ```sh + systemctl restart matrix-synapse.service + ``` +... and then reload any clients (or wait an hour for them to refresh their +settings). + +## Troubleshooting + +The normal symptoms of a misconfigured TURN server are that calls between +devices on different networks ring, but get stuck at "call +connecting". Unfortunately, troubleshooting this can be tricky. + +Here are a few things to try: + + * Check that your TURN server is not behind NAT. As above, we're not aware of + anyone who has successfully set this up. + + * Check that you have opened your firewall to allow TCP and UDP traffic to the + TURN ports (normally 3478 and 5349). + + * Check that you have opened your firewall to allow UDP traffic to the UDP + relay ports (49152-65535 by default). + + * Some WebRTC implementations (notably, that of Google Chrome) appear to get + confused by TURN servers which are reachable over IPv6 (this appears to be + an unexpected side-effect of its handling of multiple IP addresses as + defined by + [`draft-ietf-rtcweb-ip-handling`](https://tools.ietf.org/html/draft-ietf-rtcweb-ip-handling-12)). + + Try removing any AAAA records for your TURN server, so that it is only + reachable over IPv4. + + * Enable more verbose logging in coturn via the `verbose` setting: + + ``` + verbose + ``` + + ... and then see if there are any clues in its logs. + + * If you are using a browser-based client under Chrome, check + `chrome://webrtc-internals/` for insights into the internals of the + negotiation. On Firefox, check the "Connection Log" on `about:webrtc`. + + (Understanding the output is beyond the scope of this document!) + + * You can test your Matrix homeserver TURN setup with https://test.voip.librepush.net/. + Note that this test is not fully reliable yet, so don't be discouraged if + the test fails. + [Here](https://github.com/matrix-org/voip-tester) is the github repo of the + source of the tester, where you can file bug reports. + + * There is a WebRTC test tool at + https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/. To + use it, you will need a username/password for your TURN server. You can + either: + + * look for the `GET /_matrix/client/r0/voip/turnServer` request made by a + matrix client to your homeserver in your browser's network inspector. In + the response you should see `username` and `password`. Or: + + * Use the following shell commands: + + ```sh + secret=staticAuthSecretHere + + u=$((`date +%s` + 3600)):test + p=$(echo -n $u | openssl dgst -hmac $secret -sha1 -binary | base64) + echo -e "username: $u\npassword: $p" + ``` + + Or: + + * Temporarily configure coturn to accept a static username/password. To do + this, comment out `use-auth-secret` and `static-auth-secret` and add the + following: + + ``` + lt-cred-mech + user=username:password + ``` + + **Note**: these settings will not take effect unless `use-auth-secret` + and `static-auth-secret` are disabled. + + Restart coturn after changing the configuration file. + + Remember to restore the original settings to go back to testing with + Matrix clients! + + If the TURN server is working correctly, you should see at least one `relay` + entry in the results. diff --git a/docs/sso_mapping_providers.md b/docs/sso_mapping_providers.md deleted file mode 100644 index 7a407012e0..0000000000 --- a/docs/sso_mapping_providers.md +++ /dev/null @@ -1,197 +0,0 @@ -# SSO Mapping Providers - -A mapping provider is a Python class (loaded via a Python module) that -works out how to map attributes of a SSO response to Matrix-specific -user attributes. Details such as user ID localpart, displayname, and even avatar -URLs are all things that can be mapped from talking to a SSO service. - -As an example, a SSO service may return the email address -"john.smith@example.com" for a user, whereas Synapse will need to figure out how -to turn that into a displayname when creating a Matrix user for this individual. -It may choose `John Smith`, or `Smith, John [Example.com]` or any number of -variations. As each Synapse configuration may want something different, this is -where SAML mapping providers come into play. - -SSO mapping providers are currently supported for OpenID and SAML SSO -configurations. Please see the details below for how to implement your own. - -It is up to the mapping provider whether the user should be assigned a predefined -Matrix ID based on the SSO attributes, or if the user should be allowed to -choose their own username. - -In the first case - where users are automatically allocated a Matrix ID - it is -the responsibility of the mapping provider to normalise the SSO attributes and -map them to a valid Matrix ID. The [specification for Matrix -IDs](https://matrix.org/docs/spec/appendices#user-identifiers) has some -information about what is considered valid. - -If the mapping provider does not assign a Matrix ID, then Synapse will -automatically serve an HTML page allowing the user to pick their own username. - -External mapping providers are provided to Synapse in the form of an external -Python module. You can retrieve this module from [PyPI](https://pypi.org) or elsewhere, -but it must be importable via Synapse (e.g. it must be in the same virtualenv -as Synapse). The Synapse config is then modified to point to the mapping provider -(and optionally provide additional configuration for it). - -## OpenID Mapping Providers - -The OpenID mapping provider can be customized by editing the -`oidc_config.user_mapping_provider.module` config option. - -`oidc_config.user_mapping_provider.config` allows you to provide custom -configuration options to the module. Check with the module's documentation for -what options it provides (if any). The options listed by default are for the -user mapping provider built in to Synapse. If using a custom module, you should -comment these options out and use those specified by the module instead. - -### Building a Custom OpenID Mapping Provider - -A custom mapping provider must specify the following methods: - -* `__init__(self, parsed_config)` - - Arguments: - - `parsed_config` - A configuration object that is the return value of the - `parse_config` method. You should set any configuration options needed by - the module here. -* `parse_config(config)` - - This method should have the `@staticmethod` decoration. - - Arguments: - - `config` - A `dict` representing the parsed content of the - `oidc_config.user_mapping_provider.config` homeserver config option. - Runs on homeserver startup. Providers should extract and validate - any option values they need here. - - Whatever is returned will be passed back to the user mapping provider module's - `__init__` method during construction. -* `get_remote_user_id(self, userinfo)` - - Arguments: - - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user - information from. - - This method must return a string, which is the unique, immutable identifier - for the user. Commonly the `sub` claim of the response. -* `map_user_attributes(self, userinfo, token, failures)` - - This method must be async. - - Arguments: - - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user - information from. - - `token` - A dictionary which includes information necessary to make - further requests to the OpenID provider. - - `failures` - An `int` that represents the amount of times the returned - mxid localpart mapping has failed. This should be used - to create a deduplicated mxid localpart which should be - returned instead. For example, if this method returns - `john.doe` as the value of `localpart` in the returned - dict, and that is already taken on the homeserver, this - method will be called again with the same parameters but - with failures=1. The method should then return a different - `localpart` value, such as `john.doe1`. - - Returns a dictionary with two keys: - - `localpart`: A string, used to generate the Matrix ID. If this is - `None`, the user is prompted to pick their own username. This is only used - during a user's first login. Once a localpart has been associated with a - remote user ID (see `get_remote_user_id`) it cannot be updated. - - `displayname`: An optional string, the display name for the user. -* `get_extra_attributes(self, userinfo, token)` - - This method must be async. - - Arguments: - - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user - information from. - - `token` - A dictionary which includes information necessary to make - further requests to the OpenID provider. - - Returns a dictionary that is suitable to be serialized to JSON. This - will be returned as part of the response during a successful login. - - Note that care should be taken to not overwrite any of the parameters - usually returned as part of the [login response](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login). - -### Default OpenID Mapping Provider - -Synapse has a built-in OpenID mapping provider if a custom provider isn't -specified in the config. It is located at -[`synapse.handlers.oidc.JinjaOidcMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/oidc.py). - -## SAML Mapping Providers - -The SAML mapping provider can be customized by editing the -`saml2_config.user_mapping_provider.module` config option. - -`saml2_config.user_mapping_provider.config` allows you to provide custom -configuration options to the module. Check with the module's documentation for -what options it provides (if any). The options listed by default are for the -user mapping provider built in to Synapse. If using a custom module, you should -comment these options out and use those specified by the module instead. - -### Building a Custom SAML Mapping Provider - -A custom mapping provider must specify the following methods: - -* `__init__(self, parsed_config, module_api)` - - Arguments: - - `parsed_config` - A configuration object that is the return value of the - `parse_config` method. You should set any configuration options needed by - the module here. - - `module_api` - a `synapse.module_api.ModuleApi` object which provides the - stable API available for extension modules. -* `parse_config(config)` - - This method should have the `@staticmethod` decoration. - - Arguments: - - `config` - A `dict` representing the parsed content of the - `saml_config.user_mapping_provider.config` homeserver config option. - Runs on homeserver startup. Providers should extract and validate - any option values they need here. - - Whatever is returned will be passed back to the user mapping provider module's - `__init__` method during construction. -* `get_saml_attributes(config)` - - This method should have the `@staticmethod` decoration. - - Arguments: - - `config` - A object resulting from a call to `parse_config`. - - Returns a tuple of two sets. The first set equates to the SAML auth - response attributes that are required for the module to function, whereas - the second set consists of those attributes which can be used if available, - but are not necessary. -* `get_remote_user_id(self, saml_response, client_redirect_url)` - - Arguments: - - `saml_response` - A `saml2.response.AuthnResponse` object to extract user - information from. - - `client_redirect_url` - A string, the URL that the client will be - redirected to. - - This method must return a string, which is the unique, immutable identifier - for the user. Commonly the `uid` claim of the response. -* `saml_response_to_user_attributes(self, saml_response, failures, client_redirect_url)` - - Arguments: - - `saml_response` - A `saml2.response.AuthnResponse` object to extract user - information from. - - `failures` - An `int` that represents the amount of times the returned - mxid localpart mapping has failed. This should be used - to create a deduplicated mxid localpart which should be - returned instead. For example, if this method returns - `john.doe` as the value of `mxid_localpart` in the returned - dict, and that is already taken on the homeserver, this - method will be called again with the same parameters but - with failures=1. The method should then return a different - `mxid_localpart` value, such as `john.doe1`. - - `client_redirect_url` - A string, the URL that the client will be - redirected to. - - This method must return a dictionary, which will then be used by Synapse - to build a new user. The following keys are allowed: - * `mxid_localpart` - A string, the mxid localpart of the new user. If this is - `None`, the user is prompted to pick their own username. This is only used - during a user's first login. Once a localpart has been associated with a - remote user ID (see `get_remote_user_id`) it cannot be updated. - * `displayname` - The displayname of the new user. If not provided, will default to - the value of `mxid_localpart`. - * `emails` - A list of emails for the new user. If not provided, will - default to an empty list. - - Alternatively it can raise a `synapse.api.errors.RedirectException` to - redirect the user to another page. This is useful to prompt the user for - additional information, e.g. if you want them to provide their own username. - It is the responsibility of the mapping provider to either redirect back - to `client_redirect_url` (including any additional information) or to - complete registration using methods from the `ModuleApi`. - -### Default SAML Mapping Provider - -Synapse has a built-in SAML mapping provider if a custom provider isn't -specified in the config. It is located at -[`synapse.handlers.saml.DefaultSamlMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/saml.py). diff --git a/docs/structured_logging.md b/docs/structured_logging.md deleted file mode 100644 index b1281667e0..0000000000 --- a/docs/structured_logging.md +++ /dev/null @@ -1,161 +0,0 @@ -# Structured Logging - -A structured logging system can be useful when your logs are destined for a -machine to parse and process. By maintaining its machine-readable characteristics, -it enables more efficient searching and aggregations when consumed by software -such as the "ELK stack". - -Synapse's structured logging system is configured via the file that Synapse's -`log_config` config option points to. The file should include a formatter which -uses the `synapse.logging.TerseJsonFormatter` class included with Synapse and a -handler which uses the above formatter. - -There is also a `synapse.logging.JsonFormatter` option which does not include -a timestamp in the resulting JSON. This is useful if the log ingester adds its -own timestamp. - -A structured logging configuration looks similar to the following: - -```yaml -version: 1 - -formatters: - structured: - class: synapse.logging.TerseJsonFormatter - -handlers: - file: - class: logging.handlers.TimedRotatingFileHandler - formatter: structured - filename: /path/to/my/logs/homeserver.log - when: midnight - backupCount: 3 # Does not include the current log file. - encoding: utf8 - -loggers: - synapse: - level: INFO - handlers: [remote] - synapse.storage.SQL: - level: WARNING -``` - -The above logging config will set Synapse as 'INFO' logging level by default, -with the SQL layer at 'WARNING', and will log to a file, stored as JSON. - -It is also possible to figure Synapse to log to a remote endpoint by using the -`synapse.logging.RemoteHandler` class included with Synapse. It takes the -following arguments: - -- `host`: Hostname or IP address of the log aggregator. -- `port`: Numerical port to contact on the host. -- `maximum_buffer`: (Optional, defaults to 1000) The maximum buffer size to allow. - -A remote structured logging configuration looks similar to the following: - -```yaml -version: 1 - -formatters: - structured: - class: synapse.logging.TerseJsonFormatter - -handlers: - remote: - class: synapse.logging.RemoteHandler - formatter: structured - host: 10.1.2.3 - port: 9999 - -loggers: - synapse: - level: INFO - handlers: [remote] - synapse.storage.SQL: - level: WARNING -``` - -The above logging config will set Synapse as 'INFO' logging level by default, -with the SQL layer at 'WARNING', and will log JSON formatted messages to a -remote endpoint at 10.1.2.3:9999. - -## Upgrading from legacy structured logging configuration - -Versions of Synapse prior to v1.23.0 included a custom structured logging -configuration which is deprecated. It used a `structured: true` flag and -configured `drains` instead of ``handlers`` and `formatters`. - -Synapse currently automatically converts the old configuration to the new -configuration, but this will be removed in a future version of Synapse. The -following reference can be used to update your configuration. Based on the drain -`type`, we can pick a new handler: - -1. For a type of `console`, `console_json`, or `console_json_terse`: a handler - with a class of `logging.StreamHandler` and a `stream` of `ext://sys.stdout` - or `ext://sys.stderr` should be used. -2. For a type of `file` or `file_json`: a handler of `logging.FileHandler` with - a location of the file path should be used. -3. For a type of `network_json_terse`: a handler of `synapse.logging.RemoteHandler` - with the host and port should be used. - -Then based on the drain `type` we can pick a new formatter: - -1. For a type of `console` or `file` no formatter is necessary. -2. For a type of `console_json` or `file_json`: a formatter of - `synapse.logging.JsonFormatter` should be used. -3. For a type of `console_json_terse` or `network_json_terse`: a formatter of - `synapse.logging.TerseJsonFormatter` should be used. - -For each new handler and formatter they should be added to the logging configuration -and then assigned to either a logger or the root logger. - -An example legacy configuration: - -```yaml -structured: true - -loggers: - synapse: - level: INFO - synapse.storage.SQL: - level: WARNING - -drains: - console: - type: console - location: stdout - file: - type: file_json - location: homeserver.log -``` - -Would be converted into a new configuration: - -```yaml -version: 1 - -formatters: - json: - class: synapse.logging.JsonFormatter - -handlers: - console: - class: logging.StreamHandler - location: ext://sys.stdout - file: - class: logging.FileHandler - formatter: json - filename: homeserver.log - -loggers: - synapse: - level: INFO - handlers: [console, file] - synapse.storage.SQL: - level: WARNING -``` - -The new logging configuration is a bit more verbose, but significantly more -flexible. It allows for configuration that were not previously possible, such as -sending plain logs over the network, or using different handlers for different -modules. diff --git a/docs/synctl_workers.md b/docs/synctl_workers.md deleted file mode 100644 index 15e37f608d..0000000000 --- a/docs/synctl_workers.md +++ /dev/null @@ -1,36 +0,0 @@ -### Using synctl with workers - -If you want to use `synctl` to manage your synapse processes, you will need to -create an an additional configuration file for the main synapse process. That -configuration should look like this: - -```yaml -worker_app: synapse.app.homeserver -``` - -Additionally, each worker app must be configured with the name of a "pid file", -to which it will write its process ID when it starts. For example, for a -synchrotron, you might write: - -```yaml -worker_pid_file: /home/matrix/synapse/worker1.pid -``` - -Finally, to actually run your worker-based synapse, you must pass synctl the `-a` -commandline option to tell it to operate on all the worker configurations found -in the given directory, e.g.: - -```sh -synctl -a $CONFIG/workers start -``` - -Currently one should always restart all workers when restarting or upgrading -synapse, unless you explicitly know it's safe not to. For instance, restarting -synapse without restarting all the synchrotrons may result in broken typing -notifications. - -To manipulate a specific worker, you pass the -w option to synctl: - -```sh -synctl -w $CONFIG/workers/worker1.yaml restart -``` diff --git a/docs/tcp_replication.md b/docs/tcp_replication.md deleted file mode 100644 index 15df949deb..0000000000 --- a/docs/tcp_replication.md +++ /dev/null @@ -1,257 +0,0 @@ -# TCP Replication - -## Motivation - -Previously the workers used an HTTP long poll mechanism to get updates -from the master, which had the problem of causing a lot of duplicate -work on the server. This TCP protocol replaces those APIs with the aim -of increased efficiency. - -## Overview - -The protocol is based on fire and forget, line based commands. An -example flow would be (where '>' indicates master to worker and -'<' worker to master flows): - - > SERVER example.com - < REPLICATE - > POSITION events master 53 53 - > RDATA events master 54 ["$foo1:bar.com", ...] - > RDATA events master 55 ["$foo4:bar.com", ...] - -The example shows the server accepting a new connection and sending its identity -with the `SERVER` command, followed by the client server to respond with the -position of all streams. The server then periodically sends `RDATA` commands -which have the format `RDATA `, where -the format of `` is defined by the individual streams. The -`` is the name of the Synapse process that generated the data -(usually "master"). - -Error reporting happens by either the client or server sending an ERROR -command, and usually the connection will be closed. - -Since the protocol is a simple line based, its possible to manually -connect to the server using a tool like netcat. A few things should be -noted when manually using the protocol: - -- The federation stream is only available if federation sending has - been disabled on the main process. -- The server will only time connections out that have sent a `PING` - command. If a ping is sent then the connection will be closed if no - further commands are receieved within 15s. Both the client and - server protocol implementations will send an initial PING on - connection and ensure at least one command every 5s is sent (not - necessarily `PING`). -- `RDATA` commands *usually* include a numeric token, however if the - stream has multiple rows to replicate per token the server will send - multiple `RDATA` commands, with all but the last having a token of - `batch`. See the documentation on `commands.RdataCommand` for - further details. - -## Architecture - -The basic structure of the protocol is line based, where the initial -word of each line specifies the command. The rest of the line is parsed -based on the command. For example, the RDATA command is defined as: - - RDATA - -(Note that may contains spaces, but cannot contain -newlines.) - -Blank lines are ignored. - -### Keep alives - -Both sides are expected to send at least one command every 5s or so, and -should send a `PING` command if necessary. If either side do not receive -a command within e.g. 15s then the connection should be closed. - -Because the server may be connected to manually using e.g. netcat, the -timeouts aren't enabled until an initial `PING` command is seen. Both -the client and server implementations below send a `PING` command -immediately on connection to ensure the timeouts are enabled. - -This ensures that both sides can quickly realize if the tcp connection -has gone and handle the situation appropriately. - -### Start up - -When a new connection is made, the server: - -- Sends a `SERVER` command, which includes the identity of the server, - allowing the client to detect if its connected to the expected - server -- Sends a `PING` command as above, to enable the client to time out - connections promptly. - -The client: - -- Sends a `NAME` command, allowing the server to associate a human - friendly name with the connection. This is optional. -- Sends a `PING` as above -- Sends a `REPLICATE` to get the current position of all streams. -- On receipt of a `SERVER` command, checks that the server name - matches the expected server name. - -### Error handling - -If either side detects an error it can send an `ERROR` command and close -the connection. - -If the client side loses the connection to the server it should -reconnect, following the steps above. - -### Congestion - -If the server sends messages faster than the client can consume them the -server will first buffer a (fairly large) number of commands and then -disconnect the client. This ensures that we don't queue up an unbounded -number of commands in memory and gives us a potential oppurtunity to -squawk loudly. When/if the client recovers it can reconnect to the -server and ask for missed messages. - -### Reliability - -In general the replication stream should be considered an unreliable -transport since e.g. commands are not resent if the connection -disappears. - -The exception to that are the replication streams, i.e. RDATA commands, -since these include tokens which can be used to restart the stream on -connection errors. - -The client should keep track of the token in the last RDATA command -received for each stream so that on reconneciton it can start streaming -from the correct place. Note: not all RDATA have valid tokens due to -batching. See `RdataCommand` for more details. - -### Example - -An example iteraction is shown below. Each line is prefixed with '>' -or '<' to indicate which side is sending, these are *not* included on -the wire: - - * connection established * - > SERVER localhost:8823 - > PING 1490197665618 - < NAME synapse.app.appservice - < PING 1490197665618 - < REPLICATE - > POSITION events master 1 1 - > POSITION backfill master 1 1 - > POSITION caches master 1 1 - > RDATA caches master 2 ["get_user_by_id",["@01register-user:localhost:8823"],1490197670513] - > RDATA events master 14 ["$149019767112vOHxz:localhost:8823", - "!AFDCvgApUmpdfVjIXm:localhost:8823","m.room.guest_access","",null] - < PING 1490197675618 - > ERROR server stopping - * connection closed by server * - -The `POSITION` command sent by the server is used to set the clients -position without needing to send data with the `RDATA` command. - -An example of a batched set of `RDATA` is: - - > RDATA caches master batch ["get_user_by_id",["@test:localhost:8823"],1490197670513] - > RDATA caches master batch ["get_user_by_id",["@test2:localhost:8823"],1490197670513] - > RDATA caches master batch ["get_user_by_id",["@test3:localhost:8823"],1490197670513] - > RDATA caches master 54 ["get_user_by_id",["@test4:localhost:8823"],1490197670513] - -In this case the client shouldn't advance their caches token until it -sees the the last `RDATA`. - -### List of commands - -The list of valid commands, with which side can send it: server (S) or -client (C): - -#### SERVER (S) - - Sent at the start to identify which server the client is talking to - -#### RDATA (S) - - A single update in a stream - -#### POSITION (S) - - On receipt of a POSITION command clients should check if they have missed any - updates, and if so then fetch them out of band. Sent in response to a - REPLICATE command (but can happen at any time). - - The POSITION command includes the source of the stream. Currently all streams - are written by a single process (usually "master"). If fetching missing - updates via HTTP API, rather than via the DB, then processes should make the - request to the appropriate process. - - Two positions are included, the "new" position and the last position sent respectively. - This allows servers to tell instances that the positions have advanced but no - data has been written, without clients needlessly checking to see if they - have missed any updates. - -#### ERROR (S, C) - - There was an error - -#### PING (S, C) - - Sent periodically to ensure the connection is still alive - -#### NAME (C) - - Sent at the start by client to inform the server who they are - -#### REPLICATE (C) - -Asks the server for the current position of all streams. - -#### USER_SYNC (C) - - A user has started or stopped syncing on this process. - -#### CLEAR_USER_SYNC (C) - - The server should clear all associated user sync data from the worker. - - This is used when a worker is shutting down. - -#### FEDERATION_ACK (C) - - Acknowledge receipt of some federation data - -### REMOTE_SERVER_UP (S, C) - - Inform other processes that a remote server may have come back online. - -See `synapse/replication/tcp/commands.py` for a detailed description and -the format of each command. - -### Cache Invalidation Stream - -The cache invalidation stream is used to inform workers when they need -to invalidate any of their caches in the data store. This is done by -streaming all cache invalidations done on master down to the workers, -assuming that any caches on the workers also exist on the master. - -Each individual cache invalidation results in a row being sent down -replication, which includes the cache name (the name of the function) -and they key to invalidate. For example: - - > RDATA caches master 550953771 ["get_user_by_id", ["@bob:example.com"], 1550574873251] - -Alternatively, an entire cache can be invalidated by sending down a `null` -instead of the key. For example: - - > RDATA caches master 550953772 ["get_user_by_id", null, 1550574873252] - -However, there are times when a number of caches need to be invalidated -at the same time with the same key. To reduce traffic we batch those -invalidations into a single poke by defining a special cache name that -workers understand to mean to expand to invalidate the correct caches. - -Currently the special cache names are declared in -`synapse/storage/_base.py` and are: - -1. `cs_cache_fake` ─ invalidates caches that depend on the current - state diff --git a/docs/templates.md b/docs/templates.md deleted file mode 100644 index a240f58b54..0000000000 --- a/docs/templates.md +++ /dev/null @@ -1,239 +0,0 @@ -# Templates - -Synapse uses parametrised templates to generate the content of emails it sends and -webpages it shows to users. - -By default, Synapse will use the templates listed [here](https://github.com/matrix-org/synapse/tree/master/synapse/res/templates). -Server admins can configure an additional directory for Synapse to look for templates -in, allowing them to specify custom templates: - -```yaml -templates: - custom_templates_directory: /path/to/custom/templates/ -``` - -If this setting is not set, or the files named below are not found within the directory, -default templates from within the Synapse package will be used. - -Templates that are given variables when being rendered are rendered using [Jinja 2](https://jinja.palletsprojects.com/en/2.11.x/). -Templates rendered by Jinja 2 can also access two functions on top of the functions -already available as part of Jinja 2: - -```python -format_ts(value: int, format: str) -> str -``` - -Formats a timestamp in milliseconds. - -Example: `reason.last_sent_ts|format_ts("%c")` - -```python -mxc_to_http(value: str, width: int, height: int, resize_method: str = "crop") -> str -``` - -Turns a `mxc://` URL for media content into an HTTP(S) one using the homeserver's -`public_baseurl` configuration setting as the URL's base. - -Example: `message.sender_avatar_url|mxc_to_http(32,32)` - - -## Email templates - -Below are the templates Synapse will look for when generating the content of an email: - -* `notif_mail.html` and `notif_mail.txt`: The contents of email notifications of missed - events. - When rendering, this template is given the following variables: - * `user_display_name`: the display name for the user receiving the notification - * `unsubscribe_link`: the link users can click to unsubscribe from email notifications - * `summary_text`: a summary of the notification(s). The text used can be customised - by configuring the various settings in the `email.subjects` section of the - configuration file. - * `rooms`: a list of rooms containing events to include in the email. Each element is - an object with the following attributes: - * `title`: a human-readable name for the room - * `hash`: a hash of the ID of the room - * `invite`: a boolean, which is `True` if the room is an invite the user hasn't - accepted yet, `False` otherwise - * `notifs`: a list of events, or an empty list if `invite` is `True`. Each element - is an object with the following attributes: - * `link`: a `matrix.to` link to the event - * `ts`: the time in milliseconds at which the event was received - * `messages`: a list of messages containing one message before the event, the - message in the event, and one message after the event. Each element is an - object with the following attributes: - * `event_type`: the type of the event - * `is_historical`: a boolean, which is `False` if the message is the one - that triggered the notification, `True` otherwise - * `id`: the ID of the event - * `ts`: the time in milliseconds at which the event was sent - * `sender_name`: the display name for the event's sender - * `sender_avatar_url`: the avatar URL (as a `mxc://` URL) for the event's - sender - * `sender_hash`: a hash of the user ID of the sender - * `link`: a `matrix.to` link to the room - * `reason`: information on the event that triggered the email to be sent. It's an - object with the following attributes: - * `room_id`: the ID of the room the event was sent in - * `room_name`: a human-readable name for the room the event was sent in - * `now`: the current time in milliseconds - * `received_at`: the time in milliseconds at which the event was received - * `delay_before_mail_ms`: the amount of time in milliseconds Synapse always waits - before ever emailing about a notification (to give the user a chance to respond - to other push or notice the window) - * `last_sent_ts`: the time in milliseconds at which a notification was last sent - for an event in this room - * `throttle_ms`: the minimum amount of time in milliseconds between two - notifications can be sent for this room -* `password_reset.html` and `password_reset.txt`: The contents of password reset emails - sent by the homeserver. - When rendering, these templates are given a `link` variable which contains the link the - user must click in order to reset their password. -* `registration.html` and `registration.txt`: The contents of address verification emails - sent during registration. - When rendering, these templates are given a `link` variable which contains the link the - user must click in order to validate their email address. -* `add_threepid.html` and `add_threepid.txt`: The contents of address verification emails - sent when an address is added to a Matrix account. - When rendering, these templates are given a `link` variable which contains the link the - user must click in order to validate their email address. - - -## HTML page templates for registration and password reset - -Below are the templates Synapse will look for when generating pages related to -registration and password reset: - -* `password_reset_confirmation.html`: An HTML page that a user will see when they follow - the link in the password reset email. The user will be asked to confirm the action - before their password is reset. - When rendering, this template is given the following variables: - * `sid`: the session ID for the password reset - * `token`: the token for the password reset - * `client_secret`: the client secret for the password reset -* `password_reset_success.html` and `password_reset_failure.html`: HTML pages for success - and failure that a user will see when they confirm the password reset flow using the - page above. - When rendering, `password_reset_success.html` is given no variable, and - `password_reset_failure.html` is given a `failure_reason`, which contains the reason - for the password reset failure. -* `registration_success.html` and `registration_failure.html`: HTML pages for success and - failure that a user will see when they follow the link in an address verification email - sent during registration. - When rendering, `registration_success.html` is given no variable, and - `registration_failure.html` is given a `failure_reason`, which contains the reason - for the registration failure. -* `add_threepid_success.html` and `add_threepid_failure.html`: HTML pages for success and - failure that a user will see when they follow the link in an address verification email - sent when an address is added to a Matrix account. - When rendering, `add_threepid_success.html` is given no variable, and - `add_threepid_failure.html` is given a `failure_reason`, which contains the reason - for the registration failure. - - -## HTML page templates for Single Sign-On (SSO) - -Below are the templates Synapse will look for when generating pages related to SSO: - -* `sso_login_idp_picker.html`: HTML page to prompt the user to choose an - Identity Provider during login. - This is only used if multiple SSO Identity Providers are configured. - When rendering, this template is given the following variables: - * `redirect_url`: the URL that the user will be redirected to after - login. - * `server_name`: the homeserver's name. - * `providers`: a list of available Identity Providers. Each element is - an object with the following attributes: - * `idp_id`: unique identifier for the IdP - * `idp_name`: user-facing name for the IdP - * `idp_icon`: if specified in the IdP config, an MXC URI for an icon - for the IdP - * `idp_brand`: if specified in the IdP config, a textual identifier - for the brand of the IdP - The rendered HTML page should contain a form which submits its results - back as a GET request, with the following query parameters: - * `redirectUrl`: the client redirect URI (ie, the `redirect_url` passed - to the template) - * `idp`: the 'idp_id' of the chosen IDP. -* `sso_auth_account_details.html`: HTML page to prompt new users to enter a - userid and confirm other details. This is only shown if the - SSO implementation (with any `user_mapping_provider`) does not return - a localpart. - When rendering, this template is given the following variables: - * `server_name`: the homeserver's name. - * `idp`: details of the SSO Identity Provider that the user logged in - with: an object with the following attributes: - * `idp_id`: unique identifier for the IdP - * `idp_name`: user-facing name for the IdP - * `idp_icon`: if specified in the IdP config, an MXC URI for an icon - for the IdP - * `idp_brand`: if specified in the IdP config, a textual identifier - for the brand of the IdP - * `user_attributes`: an object containing details about the user that - we received from the IdP. May have the following attributes: - * display_name: the user's display_name - * emails: a list of email addresses - The template should render a form which submits the following fields: - * `username`: the localpart of the user's chosen user id -* `sso_new_user_consent.html`: HTML page allowing the user to consent to the - server's terms and conditions. This is only shown for new users, and only if - `user_consent.require_at_registration` is set. - When rendering, this template is given the following variables: - * `server_name`: the homeserver's name. - * `user_id`: the user's matrix proposed ID. - * `user_profile.display_name`: the user's proposed display name, if any. - * consent_version: the version of the terms that the user will be - shown - * `terms_url`: a link to the page showing the terms. - The template should render a form which submits the following fields: - * `accepted_version`: the version of the terms accepted by the user - (ie, 'consent_version' from the input variables). -* `sso_redirect_confirm.html`: HTML page for a confirmation step before redirecting back - to the client with the login token. - When rendering, this template is given the following variables: - * `redirect_url`: the URL the user is about to be redirected to. - * `display_url`: the same as `redirect_url`, but with the query - parameters stripped. The intention is to have a - human-readable URL to show to users, not to use it as - the final address to redirect to. - * `server_name`: the homeserver's name. - * `new_user`: a boolean indicating whether this is the user's first time - logging in. - * `user_id`: the user's matrix ID. - * `user_profile.avatar_url`: an MXC URI for the user's avatar, if any. - `None` if the user has not set an avatar. - * `user_profile.display_name`: the user's display name. `None` if the user - has not set a display name. -* `sso_auth_confirm.html`: HTML page which notifies the user that they are authenticating - to confirm an operation on their account during the user interactive authentication - process. - When rendering, this template is given the following variables: - * `redirect_url`: the URL the user is about to be redirected to. - * `description`: the operation which the user is being asked to confirm - * `idp`: details of the Identity Provider that we will use to confirm - the user's identity: an object with the following attributes: - * `idp_id`: unique identifier for the IdP - * `idp_name`: user-facing name for the IdP - * `idp_icon`: if specified in the IdP config, an MXC URI for an icon - for the IdP - * `idp_brand`: if specified in the IdP config, a textual identifier - for the brand of the IdP -* `sso_auth_success.html`: HTML page shown after a successful user interactive - authentication session. - Note that this page must include the JavaScript which notifies of a successful - authentication (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback). - This template has no additional variables. -* `sso_auth_bad_user.html`: HTML page shown after a user-interactive authentication - session which does not map correctly onto the expected user. - When rendering, this template is given the following variables: - * `server_name`: the homeserver's name. - * `user_id_to_verify`: the MXID of the user that we are trying to - validate. -* `sso_account_deactivated.html`: HTML page shown during single sign-on if a deactivated - user (according to Synapse's database) attempts to login. - This template has no additional variables. -* `sso_error.html`: HTML page to display to users if something goes wrong during the - OpenID Connect authentication process. - When rendering, this template is given two variables: - * `error`: the technical name of the error - * `error_description`: a human-readable message for the error diff --git a/docs/turn-howto.md b/docs/turn-howto.md deleted file mode 100644 index e6812de69e..0000000000 --- a/docs/turn-howto.md +++ /dev/null @@ -1,296 +0,0 @@ -# Overview - -This document explains how to enable VoIP relaying on your homeserver with -TURN. - -The synapse Matrix homeserver supports integration with TURN server via the -[TURN server REST API](). This -allows the homeserver to generate credentials that are valid for use on the -TURN server through the use of a secret shared between the homeserver and the -TURN server. - -The following sections describe how to install [coturn]() (which implements the TURN REST API) and integrate it with synapse. - -## Requirements - -For TURN relaying with `coturn` to work, it must be hosted on a server/endpoint with a public IP. - -Hosting TURN behind a NAT (even with appropriate port forwarding) is known to cause issues -and to often not work. - -## `coturn` setup - -### Initial installation - -The TURN daemon `coturn` is available from a variety of sources such as native package managers, or installation from source. - -#### Debian installation - -Just install the debian package: - -```sh -apt install coturn -``` - -This will install and start a systemd service called `coturn`. - -#### Source installation - -1. Download the [latest release](https://github.com/coturn/coturn/releases/latest) from github. Unpack it and `cd` into the directory. - -1. Configure it: - - ```sh - ./configure - ``` - - You may need to install `libevent2`: if so, you should do so in - the way recommended by your operating system. You can ignore - warnings about lack of database support: a database is unnecessary - for this purpose. - -1. Build and install it: - - ```sh - make - make install - ``` - -### Configuration - -1. Create or edit the config file in `/etc/turnserver.conf`. The relevant - lines, with example values, are: - - ``` - use-auth-secret - static-auth-secret=[your secret key here] - realm=turn.myserver.org - ``` - - See `turnserver.conf` for explanations of the options. One way to generate - the `static-auth-secret` is with `pwgen`: - - ```sh - pwgen -s 64 1 - ``` - - A `realm` must be specified, but its value is somewhat arbitrary. (It is - sent to clients as part of the authentication flow.) It is conventional to - set it to be your server name. - -1. You will most likely want to configure coturn to write logs somewhere. The - easiest way is normally to send them to the syslog: - - ```sh - syslog - ``` - - (in which case, the logs will be available via `journalctl -u coturn` on a - systemd system). Alternatively, coturn can be configured to write to a - logfile - check the example config file supplied with coturn. - -1. Consider your security settings. TURN lets users request a relay which will - connect to arbitrary IP addresses and ports. The following configuration is - suggested as a minimum starting point: - - ``` - # VoIP traffic is all UDP. There is no reason to let users connect to arbitrary TCP endpoints via the relay. - no-tcp-relay - - # don't let the relay ever try to connect to private IP address ranges within your network (if any) - # given the turn server is likely behind your firewall, remember to include any privileged public IPs too. - denied-peer-ip=10.0.0.0-10.255.255.255 - denied-peer-ip=192.168.0.0-192.168.255.255 - denied-peer-ip=172.16.0.0-172.31.255.255 - - # special case the turn server itself so that client->TURN->TURN->client flows work - allowed-peer-ip=10.0.0.1 - - # consider whether you want to limit the quota of relayed streams per user (or total) to avoid risk of DoS. - user-quota=12 # 4 streams per video call, so 12 streams = 3 simultaneous relayed calls per user. - total-quota=1200 - ``` - -1. Also consider supporting TLS/DTLS. To do this, add the following settings - to `turnserver.conf`: - - ``` - # TLS certificates, including intermediate certs. - # For Let's Encrypt certificates, use `fullchain.pem` here. - cert=/path/to/fullchain.pem - - # TLS private key file - pkey=/path/to/privkey.pem - ``` - - In this case, replace the `turn:` schemes in the `turn_uri` settings below - with `turns:`. - - We recommend that you only try to set up TLS/DTLS once you have set up a - basic installation and got it working. - -1. Ensure your firewall allows traffic into the TURN server on the ports - you've configured it to listen on (By default: 3478 and 5349 for TURN - traffic (remember to allow both TCP and UDP traffic), and ports 49152-65535 - for the UDP relay.) - -1. We do not recommend running a TURN server behind NAT, and are not aware of - anyone doing so successfully. - - If you want to try it anyway, you will at least need to tell coturn its - external IP address: - - ``` - external-ip=192.88.99.1 - ``` - - ... and your NAT gateway must forward all of the relayed ports directly - (eg, port 56789 on the external IP must be always be forwarded to port - 56789 on the internal IP). - - If you get this working, let us know! - -1. (Re)start the turn server: - - * If you used the Debian package (or have set up a systemd unit yourself): - ```sh - systemctl restart coturn - ``` - - * If you installed from source: - - ```sh - bin/turnserver -o - ``` - -## Synapse setup - -Your homeserver configuration file needs the following extra keys: - -1. "`turn_uris`": This needs to be a yaml list of public-facing URIs - for your TURN server to be given out to your clients. Add separate - entries for each transport your TURN server supports. -2. "`turn_shared_secret`": This is the secret shared between your - homeserver and your TURN server, so you should set it to the same - string you used in turnserver.conf. -3. "`turn_user_lifetime`": This is the amount of time credentials - generated by your homeserver are valid for (in milliseconds). - Shorter times offer less potential for abuse at the expense of - increased traffic between web clients and your homeserver to - refresh credentials. The TURN REST API specification recommends - one day (86400000). -4. "`turn_allow_guests`": Whether to allow guest users to use the - TURN server. This is enabled by default, as otherwise VoIP will - not work reliably for guests. However, it does introduce a - security risk as it lets guests connect to arbitrary endpoints - without having gone through a CAPTCHA or similar to register a - real account. - -As an example, here is the relevant section of the config file for `matrix.org`. The -`turn_uris` are appropriate for TURN servers listening on the default ports, with no TLS. - - turn_uris: [ "turn:turn.matrix.org?transport=udp", "turn:turn.matrix.org?transport=tcp" ] - turn_shared_secret: "n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons" - turn_user_lifetime: 86400000 - turn_allow_guests: True - -After updating the homeserver configuration, you must restart synapse: - - * If you use synctl: - ```sh - cd /where/you/run/synapse - ./synctl restart - ``` - * If you use systemd: - ```sh - systemctl restart matrix-synapse.service - ``` -... and then reload any clients (or wait an hour for them to refresh their -settings). - -## Troubleshooting - -The normal symptoms of a misconfigured TURN server are that calls between -devices on different networks ring, but get stuck at "call -connecting". Unfortunately, troubleshooting this can be tricky. - -Here are a few things to try: - - * Check that your TURN server is not behind NAT. As above, we're not aware of - anyone who has successfully set this up. - - * Check that you have opened your firewall to allow TCP and UDP traffic to the - TURN ports (normally 3478 and 5349). - - * Check that you have opened your firewall to allow UDP traffic to the UDP - relay ports (49152-65535 by default). - - * Some WebRTC implementations (notably, that of Google Chrome) appear to get - confused by TURN servers which are reachable over IPv6 (this appears to be - an unexpected side-effect of its handling of multiple IP addresses as - defined by - [`draft-ietf-rtcweb-ip-handling`](https://tools.ietf.org/html/draft-ietf-rtcweb-ip-handling-12)). - - Try removing any AAAA records for your TURN server, so that it is only - reachable over IPv4. - - * Enable more verbose logging in coturn via the `verbose` setting: - - ``` - verbose - ``` - - ... and then see if there are any clues in its logs. - - * If you are using a browser-based client under Chrome, check - `chrome://webrtc-internals/` for insights into the internals of the - negotiation. On Firefox, check the "Connection Log" on `about:webrtc`. - - (Understanding the output is beyond the scope of this document!) - - * You can test your Matrix homeserver TURN setup with https://test.voip.librepush.net/. - Note that this test is not fully reliable yet, so don't be discouraged if - the test fails. - [Here](https://github.com/matrix-org/voip-tester) is the github repo of the - source of the tester, where you can file bug reports. - - * There is a WebRTC test tool at - https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/. To - use it, you will need a username/password for your TURN server. You can - either: - - * look for the `GET /_matrix/client/r0/voip/turnServer` request made by a - matrix client to your homeserver in your browser's network inspector. In - the response you should see `username` and `password`. Or: - - * Use the following shell commands: - - ```sh - secret=staticAuthSecretHere - - u=$((`date +%s` + 3600)):test - p=$(echo -n $u | openssl dgst -hmac $secret -sha1 -binary | base64) - echo -e "username: $u\npassword: $p" - ``` - - Or: - - * Temporarily configure coturn to accept a static username/password. To do - this, comment out `use-auth-secret` and `static-auth-secret` and add the - following: - - ``` - lt-cred-mech - user=username:password - ``` - - **Note**: these settings will not take effect unless `use-auth-secret` - and `static-auth-secret` are disabled. - - Restart coturn after changing the configuration file. - - Remember to restore the original settings to go back to testing with - Matrix clients! - - If the TURN server is working correctly, you should see at least one `relay` - entry in the results. diff --git a/docs/upgrading/upgrading_from_pre_synapse_1.0.md b/docs/upgrading/upgrading_from_pre_synapse_1.0.md new file mode 100644 index 0000000000..086899a9d8 --- /dev/null +++ b/docs/upgrading/upgrading_from_pre_synapse_1.0.md @@ -0,0 +1,335 @@ +# MSC1711 Certificates FAQ + +## Historical Note +This document was originally written to guide server admins through the upgrade +path towards Synapse 1.0. Specifically, +[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md) +required that all servers present valid TLS certificates on their federation +API. Admins were encouraged to achieve compliance from version 0.99.0 (released +in February 2019) ahead of version 1.0 (released June 2019) enforcing the +certificate checks. + +Much of what follows is now outdated since most admins will have already +upgraded, however it may be of use to those with old installs returning to the +project. + +If you are setting up a server from scratch you almost certainly should look at +the [installation guide](setup/installation.md) instead. + +## Introduction +The goal of Synapse 0.99.0 is to act as a stepping stone to Synapse 1.0.0. It +supports the r0.1 release of the server to server specification, but is +compatible with both the legacy Matrix federation behaviour (pre-r0.1) as well +as post-r0.1 behaviour, in order to allow for a smooth upgrade across the +federation. + +The most important thing to know is that Synapse 1.0.0 will require a valid TLS +certificate on federation endpoints. Self signed certificates will not be +sufficient. + +Synapse 0.99.0 makes it easy to configure TLS certificates and will +interoperate with both >= 1.0.0 servers as well as existing servers yet to +upgrade. + +**It is critical that all admins upgrade to 0.99.0 and configure a valid TLS +certificate.** Admins will have 1 month to do so, after which 1.0.0 will be +released and those servers without a valid certificate will not longer be able +to federate with >= 1.0.0 servers. + +Full details on how to carry out this configuration change is given +[below](#configuring-certificates-for-compatibility-with-synapse-100). A +timeline and some frequently asked questions are also given below. + +For more details and context on the release of the r0.1 Server/Server API and +imminent Matrix 1.0 release, you can also see our +[main talk from FOSDEM 2019](https://matrix.org/blog/2019/02/04/matrix-at-fosdem-2019/). + +## Contents +* Timeline +* Configuring certificates for compatibility with Synapse 1.0 +* FAQ + * Synapse 0.99.0 has just been released, what do I need to do right now? + * How do I upgrade? + * What will happen if I do not set up a valid federation certificate + immediately? + * What will happen if I do nothing at all? + * When do I need a SRV record or .well-known URI? + * Can I still use an SRV record? + * I have created a .well-known URI. Do I still need an SRV record? + * It used to work just fine, why are you breaking everything? + * Can I manage my own certificates rather than having Synapse renew + certificates itself? + * Do you still recommend against using a reverse proxy on the federation port? + * Do I still need to give my TLS certificates to Synapse if I am using a + reverse proxy? + * Do I need the same certificate for the client and federation port? + * How do I tell Synapse to reload my keys/certificates after I replace them? + +## Timeline + +**5th Feb 2019 - Synapse 0.99.0 is released.** + +All server admins are encouraged to upgrade. + +0.99.0: + +- provides support for ACME to make setting up Let's Encrypt certs easy, as + well as .well-known support. + +- does not enforce that a valid CA cert is present on the federation API, but + rather makes it easy to set one up. + +- provides support for .well-known + +Admins should upgrade and configure a valid CA cert. Homeservers that require a +.well-known entry (see below), should retain their SRV record and use it +alongside their .well-known record. + +**10th June 2019 - Synapse 1.0.0 is released** + +1.0.0 is scheduled for release on 10th June. In +accordance with the the [S2S spec](https://matrix.org/docs/spec/server_server/r0.1.0.html) +1.0.0 will enforce certificate validity. This means that any homeserver without a +valid certificate after this point will no longer be able to federate with +1.0.0 servers. + +## Configuring certificates for compatibility with Synapse 1.0.0 + +### If you do not currently have an SRV record + +In this case, your `server_name` points to the host where your Synapse is +running. There is no need to create a `.well-known` URI or an SRV record, but +you will need to give Synapse a valid, signed, certificate. + +### If you do have an SRV record currently + +If you are using an SRV record, your matrix domain (`server_name`) may not +point to the same host that your Synapse is running on (the 'target +domain'). (If it does, you can follow the recommendation above; otherwise, read +on.) + +Let's assume that your `server_name` is `example.com`, and your Synapse is +hosted at a target domain of `customer.example.net`. Currently you should have +an SRV record which looks like: + +``` +_matrix._tcp.example.com. IN SRV 10 5 8000 customer.example.net. +``` + +In this situation, you have three choices for how to proceed: + +#### Option 1: give Synapse a certificate for your matrix domain + +Synapse 1.0 will expect your server to present a TLS certificate for your +`server_name` (`example.com` in the above example). You can achieve this by acquiring a +certificate for the `server_name` yourself (for example, using `certbot`), and giving it +and the key to Synapse via `tls_certificate_path` and `tls_private_key_path`. + +#### Option 2: run Synapse behind a reverse proxy + +If you have an existing reverse proxy set up with correct TLS certificates for +your domain, you can simply route all traffic through the reverse proxy by +updating the SRV record appropriately (or removing it, if the proxy listens on +8448). + +See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a +reverse proxy. + +#### Option 3: add a .well-known file to delegate your matrix traffic + +This will allow you to keep Synapse on a separate domain, without having to +give it a certificate for the matrix domain. + +You can do this with a `.well-known` file as follows: + + 1. Keep the SRV record in place - it is needed for backwards compatibility + with Synapse 0.34 and earlier. + + 2. Give Synapse a certificate corresponding to the target domain + (`customer.example.net` in the above example). You can do this by acquire a + certificate for the target domain and giving it to Synapse via `tls_certificate_path` + and `tls_private_key_path`. + + 3. Restart Synapse to ensure the new certificate is loaded. + + 4. Arrange for a `.well-known` file at + `https:///.well-known/matrix/server` with contents: + + ```json + {"m.server": ""} + ``` + + where the target server name is resolved as usual (i.e. SRV lookup, falling + back to talking to port 8448). + + In the above example, where synapse is listening on port 8000, + `https://example.com/.well-known/matrix/server` should have `m.server` set to one of: + + 1. `customer.example.net` ─ with a SRV record on + `_matrix._tcp.customer.example.com` pointing to port 8000, or: + + 2. `customer.example.net` ─ updating synapse to listen on the default port + 8448, or: + + 3. `customer.example.net:8000` ─ ensuring that if there is a reverse proxy + on `customer.example.net:8000` it correctly handles HTTP requests with + Host header set to `customer.example.net:8000`. + +## FAQ + +### Synapse 0.99.0 has just been released, what do I need to do right now? + +Upgrade as soon as you can in preparation for Synapse 1.0.0, and update your +TLS certificates as [above](#configuring-certificates-for-compatibility-with-synapse-100). + +### What will happen if I do not set up a valid federation certificate immediately? + +Nothing initially, but once 1.0.0 is in the wild it will not be possible to +federate with 1.0.0 servers. + +### What will happen if I do nothing at all? + +If the admin takes no action at all, and remains on a Synapse < 0.99.0 then the +homeserver will be unable to federate with those who have implemented +.well-known. Then, as above, once the month upgrade window has expired the +homeserver will not be able to federate with any Synapse >= 1.0.0 + +### When do I need a SRV record or .well-known URI? + +If your homeserver listens on the default federation port (8448), and your +`server_name` points to the host that your homeserver runs on, you do not need an +SRV record or `.well-known/matrix/server` URI. + +For instance, if you registered `example.com` and pointed its DNS A record at a +fresh Upcloud VPS or similar, you could install Synapse 0.99 on that host, +giving it a server_name of `example.com`, and it would automatically generate a +valid TLS certificate for you via Let's Encrypt and no SRV record or +`.well-known` URI would be needed. + +This is the common case, although you can add an SRV record or +`.well-known/matrix/server` URI for completeness if you wish. + +**However**, if your server does not listen on port 8448, or if your `server_name` +does not point to the host that your homeserver runs on, you will need to let +other servers know how to find it. + +In this case, you should see ["If you do have an SRV record +currently"](#if-you-do-have-an-srv-record-currently) above. + +### Can I still use an SRV record? + +Firstly, if you didn't need an SRV record before (because your server is +listening on port 8448 of your server_name), you certainly don't need one now: +the defaults are still the same. + +If you previously had an SRV record, you can keep using it provided you are +able to give Synapse a TLS certificate corresponding to your server name. For +example, suppose you had the following SRV record, which directs matrix traffic +for example.com to matrix.example.com:443: + +``` +_matrix._tcp.example.com. IN SRV 10 5 443 matrix.example.com +``` + +In this case, Synapse must be given a certificate for example.com - or be +configured to acquire one from Let's Encrypt. + +If you are unable to give Synapse a certificate for your server_name, you will +also need to use a .well-known URI instead. However, see also "I have created a +.well-known URI. Do I still need an SRV record?". + +### I have created a .well-known URI. Do I still need an SRV record? + +As of Synapse 0.99, Synapse will first check for the existence of a `.well-known` +URI and follow any delegation it suggests. It will only then check for the +existence of an SRV record. + +That means that the SRV record will often be redundant. However, you should +remember that there may still be older versions of Synapse in the federation +which do not understand `.well-known` URIs, so if you removed your SRV record you +would no longer be able to federate with them. + +It is therefore best to leave the SRV record in place for now. Synapse 0.34 and +earlier will follow the SRV record (and not care about the invalid +certificate). Synapse 0.99 and later will follow the .well-known URI, with the +correct certificate chain. + +### It used to work just fine, why are you breaking everything? + +We have always wanted Matrix servers to be as easy to set up as possible, and +so back when we started federation in 2014 we didn't want admins to have to go +through the cumbersome process of buying a valid TLS certificate to run a +server. This was before Let's Encrypt came along and made getting a free and +valid TLS certificate straightforward. So instead, we adopted a system based on +[Perspectives](https://en.wikipedia.org/wiki/Convergence_(SSL)): an approach +where you check a set of "notary servers" (in practice, homeservers) to vouch +for the validity of a certificate rather than having it signed by a CA. As long +as enough different notaries agree on the certificate's validity, then it is +trusted. + +However, in practice this has never worked properly. Most people only use the +default notary server (matrix.org), leading to inadvertent centralisation which +we want to eliminate. Meanwhile, we never implemented the full consensus +algorithm to query the servers participating in a room to determine consensus +on whether a given certificate is valid. This is fiddly to get right +(especially in face of sybil attacks), and we found ourselves questioning +whether it was worth the effort to finish the work and commit to maintaining a +secure certificate validation system as opposed to focusing on core Matrix +development. + +Meanwhile, Let's Encrypt came along in 2016, and put the final nail in the +coffin of the Perspectives project (which was already pretty dead). So, the +Spec Core Team decided that a better approach would be to mandate valid TLS +certificates for federation alongside the rest of the Web. More details can be +found in +[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach). + +This results in a breaking change, which is disruptive, but absolutely critical +for the security model. However, the existence of Let's Encrypt as a trivial +way to replace the old self-signed certificates with valid CA-signed ones helps +smooth things over massively, especially as Synapse can now automate Let's +Encrypt certificate generation if needed. + +### Can I manage my own certificates rather than having Synapse renew certificates itself? + +Yes, you are welcome to manage your certificates yourself. Synapse will only +attempt to obtain certificates from Let's Encrypt if you configure it to do +so.The only requirement is that there is a valid TLS cert present for +federation end points. + +### Do you still recommend against using a reverse proxy on the federation port? + +We no longer actively recommend against using a reverse proxy. Many admins will +find it easier to direct federation traffic to a reverse proxy and manage their +own TLS certificates, and this is a supported configuration. + +See [the reverse proxy documentation](reverse_proxy.md) for information on setting up a +reverse proxy. + +### Do I still need to give my TLS certificates to Synapse if I am using a reverse proxy? + +Practically speaking, this is no longer necessary. + +If you are using a reverse proxy for all of your TLS traffic, then you can set +`no_tls: True`. In that case, the only reason Synapse needs the certificate is +to populate a legacy 'tls_fingerprints' field in the federation API. This is +ignored by Synapse 0.99.0 and later, and the only time pre-0.99 Synapses will +check it is when attempting to fetch the server keys - and generally this is +delegated via `matrix.org`, which is on 0.99.0. + +However, there is a bug in Synapse 0.99.0 +[4554]() which prevents +Synapse from starting if you do not give it a TLS certificate. To work around +this, you can give it any TLS certificate at all. This will be fixed soon. + +### Do I need the same certificate for the client and federation port? + +No. There is nothing stopping you from using different certificates, +particularly if you are using a reverse proxy. However, Synapse will use the +same certificate on any ports where TLS is configured. + +### How do I tell Synapse to reload my keys/certificates after I replace them? + +Synapse will reload the keys and certificates when it receives a SIGHUP - for +example `kill -HUP $(cat homeserver.pid)`. Alternatively, simply restart +Synapse, though this will result in downtime while it restarts. diff --git a/docs/usage/administration/manhole.md b/docs/usage/administration/manhole.md new file mode 100644 index 0000000000..715ed840f2 --- /dev/null +++ b/docs/usage/administration/manhole.md @@ -0,0 +1,99 @@ +Using the synapse manhole +========================= + +The "manhole" allows server administrators to access a Python shell on a running +Synapse installation. This is a very powerful mechanism for administration and +debugging. + +**_Security Warning_** + +Note that this will give administrative access to synapse to **all users** with +shell access to the server. It should therefore **not** be enabled in +environments where untrusted users have shell access. + +## Configuring the manhole + +To enable it, first uncomment the `manhole` listener configuration in +`homeserver.yaml`. The configuration is slightly different if you're using docker. + +#### Docker config + +If you are using Docker, set `bind_addresses` to `['0.0.0.0']` as shown: + +```yaml +listeners: + - port: 9000 + bind_addresses: ['0.0.0.0'] + type: manhole +``` + +When using `docker run` to start the server, you will then need to change the command to the following to include the +`manhole` port forwarding. The `-p 127.0.0.1:9000:9000` below is important: it +ensures that access to the `manhole` is only possible for local users. + +```bash +docker run -d --name synapse \ + --mount type=volume,src=synapse-data,dst=/data \ + -p 8008:8008 \ + -p 127.0.0.1:9000:9000 \ + matrixdotorg/synapse:latest +``` + +#### Native config + +If you are not using docker, set `bind_addresses` to `['::1', '127.0.0.1']` as shown. +The `bind_addresses` in the example below is important: it ensures that access to the +`manhole` is only possible for local users). + +```yaml +listeners: + - port: 9000 + bind_addresses: ['::1', '127.0.0.1'] + type: manhole +``` + +### Security settings + +The following config options are available: + +- `username` - The username for the manhole (defaults to `matrix`) +- `password` - The password for the manhole (defaults to `rabbithole`) +- `ssh_priv_key` - The path to a private SSH key (defaults to a hardcoded value) +- `ssh_pub_key` - The path to a public SSH key (defaults to a hardcoded value) + +For example: + +```yaml +manhole_settings: + username: manhole + password: mypassword + ssh_priv_key: "/home/synapse/manhole_keys/id_rsa" + ssh_pub_key: "/home/synapse/manhole_keys/id_rsa.pub" +``` + + +## Accessing synapse manhole + +Then restart synapse, and point an ssh client at port 9000 on localhost, using +the username and password configured in `homeserver.yaml` - with the default +configuration, this would be: + +```bash +ssh -p9000 matrix@localhost +``` + +Then enter the password when prompted (the default is `rabbithole`). + +This gives a Python REPL in which `hs` gives access to the +`synapse.server.HomeServer` object - which in turn gives access to many other +parts of the process. + +Note that, prior to Synapse 1.41, any call which returns a coroutine will need to be wrapped in `ensureDeferred`. + +As a simple example, retrieving an event from the database: + +```pycon +>>> from twisted.internet import defer +>>> defer.ensureDeferred(hs.get_datastore().get_event('$1416420717069yeQaw:matrix.org')) +> +``` diff --git a/docs/usage/administration/monitoring.md b/docs/usage/administration/monitoring.md new file mode 100644 index 0000000000..4a77d5604c --- /dev/null +++ b/docs/usage/administration/monitoring.md @@ -0,0 +1,283 @@ +# How to monitor Synapse metrics using Prometheus + +1. Install Prometheus: + + Follow instructions at + + +1. Enable Synapse metrics: + + There are two methods of enabling metrics in Synapse. + + The first serves the metrics as a part of the usual web server and + can be enabled by adding the \"metrics\" resource to the existing + listener as such: + + ```yaml + resources: + - names: + - client + - metrics + ``` + + This provides a simple way of adding metrics to your Synapse + installation, and serves under `/_synapse/metrics`. If you do not + wish your metrics be publicly exposed, you will need to either + filter it out at your load balancer, or use the second method. + + The second method runs the metrics server on a different port, in a + different thread to Synapse. This can make it more resilient to + heavy load meaning metrics cannot be retrieved, and can be exposed + to just internal networks easier. The served metrics are available + over HTTP only, and will be available at `/_synapse/metrics`. + + Add a new listener to homeserver.yaml: + + ```yaml + listeners: + - type: metrics + port: 9000 + bind_addresses: + - '0.0.0.0' + ``` + + For both options, you will need to ensure that `enable_metrics` is + set to `True`. + +1. Restart Synapse. + +1. Add a Prometheus target for Synapse. + + It needs to set the `metrics_path` to a non-default value (under + `scrape_configs`): + + ```yaml + - job_name: "synapse" + scrape_interval: 15s + metrics_path: "/_synapse/metrics" + static_configs: + - targets: ["my.server.here:port"] + ``` + + where `my.server.here` is the IP address of Synapse, and `port` is + the listener port configured with the `metrics` resource. + + If your prometheus is older than 1.5.2, you will need to replace + `static_configs` in the above with `target_groups`. + +1. Restart Prometheus. + +1. Consider using the [grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana/) + and required [recording rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus/) + +## Monitoring workers + +To monitor a Synapse installation using [workers](workers.md), +every worker needs to be monitored independently, in addition to +the main homeserver process. This is because workers don't send +their metrics to the main homeserver process, but expose them +directly (if they are configured to do so). + +To allow collecting metrics from a worker, you need to add a +`metrics` listener to its configuration, by adding the following +under `worker_listeners`: + +```yaml + - type: metrics + bind_address: '' + port: 9101 +``` + +The `bind_address` and `port` parameters should be set so that +the resulting listener can be reached by prometheus, and they +don't clash with an existing worker. +With this example, the worker's metrics would then be available +on `http://127.0.0.1:9101`. + +Example Prometheus target for Synapse with workers: + +```yaml + - job_name: "synapse" + scrape_interval: 15s + metrics_path: "/_synapse/metrics" + static_configs: + - targets: ["my.server.here:port"] + labels: + instance: "my.server" + job: "master" + index: 1 + - targets: ["my.workerserver.here:port"] + labels: + instance: "my.server" + job: "generic_worker" + index: 1 + - targets: ["my.workerserver.here:port"] + labels: + instance: "my.server" + job: "generic_worker" + index: 2 + - targets: ["my.workerserver.here:port"] + labels: + instance: "my.server" + job: "media_repository" + index: 1 +``` + +Labels (`instance`, `job`, `index`) can be defined as anything. +The labels are used to group graphs in grafana. + +## Renaming of metrics & deprecation of old names in 1.2 + +Synapse 1.2 updates the Prometheus metrics to match the naming +convention of the upstream `prometheus_client`. The old names are +considered deprecated and will be removed in a future version of +Synapse. + +| New Name | Old Name | +| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------- | +| python_gc_objects_collected_total | python_gc_objects_collected | +| python_gc_objects_uncollectable_total | python_gc_objects_uncollectable | +| python_gc_collections_total | python_gc_collections | +| process_cpu_seconds_total | process_cpu_seconds | +| synapse_federation_client_sent_transactions_total | synapse_federation_client_sent_transactions | +| synapse_federation_client_events_processed_total | synapse_federation_client_events_processed | +| synapse_event_processing_loop_count_total | synapse_event_processing_loop_count | +| synapse_event_processing_loop_room_count_total | synapse_event_processing_loop_room_count | +| synapse_util_metrics_block_count_total | synapse_util_metrics_block_count | +| synapse_util_metrics_block_time_seconds_total | synapse_util_metrics_block_time_seconds | +| synapse_util_metrics_block_ru_utime_seconds_total | synapse_util_metrics_block_ru_utime_seconds | +| synapse_util_metrics_block_ru_stime_seconds_total | synapse_util_metrics_block_ru_stime_seconds | +| synapse_util_metrics_block_db_txn_count_total | synapse_util_metrics_block_db_txn_count | +| synapse_util_metrics_block_db_txn_duration_seconds_total | synapse_util_metrics_block_db_txn_duration_seconds | +| synapse_util_metrics_block_db_sched_duration_seconds_total | synapse_util_metrics_block_db_sched_duration_seconds | +| synapse_background_process_start_count_total | synapse_background_process_start_count | +| synapse_background_process_ru_utime_seconds_total | synapse_background_process_ru_utime_seconds | +| synapse_background_process_ru_stime_seconds_total | synapse_background_process_ru_stime_seconds | +| synapse_background_process_db_txn_count_total | synapse_background_process_db_txn_count | +| synapse_background_process_db_txn_duration_seconds_total | synapse_background_process_db_txn_duration_seconds | +| synapse_background_process_db_sched_duration_seconds_total | synapse_background_process_db_sched_duration_seconds | +| synapse_storage_events_persisted_events_total | synapse_storage_events_persisted_events | +| synapse_storage_events_persisted_events_sep_total | synapse_storage_events_persisted_events_sep | +| synapse_storage_events_state_delta_total | synapse_storage_events_state_delta | +| synapse_storage_events_state_delta_single_event_total | synapse_storage_events_state_delta_single_event | +| synapse_storage_events_state_delta_reuse_delta_total | synapse_storage_events_state_delta_reuse_delta | +| synapse_federation_server_received_pdus_total | synapse_federation_server_received_pdus | +| synapse_federation_server_received_edus_total | synapse_federation_server_received_edus | +| synapse_handler_presence_notified_presence_total | synapse_handler_presence_notified_presence | +| synapse_handler_presence_federation_presence_out_total | synapse_handler_presence_federation_presence_out | +| synapse_handler_presence_presence_updates_total | synapse_handler_presence_presence_updates | +| synapse_handler_presence_timers_fired_total | synapse_handler_presence_timers_fired | +| synapse_handler_presence_federation_presence_total | synapse_handler_presence_federation_presence | +| synapse_handler_presence_bump_active_time_total | synapse_handler_presence_bump_active_time | +| synapse_federation_client_sent_edus_total | synapse_federation_client_sent_edus | +| synapse_federation_client_sent_pdu_destinations_count_total | synapse_federation_client_sent_pdu_destinations:count | +| synapse_federation_client_sent_pdu_destinations_total | synapse_federation_client_sent_pdu_destinations:total | +| synapse_handlers_appservice_events_processed_total | synapse_handlers_appservice_events_processed | +| synapse_notifier_notified_events_total | synapse_notifier_notified_events | +| synapse_push_bulk_push_rule_evaluator_push_rules_invalidation_counter_total | synapse_push_bulk_push_rule_evaluator_push_rules_invalidation_counter | +| synapse_push_bulk_push_rule_evaluator_push_rules_state_size_counter_total | synapse_push_bulk_push_rule_evaluator_push_rules_state_size_counter | +| synapse_http_httppusher_http_pushes_processed_total | synapse_http_httppusher_http_pushes_processed | +| synapse_http_httppusher_http_pushes_failed_total | synapse_http_httppusher_http_pushes_failed | +| synapse_http_httppusher_badge_updates_processed_total | synapse_http_httppusher_badge_updates_processed | +| synapse_http_httppusher_badge_updates_failed_total | synapse_http_httppusher_badge_updates_failed | + +Removal of deprecated metrics & time based counters becoming histograms in 0.31.0 +--------------------------------------------------------------------------------- + +The duplicated metrics deprecated in Synapse 0.27.0 have been removed. + +All time duration-based metrics have been changed to be seconds. This +affects: + +| msec -> sec metrics | +| -------------------------------------- | +| python_gc_time | +| python_twisted_reactor_tick_time | +| synapse_storage_query_time | +| synapse_storage_schedule_time | +| synapse_storage_transaction_time | + +Several metrics have been changed to be histograms, which sort entries +into buckets and allow better analysis. The following metrics are now +histograms: + +| Altered metrics | +| ------------------------------------------------ | +| python_gc_time | +| python_twisted_reactor_pending_calls | +| python_twisted_reactor_tick_time | +| synapse_http_server_response_time_seconds | +| synapse_storage_query_time | +| synapse_storage_schedule_time | +| synapse_storage_transaction_time | + +Block and response metrics renamed for 0.27.0 +--------------------------------------------- + +Synapse 0.27.0 begins the process of rationalising the duplicate +`*:count` metrics reported for the resource tracking for code blocks and +HTTP requests. + +At the same time, the corresponding `*:total` metrics are being renamed, +as the `:total` suffix no longer makes sense in the absence of a +corresponding `:count` metric. + +To enable a graceful migration path, this release just adds new names +for the metrics being renamed. A future release will remove the old +ones. + +The following table shows the new metrics, and the old metrics which +they are replacing. + +| New name | Old name | +| ------------------------------------------------------------- | ---------------------------------------------------------- | +| synapse_util_metrics_block_count | synapse_util_metrics_block_timer:count | +| synapse_util_metrics_block_count | synapse_util_metrics_block_ru_utime:count | +| synapse_util_metrics_block_count | synapse_util_metrics_block_ru_stime:count | +| synapse_util_metrics_block_count | synapse_util_metrics_block_db_txn_count:count | +| synapse_util_metrics_block_count | synapse_util_metrics_block_db_txn_duration:count | +| synapse_util_metrics_block_time_seconds | synapse_util_metrics_block_timer:total | +| synapse_util_metrics_block_ru_utime_seconds | synapse_util_metrics_block_ru_utime:total | +| synapse_util_metrics_block_ru_stime_seconds | synapse_util_metrics_block_ru_stime:total | +| synapse_util_metrics_block_db_txn_count | synapse_util_metrics_block_db_txn_count:total | +| synapse_util_metrics_block_db_txn_duration_seconds | synapse_util_metrics_block_db_txn_duration:total | +| synapse_http_server_response_count | synapse_http_server_requests | +| synapse_http_server_response_count | synapse_http_server_response_time:count | +| synapse_http_server_response_count | synapse_http_server_response_ru_utime:count | +| synapse_http_server_response_count | synapse_http_server_response_ru_stime:count | +| synapse_http_server_response_count | synapse_http_server_response_db_txn_count:count | +| synapse_http_server_response_count | synapse_http_server_response_db_txn_duration:count | +| synapse_http_server_response_time_seconds | synapse_http_server_response_time:total | +| synapse_http_server_response_ru_utime_seconds | synapse_http_server_response_ru_utime:total | +| synapse_http_server_response_ru_stime_seconds | synapse_http_server_response_ru_stime:total | +| synapse_http_server_response_db_txn_count | synapse_http_server_response_db_txn_count:total | +| synapse_http_server_response_db_txn_duration_seconds | synapse_http_server_response_db_txn_duration:total | + +Standard Metric Names +--------------------- + +As of synapse version 0.18.2, the format of the process-wide metrics has +been changed to fit prometheus standard naming conventions. Additionally +the units have been changed to seconds, from miliseconds. + +| New name | Old name | +| ---------------------------------------- | --------------------------------- | +| process_cpu_user_seconds_total | process_resource_utime / 1000 | +| process_cpu_system_seconds_total | process_resource_stime / 1000 | +| process_open_fds (no \'type\' label) | process_fds | + +The python-specific counts of garbage collector performance have been +renamed. + +| New name | Old name | +| -------------------------------- | -------------------------- | +| python_gc_time | reactor_gc_time | +| python_gc_unreachable_total | reactor_gc_unreachable | +| python_gc_counts | reactor_gc_counts | + +The twisted-specific reactor metrics have been renamed. + +| New name | Old name | +| -------------------------------------- | ----------------------- | +| python_twisted_reactor_pending_calls | reactor_pending_calls | +| python_twisted_reactor_tick_time | reactor_tick_time | diff --git a/docs/usage/configuration/application_services.md b/docs/usage/configuration/application_services.md new file mode 100644 index 0000000000..e4592010a2 --- /dev/null +++ b/docs/usage/configuration/application_services.md @@ -0,0 +1,35 @@ +# Registering an Application Service + +The registration of new application services depends on the homeserver used. +In synapse, you need to create a new configuration file for your AS and add it +to the list specified under the `app_service_config_files` config +option in your synapse config. + +For example: + +```yaml +app_service_config_files: +- /home/matrix/.synapse/.yaml +``` + +The format of the AS configuration file is as follows: + +```yaml +url: +as_token: +hs_token: +sender_localpart: +namespaces: + users: # List of users we're interested in + - exclusive: + regex: + group_id: + - ... + aliases: [] # List of aliases we're interested in + rooms: [] # List of room ids we're interested in +``` + +`exclusive`: If enabled, only this application service is allowed to register users in its namespace(s). +`group_id`: All users of this application service are dynamically joined to this group. This is useful for e.g user organisation or flairs. + +See the [spec](https://matrix.org/docs/spec/application_service/unstable.html) for further details on how application services work. diff --git a/docs/usage/configuration/consent_tracking.md b/docs/usage/configuration/consent_tracking.md new file mode 100644 index 0000000000..fb1fec80fe --- /dev/null +++ b/docs/usage/configuration/consent_tracking.md @@ -0,0 +1,197 @@ +Support in Synapse for tracking agreement to server terms and conditions +======================================================================== + +Synapse 0.30 introduces support for tracking whether users have agreed to the +terms and conditions set by the administrator of a server - and blocking access +to the server until they have. + +There are several parts to this functionality; each requires some specific +configuration in `homeserver.yaml` to be enabled. + +Note that various parts of the configuation and this document refer to the +"privacy policy": agreement with a privacy policy is one particular use of this +feature, but of course adminstrators can specify other terms and conditions +unrelated to "privacy" per se. + +Collecting policy agreement from a user +--------------------------------------- + +Synapse can be configured to serve the user a simple policy form with an +"accept" button. Clicking "Accept" records the user's acceptance in the +database and shows a success page. + +To enable this, first create templates for the policy and success pages. +These should be stored on the local filesystem. + +These templates use the [Jinja2](http://jinja.pocoo.org) templating language, +and [docs/privacy_policy_templates](https://github.com/matrix-org/synapse/tree/develop/docs/privacy_policy_templates/) +gives examples of the sort of thing that can be done. + +Note that the templates must be stored under a name giving the language of the +template - currently this must always be `en` (for "English"); +internationalisation support is intended for the future. + +The template for the policy itself should be versioned and named according to +the version: for example `1.0.html`. The version of the policy which the user +has agreed to is stored in the database. + +Once the templates are in place, make the following changes to `homeserver.yaml`: + + 1. Add a `user_consent` section, which should look like: + + ```yaml + user_consent: + template_dir: privacy_policy_templates + version: 1.0 + ``` + + `template_dir` points to the directory containing the policy + templates. `version` defines the version of the policy which will be served + to the user. In the example above, Synapse will serve + `privacy_policy_templates/en/1.0.html`. + + + 2. Add a `form_secret` setting at the top level: + + + ```yaml + form_secret: "" + ``` + + This should be set to an arbitrary secret string (try `pwgen -y 30` to + generate suitable secrets). + + More on what this is used for below. + + 3. Add `consent` wherever the `client` resource is currently enabled in the + `listeners` configuration. For example: + + ```yaml + listeners: + - port: 8008 + resources: + - names: + - client + - consent + ``` + + +Finally, ensure that `jinja2` is installed. If you are using a virtualenv, this +should be a matter of `pip install Jinja2`. On debian, try `apt-get install +python-jinja2`. + +Once this is complete, and the server has been restarted, try visiting +`https:///_matrix/consent`. If correctly configured, this should give +an error "Missing string query parameter 'u'". It is now possible to manually +construct URIs where users can give their consent. + +### Enabling consent tracking at registration + +1. Add the following to your configuration: + + ```yaml + user_consent: + require_at_registration: true + policy_name: "Privacy Policy" # or whatever you'd like to call the policy + ``` + +2. In your consent templates, make use of the `public_version` variable to + see if an unauthenticated user is viewing the page. This is typically + wrapped around the form that would be used to actually agree to the document: + + ```html + {% if not public_version %} + +
+ + + + +
+ {% endif %} + ``` + +3. Restart Synapse to apply the changes. + +Visiting `https:///_matrix/consent` should now give you a view of the privacy +document. This is what users will be able to see when registering for accounts. + +### Constructing the consent URI + +It may be useful to manually construct the "consent URI" for a given user - for +instance, in order to send them an email asking them to consent. To do this, +take the base `https:///_matrix/consent` URL and add the following +query parameters: + + * `u`: the user id of the user. This can either be a full MXID + (`@user:server.com`) or just the localpart (`user`). + + * `h`: hex-encoded HMAC-SHA256 of `u` using the `form_secret` as a key. It is + possible to calculate this on the commandline with something like: + + ```bash + echo -n '' | openssl sha256 -hmac '' + ``` + + This should result in a URI which looks something like: + `https:///_matrix/consent?u=&h=68a152465a4d...`. + + +Note that not providing a `u` parameter will be interpreted as wanting to view +the document from an unauthenticated perspective, such as prior to registration. +Therefore, the `h` parameter is not required in this scenario. To enable this +behaviour, set `require_at_registration` to `true` in your `user_consent` config. + + +Sending users a server notice asking them to agree to the policy +---------------------------------------------------------------- + +It is possible to configure Synapse to send a [server +notice](server_notices.md) to anybody who has not yet agreed to the current +version of the policy. To do so: + + * ensure that the consent resource is configured, as in the previous section + + * ensure that server notices are configured, as in [the server notice documentation](server_notices.md). + + * Add `server_notice_content` under `user_consent` in `homeserver.yaml`. For + example: + + ```yaml + user_consent: + server_notice_content: + msgtype: m.text + body: >- + Please give your consent to the privacy policy at %(consent_uri)s. + ``` + + Synapse automatically replaces the placeholder `%(consent_uri)s` with the + consent uri for that user. + + * ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base + URI that clients use to connect to the server. (It is used to construct + `consent_uri` in the server notice.) + + +Blocking users from using the server until they agree to the policy +------------------------------------------------------------------- + +Synapse can be configured to block any attempts to join rooms or send messages +until the user has given their agreement to the policy. (Joining the server +notices room is exempted from this). + +To enable this, add `block_events_error` under `user_consent`. For example: + +```yaml +user_consent: + block_events_error: >- + You can't send any messages until you consent to the privacy policy at + %(consent_uri)s. +``` + +Synapse automatically replaces the placeholder `%(consent_uri)s` with the +consent uri for that user. + +ensure that `public_baseurl` is set in `homeserver.yaml`, and gives the base +URI that clients use to connect to the server. (It is used to construct +`consent_uri` in the error.) diff --git a/docs/usage/configuration/json_web_tokens.md b/docs/usage/configuration/json_web_tokens.md new file mode 100644 index 0000000000..5be9fd26e3 --- /dev/null +++ b/docs/usage/configuration/json_web_tokens.md @@ -0,0 +1,97 @@ +# JWT Login Type + +Synapse comes with a non-standard login type to support +[JSON Web Tokens](https://en.wikipedia.org/wiki/JSON_Web_Token). In general the +documentation for +[the login endpoint](https://matrix.org/docs/spec/client_server/r0.6.1#login) +is still valid (and the mechanism works similarly to the +[token based login](https://matrix.org/docs/spec/client_server/r0.6.1#token-based)). + +To log in using a JSON Web Token, clients should submit a `/login` request as +follows: + +```json +{ + "type": "org.matrix.login.jwt", + "token": "" +} +``` + +Note that the login type of `m.login.jwt` is supported, but is deprecated. This +will be removed in a future version of Synapse. + +The `token` field should include the JSON web token with the following claims: + +* The `sub` (subject) claim is required and should encode the local part of the + user ID. +* The expiration time (`exp`), not before time (`nbf`), and issued at (`iat`) + claims are optional, but validated if present. +* The issuer (`iss`) claim is optional, but required and validated if configured. +* The audience (`aud`) claim is optional, but required and validated if configured. + Providing the audience claim when not configured will cause validation to fail. + +In the case that the token is not valid, the homeserver must respond with +`403 Forbidden` and an error code of `M_FORBIDDEN`. + +As with other login types, there are additional fields (e.g. `device_id` and +`initial_device_display_name`) which can be included in the above request. + +## Preparing Synapse + +The JSON Web Token integration in Synapse uses the +[`PyJWT`](https://pypi.org/project/pyjwt/) library, which must be installed +as follows: + + * The relevant libraries are included in the Docker images and Debian packages + provided by `matrix.org` so no further action is needed. + + * If you installed Synapse into a virtualenv, run `/path/to/env/bin/pip + install synapse[pyjwt]` to install the necessary dependencies. + + * For other installation mechanisms, see the documentation provided by the + maintainer. + +To enable the JSON web token integration, you should then add an `jwt_config` section +to your configuration file (or uncomment the `enabled: true` line in the +existing section). See [sample_config.yaml](./sample_config.yaml) for some +sample settings. + +## How to test JWT as a developer + +Although JSON Web Tokens are typically generated from an external server, the +examples below use [PyJWT](https://pyjwt.readthedocs.io/en/latest/) directly. + +1. Configure Synapse with JWT logins, note that this example uses a pre-shared + secret and an algorithm of HS256: + + ```yaml + jwt_config: + enabled: true + secret: "my-secret-token" + algorithm: "HS256" + ``` +2. Generate a JSON web token: + + ```bash + $ pyjwt --key=my-secret-token --alg=HS256 encode sub=test-user + eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc + ``` +3. Query for the login types and ensure `org.matrix.login.jwt` is there: + + ```bash + curl http://localhost:8080/_matrix/client/r0/login + ``` +4. Login used the generated JSON web token from above: + + ```bash + $ curl http://localhost:8082/_matrix/client/r0/login -X POST \ + --data '{"type":"org.matrix.login.jwt","token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80aqRPTeuVPBIBZkYhNTJJ-_-zQIc"}' + { + "access_token": "", + "device_id": "ACBDEFGHI", + "home_server": "localhost:8080", + "user_id": "@test-user:localhost:8480" + } + ``` + +You should now be able to use the returned access token to query the client API. diff --git a/docs/usage/configuration/message_retention_policies.md b/docs/usage/configuration/message_retention_policies.md new file mode 100644 index 0000000000..9214d6d7e9 --- /dev/null +++ b/docs/usage/configuration/message_retention_policies.md @@ -0,0 +1,205 @@ +# Message retention policies + +Synapse admins can enable support for message retention policies on +their homeserver. Message retention policies exist at a room level, +follow the semantics described in +[MSC1763](https://github.com/matrix-org/matrix-doc/blob/matthew/msc1763/proposals/1763-configurable-retention-periods.md), +and allow server and room admins to configure how long messages should +be kept in a homeserver's database before being purged from it. +**Please note that, as this feature isn't part of the Matrix +specification yet, this implementation is to be considered as +experimental.** + +A message retention policy is mainly defined by its `max_lifetime` +parameter, which defines how long a message can be kept around after +it was sent to the room. If a room doesn't have a message retention +policy, and there's no default one for a given server, then no message +sent in that room is ever purged on that server. + +MSC1763 also specifies semantics for a `min_lifetime` parameter which +defines the amount of time after which an event _can_ get purged (after +it was sent to the room), but Synapse doesn't currently support it +beyond registering it. + +Both `max_lifetime` and `min_lifetime` are optional parameters. + +Note that message retention policies don't apply to state events. + +Once an event reaches its expiry date (defined as the time it was sent +plus the value for `max_lifetime` in the room), two things happen: + +* Synapse stops serving the event to clients via any endpoint. +* The message gets picked up by the next purge job (see the "Purge jobs" + section) and is removed from Synapse's database. + +Since purge jobs don't run continuously, this means that an event might +stay in a server's database for longer than the value for `max_lifetime` +in the room would allow, though hidden from clients. + +Similarly, if a server (with support for message retention policies +enabled) receives from another server an event that should have been +purged according to its room's policy, then the receiving server will +process and store that event until it's picked up by the next purge job, +though it will always hide it from clients. + +Synapse requires at least one message in each room, so it will never +delete the last message in a room. It will, however, hide it from +clients. + + +## Server configuration + +Support for this feature can be enabled and configured in the +`retention` section of the Synapse configuration file (see the +[sample file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518)). + +To enable support for message retention policies, set the setting +`enabled` in this section to `true`. + + +### Default policy + +A default message retention policy is a policy defined in Synapse's +configuration that is used by Synapse for every room that doesn't have a +message retention policy configured in its state. This allows server +admins to ensure that messages are never kept indefinitely in a server's +database. + +A default policy can be defined as such, in the `retention` section of +the configuration file: + +```yaml +default_policy: + min_lifetime: 1d + max_lifetime: 1y +``` + +Here, `min_lifetime` and `max_lifetime` have the same meaning and level +of support as previously described. They can be expressed either as a +duration (using the units `s` (seconds), `m` (minutes), `h` (hours), +`d` (days), `w` (weeks) and `y` (years)) or as a number of milliseconds. + + +### Purge jobs + +Purge jobs are the jobs that Synapse runs in the background to purge +expired events from the database. They are only run if support for +message retention policies is enabled in the server's configuration. If +no configuration for purge jobs is configured by the server admin, +Synapse will use a default configuration, which is described in the +[sample configuration file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518). + +Some server admins might want a finer control on when events are removed +depending on an event's room's policy. This can be done by setting the +`purge_jobs` sub-section in the `retention` section of the configuration +file. An example of such configuration could be: + +```yaml +purge_jobs: + - longest_max_lifetime: 3d + interval: 12h + - shortest_max_lifetime: 3d + longest_max_lifetime: 1w + interval: 1d + - shortest_max_lifetime: 1w + interval: 2d +``` + +In this example, we define three jobs: + +* one that runs twice a day (every 12 hours) and purges events in rooms + which policy's `max_lifetime` is lower or equal to 3 days. +* one that runs once a day and purges events in rooms which policy's + `max_lifetime` is between 3 days and a week. +* one that runs once every 2 days and purges events in rooms which + policy's `max_lifetime` is greater than a week. + +Note that this example is tailored to show different configurations and +features slightly more jobs than it's probably necessary (in practice, a +server admin would probably consider it better to replace the two last +jobs with one that runs once a day and handles rooms which which +policy's `max_lifetime` is greater than 3 days). + +Keep in mind, when configuring these jobs, that a purge job can become +quite heavy on the server if it targets many rooms, therefore prefer +having jobs with a low interval that target a limited set of rooms. Also +make sure to include a job with no minimum and one with no maximum to +make sure your configuration handles every policy. + +As previously mentioned in this documentation, while a purge job that +runs e.g. every day means that an expired event might stay in the +database for up to a day after its expiry, Synapse hides expired events +from clients as soon as they expire, so the event is not visible to +local users between its expiry date and the moment it gets purged from +the server's database. + + +### Lifetime limits + +Server admins can set limits on the values of `max_lifetime` to use when +purging old events in a room. These limits can be defined as such in the +`retention` section of the configuration file: + +```yaml +allowed_lifetime_min: 1d +allowed_lifetime_max: 1y +``` + +The limits are considered when running purge jobs. If necessary, the +effective value of `max_lifetime` will be brought between +`allowed_lifetime_min` and `allowed_lifetime_max` (inclusive). +This means that, if the value of `max_lifetime` defined in the room's state +is lower than `allowed_lifetime_min`, the value of `allowed_lifetime_min` +will be used instead. Likewise, if the value of `max_lifetime` is higher +than `allowed_lifetime_max`, the value of `allowed_lifetime_max` will be +used instead. + +In the example above, we ensure Synapse never deletes events that are less +than one day old, and that it always deletes events that are over a year +old. + +If a default policy is set, and its `max_lifetime` value is lower than +`allowed_lifetime_min` or higher than `allowed_lifetime_max`, the same +process applies. + +Both parameters are optional; if one is omitted Synapse won't use it to +adjust the effective value of `max_lifetime`. + +Like other settings in this section, these parameters can be expressed +either as a duration or as a number of milliseconds. + + +## Room configuration + +To configure a room's message retention policy, a room's admin or +moderator needs to send a state event in that room with the type +`m.room.retention` and the following content: + +```json +{ + "max_lifetime": ... +} +``` + +In this event's content, the `max_lifetime` parameter has the same +meaning as previously described, and needs to be expressed in +milliseconds. The event's content can also include a `min_lifetime` +parameter, which has the same meaning and limited support as previously +described. + +Note that over every server in the room, only the ones with support for +message retention policies will actually remove expired events. This +support is currently not enabled by default in Synapse. + + +## Note on reclaiming disk space + +While purge jobs actually delete data from the database, the disk space +used by the database might not decrease immediately on the database's +host. However, even though the database engine won't free up the disk +space, it will start writing new data into where the purged data was. + +If you want to reclaim the freed disk space anyway and return it to the +operating system, the server admin needs to run `VACUUM FULL;` (or +`VACUUM;` for SQLite databases) on Synapse's database (see the related +[PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-vacuum.html)). diff --git a/docs/usage/configuration/registration_captcha.md b/docs/usage/configuration/registration_captcha.md new file mode 100644 index 0000000000..49419ce8df --- /dev/null +++ b/docs/usage/configuration/registration_captcha.md @@ -0,0 +1,37 @@ +# Overview +A captcha can be enabled on your homeserver to help prevent bots from registering +accounts. Synapse currently uses Google's reCAPTCHA service which requires API keys +from Google. + +## Getting API keys + +1. Create a new site at +1. Set the label to anything you want +1. Set the type to reCAPTCHA v2 using the "I'm not a robot" Checkbox option. +This is the only type of captcha that works with Synapse. +1. Add the public hostname for your server, as set in `public_baseurl` +in `homeserver.yaml`, to the list of authorized domains. If you have not set +`public_baseurl`, use `server_name`. +1. Agree to the terms of service and submit. +1. Copy your site key and secret key and add them to your `homeserver.yaml` +configuration file + ```yaml + recaptcha_public_key: YOUR_SITE_KEY + recaptcha_private_key: YOUR_SECRET_KEY + ``` +1. Enable the CAPTCHA for new registrations + ```yaml + enable_registration_captcha: true + ``` +1. Go to the settings page for the CAPTCHA you just created +1. Uncheck the "Verify the origin of reCAPTCHA solutions" checkbox so that the +captcha can be displayed in any client. If you do not disable this option then you +must specify the domains of every client that is allowed to display the CAPTCHA. + +## Configuring IP used for auth + +The reCAPTCHA API requires that the IP address of the user who solved the +CAPTCHA is sent. If the client is connecting through a proxy or load balancer, +it may be required to use the `X-Forwarded-For` (XFF) header instead of the origin +IP address. This can be configured using the `x_forwarded` directive in the +listeners section of the `homeserver.yaml` configuration file. diff --git a/docs/usage/configuration/server_notices.md b/docs/usage/configuration/server_notices.md new file mode 100644 index 0000000000..339d10a0ab --- /dev/null +++ b/docs/usage/configuration/server_notices.md @@ -0,0 +1,61 @@ +# Server Notices + +'Server Notices' are a new feature introduced in Synapse 0.30. They provide a +channel whereby server administrators can send messages to users on the server. + +They are used as part of communication of the server polices (see +[Consent Tracking](consent_tracking.md)), however the intention is that +they may also find a use for features such as "Message of the day". + +This is a feature specific to Synapse, but it uses standard Matrix +communication mechanisms, so should work with any Matrix client. + +## User experience + +When the user is first sent a server notice, they will get an invitation to a +room (typically called 'Server Notices', though this is configurable in +`homeserver.yaml`). They will be **unable to reject** this invitation - +attempts to do so will receive an error. + +Once they accept the invitation, they will see the notice message in the room +history; it will appear to have come from the 'server notices user' (see +below). + +The user is prevented from sending any messages in this room by the power +levels. + +Having joined the room, the user can leave the room if they want. Subsequent +server notices will then cause a new room to be created. + +## Synapse configuration + +Server notices come from a specific user id on the server. Server +administrators are free to choose the user id - something like `server` is +suggested, meaning the notices will come from +`@server:`. Once the Server Notices user is configured, that +user id becomes a special, privileged user, so administrators should ensure +that **it is not already allocated**. + +In order to support server notices, it is necessary to add some configuration +to the `homeserver.yaml` file. In particular, you should add a `server_notices` +section, which should look like this: + +```yaml +server_notices: + system_mxid_localpart: server + system_mxid_display_name: "Server Notices" + system_mxid_avatar_url: "mxc://server.com/oumMVlgDnLYFaPVkExemNVVZ" + room_name: "Server Notices" +``` + +The only compulsory setting is `system_mxid_localpart`, which defines the user +id of the Server Notices user, as above. `room_name` defines the name of the +room which will be created. + +`system_mxid_display_name` and `system_mxid_avatar_url` can be used to set the +displayname and avatar of the Server Notices user. + +## Sending notices + +To send server notices to users you can use the +[admin_api](admin_api/server_notices.md). diff --git a/docs/usage/configuration/structured_logging.md b/docs/usage/configuration/structured_logging.md new file mode 100644 index 0000000000..b1281667e0 --- /dev/null +++ b/docs/usage/configuration/structured_logging.md @@ -0,0 +1,161 @@ +# Structured Logging + +A structured logging system can be useful when your logs are destined for a +machine to parse and process. By maintaining its machine-readable characteristics, +it enables more efficient searching and aggregations when consumed by software +such as the "ELK stack". + +Synapse's structured logging system is configured via the file that Synapse's +`log_config` config option points to. The file should include a formatter which +uses the `synapse.logging.TerseJsonFormatter` class included with Synapse and a +handler which uses the above formatter. + +There is also a `synapse.logging.JsonFormatter` option which does not include +a timestamp in the resulting JSON. This is useful if the log ingester adds its +own timestamp. + +A structured logging configuration looks similar to the following: + +```yaml +version: 1 + +formatters: + structured: + class: synapse.logging.TerseJsonFormatter + +handlers: + file: + class: logging.handlers.TimedRotatingFileHandler + formatter: structured + filename: /path/to/my/logs/homeserver.log + when: midnight + backupCount: 3 # Does not include the current log file. + encoding: utf8 + +loggers: + synapse: + level: INFO + handlers: [remote] + synapse.storage.SQL: + level: WARNING +``` + +The above logging config will set Synapse as 'INFO' logging level by default, +with the SQL layer at 'WARNING', and will log to a file, stored as JSON. + +It is also possible to figure Synapse to log to a remote endpoint by using the +`synapse.logging.RemoteHandler` class included with Synapse. It takes the +following arguments: + +- `host`: Hostname or IP address of the log aggregator. +- `port`: Numerical port to contact on the host. +- `maximum_buffer`: (Optional, defaults to 1000) The maximum buffer size to allow. + +A remote structured logging configuration looks similar to the following: + +```yaml +version: 1 + +formatters: + structured: + class: synapse.logging.TerseJsonFormatter + +handlers: + remote: + class: synapse.logging.RemoteHandler + formatter: structured + host: 10.1.2.3 + port: 9999 + +loggers: + synapse: + level: INFO + handlers: [remote] + synapse.storage.SQL: + level: WARNING +``` + +The above logging config will set Synapse as 'INFO' logging level by default, +with the SQL layer at 'WARNING', and will log JSON formatted messages to a +remote endpoint at 10.1.2.3:9999. + +## Upgrading from legacy structured logging configuration + +Versions of Synapse prior to v1.23.0 included a custom structured logging +configuration which is deprecated. It used a `structured: true` flag and +configured `drains` instead of ``handlers`` and `formatters`. + +Synapse currently automatically converts the old configuration to the new +configuration, but this will be removed in a future version of Synapse. The +following reference can be used to update your configuration. Based on the drain +`type`, we can pick a new handler: + +1. For a type of `console`, `console_json`, or `console_json_terse`: a handler + with a class of `logging.StreamHandler` and a `stream` of `ext://sys.stdout` + or `ext://sys.stderr` should be used. +2. For a type of `file` or `file_json`: a handler of `logging.FileHandler` with + a location of the file path should be used. +3. For a type of `network_json_terse`: a handler of `synapse.logging.RemoteHandler` + with the host and port should be used. + +Then based on the drain `type` we can pick a new formatter: + +1. For a type of `console` or `file` no formatter is necessary. +2. For a type of `console_json` or `file_json`: a formatter of + `synapse.logging.JsonFormatter` should be used. +3. For a type of `console_json_terse` or `network_json_terse`: a formatter of + `synapse.logging.TerseJsonFormatter` should be used. + +For each new handler and formatter they should be added to the logging configuration +and then assigned to either a logger or the root logger. + +An example legacy configuration: + +```yaml +structured: true + +loggers: + synapse: + level: INFO + synapse.storage.SQL: + level: WARNING + +drains: + console: + type: console + location: stdout + file: + type: file_json + location: homeserver.log +``` + +Would be converted into a new configuration: + +```yaml +version: 1 + +formatters: + json: + class: synapse.logging.JsonFormatter + +handlers: + console: + class: logging.StreamHandler + location: ext://sys.stdout + file: + class: logging.FileHandler + formatter: json + filename: homeserver.log + +loggers: + synapse: + level: INFO + handlers: [console, file] + synapse.storage.SQL: + level: WARNING +``` + +The new logging configuration is a bit more verbose, but significantly more +flexible. It allows for configuration that were not previously possible, such as +sending plain logs over the network, or using different handlers for different +modules. diff --git a/docs/usage/configuration/templates.md b/docs/usage/configuration/templates.md new file mode 100644 index 0000000000..a240f58b54 --- /dev/null +++ b/docs/usage/configuration/templates.md @@ -0,0 +1,239 @@ +# Templates + +Synapse uses parametrised templates to generate the content of emails it sends and +webpages it shows to users. + +By default, Synapse will use the templates listed [here](https://github.com/matrix-org/synapse/tree/master/synapse/res/templates). +Server admins can configure an additional directory for Synapse to look for templates +in, allowing them to specify custom templates: + +```yaml +templates: + custom_templates_directory: /path/to/custom/templates/ +``` + +If this setting is not set, or the files named below are not found within the directory, +default templates from within the Synapse package will be used. + +Templates that are given variables when being rendered are rendered using [Jinja 2](https://jinja.palletsprojects.com/en/2.11.x/). +Templates rendered by Jinja 2 can also access two functions on top of the functions +already available as part of Jinja 2: + +```python +format_ts(value: int, format: str) -> str +``` + +Formats a timestamp in milliseconds. + +Example: `reason.last_sent_ts|format_ts("%c")` + +```python +mxc_to_http(value: str, width: int, height: int, resize_method: str = "crop") -> str +``` + +Turns a `mxc://` URL for media content into an HTTP(S) one using the homeserver's +`public_baseurl` configuration setting as the URL's base. + +Example: `message.sender_avatar_url|mxc_to_http(32,32)` + + +## Email templates + +Below are the templates Synapse will look for when generating the content of an email: + +* `notif_mail.html` and `notif_mail.txt`: The contents of email notifications of missed + events. + When rendering, this template is given the following variables: + * `user_display_name`: the display name for the user receiving the notification + * `unsubscribe_link`: the link users can click to unsubscribe from email notifications + * `summary_text`: a summary of the notification(s). The text used can be customised + by configuring the various settings in the `email.subjects` section of the + configuration file. + * `rooms`: a list of rooms containing events to include in the email. Each element is + an object with the following attributes: + * `title`: a human-readable name for the room + * `hash`: a hash of the ID of the room + * `invite`: a boolean, which is `True` if the room is an invite the user hasn't + accepted yet, `False` otherwise + * `notifs`: a list of events, or an empty list if `invite` is `True`. Each element + is an object with the following attributes: + * `link`: a `matrix.to` link to the event + * `ts`: the time in milliseconds at which the event was received + * `messages`: a list of messages containing one message before the event, the + message in the event, and one message after the event. Each element is an + object with the following attributes: + * `event_type`: the type of the event + * `is_historical`: a boolean, which is `False` if the message is the one + that triggered the notification, `True` otherwise + * `id`: the ID of the event + * `ts`: the time in milliseconds at which the event was sent + * `sender_name`: the display name for the event's sender + * `sender_avatar_url`: the avatar URL (as a `mxc://` URL) for the event's + sender + * `sender_hash`: a hash of the user ID of the sender + * `link`: a `matrix.to` link to the room + * `reason`: information on the event that triggered the email to be sent. It's an + object with the following attributes: + * `room_id`: the ID of the room the event was sent in + * `room_name`: a human-readable name for the room the event was sent in + * `now`: the current time in milliseconds + * `received_at`: the time in milliseconds at which the event was received + * `delay_before_mail_ms`: the amount of time in milliseconds Synapse always waits + before ever emailing about a notification (to give the user a chance to respond + to other push or notice the window) + * `last_sent_ts`: the time in milliseconds at which a notification was last sent + for an event in this room + * `throttle_ms`: the minimum amount of time in milliseconds between two + notifications can be sent for this room +* `password_reset.html` and `password_reset.txt`: The contents of password reset emails + sent by the homeserver. + When rendering, these templates are given a `link` variable which contains the link the + user must click in order to reset their password. +* `registration.html` and `registration.txt`: The contents of address verification emails + sent during registration. + When rendering, these templates are given a `link` variable which contains the link the + user must click in order to validate their email address. +* `add_threepid.html` and `add_threepid.txt`: The contents of address verification emails + sent when an address is added to a Matrix account. + When rendering, these templates are given a `link` variable which contains the link the + user must click in order to validate their email address. + + +## HTML page templates for registration and password reset + +Below are the templates Synapse will look for when generating pages related to +registration and password reset: + +* `password_reset_confirmation.html`: An HTML page that a user will see when they follow + the link in the password reset email. The user will be asked to confirm the action + before their password is reset. + When rendering, this template is given the following variables: + * `sid`: the session ID for the password reset + * `token`: the token for the password reset + * `client_secret`: the client secret for the password reset +* `password_reset_success.html` and `password_reset_failure.html`: HTML pages for success + and failure that a user will see when they confirm the password reset flow using the + page above. + When rendering, `password_reset_success.html` is given no variable, and + `password_reset_failure.html` is given a `failure_reason`, which contains the reason + for the password reset failure. +* `registration_success.html` and `registration_failure.html`: HTML pages for success and + failure that a user will see when they follow the link in an address verification email + sent during registration. + When rendering, `registration_success.html` is given no variable, and + `registration_failure.html` is given a `failure_reason`, which contains the reason + for the registration failure. +* `add_threepid_success.html` and `add_threepid_failure.html`: HTML pages for success and + failure that a user will see when they follow the link in an address verification email + sent when an address is added to a Matrix account. + When rendering, `add_threepid_success.html` is given no variable, and + `add_threepid_failure.html` is given a `failure_reason`, which contains the reason + for the registration failure. + + +## HTML page templates for Single Sign-On (SSO) + +Below are the templates Synapse will look for when generating pages related to SSO: + +* `sso_login_idp_picker.html`: HTML page to prompt the user to choose an + Identity Provider during login. + This is only used if multiple SSO Identity Providers are configured. + When rendering, this template is given the following variables: + * `redirect_url`: the URL that the user will be redirected to after + login. + * `server_name`: the homeserver's name. + * `providers`: a list of available Identity Providers. Each element is + an object with the following attributes: + * `idp_id`: unique identifier for the IdP + * `idp_name`: user-facing name for the IdP + * `idp_icon`: if specified in the IdP config, an MXC URI for an icon + for the IdP + * `idp_brand`: if specified in the IdP config, a textual identifier + for the brand of the IdP + The rendered HTML page should contain a form which submits its results + back as a GET request, with the following query parameters: + * `redirectUrl`: the client redirect URI (ie, the `redirect_url` passed + to the template) + * `idp`: the 'idp_id' of the chosen IDP. +* `sso_auth_account_details.html`: HTML page to prompt new users to enter a + userid and confirm other details. This is only shown if the + SSO implementation (with any `user_mapping_provider`) does not return + a localpart. + When rendering, this template is given the following variables: + * `server_name`: the homeserver's name. + * `idp`: details of the SSO Identity Provider that the user logged in + with: an object with the following attributes: + * `idp_id`: unique identifier for the IdP + * `idp_name`: user-facing name for the IdP + * `idp_icon`: if specified in the IdP config, an MXC URI for an icon + for the IdP + * `idp_brand`: if specified in the IdP config, a textual identifier + for the brand of the IdP + * `user_attributes`: an object containing details about the user that + we received from the IdP. May have the following attributes: + * display_name: the user's display_name + * emails: a list of email addresses + The template should render a form which submits the following fields: + * `username`: the localpart of the user's chosen user id +* `sso_new_user_consent.html`: HTML page allowing the user to consent to the + server's terms and conditions. This is only shown for new users, and only if + `user_consent.require_at_registration` is set. + When rendering, this template is given the following variables: + * `server_name`: the homeserver's name. + * `user_id`: the user's matrix proposed ID. + * `user_profile.display_name`: the user's proposed display name, if any. + * consent_version: the version of the terms that the user will be + shown + * `terms_url`: a link to the page showing the terms. + The template should render a form which submits the following fields: + * `accepted_version`: the version of the terms accepted by the user + (ie, 'consent_version' from the input variables). +* `sso_redirect_confirm.html`: HTML page for a confirmation step before redirecting back + to the client with the login token. + When rendering, this template is given the following variables: + * `redirect_url`: the URL the user is about to be redirected to. + * `display_url`: the same as `redirect_url`, but with the query + parameters stripped. The intention is to have a + human-readable URL to show to users, not to use it as + the final address to redirect to. + * `server_name`: the homeserver's name. + * `new_user`: a boolean indicating whether this is the user's first time + logging in. + * `user_id`: the user's matrix ID. + * `user_profile.avatar_url`: an MXC URI for the user's avatar, if any. + `None` if the user has not set an avatar. + * `user_profile.display_name`: the user's display name. `None` if the user + has not set a display name. +* `sso_auth_confirm.html`: HTML page which notifies the user that they are authenticating + to confirm an operation on their account during the user interactive authentication + process. + When rendering, this template is given the following variables: + * `redirect_url`: the URL the user is about to be redirected to. + * `description`: the operation which the user is being asked to confirm + * `idp`: details of the Identity Provider that we will use to confirm + the user's identity: an object with the following attributes: + * `idp_id`: unique identifier for the IdP + * `idp_name`: user-facing name for the IdP + * `idp_icon`: if specified in the IdP config, an MXC URI for an icon + for the IdP + * `idp_brand`: if specified in the IdP config, a textual identifier + for the brand of the IdP +* `sso_auth_success.html`: HTML page shown after a successful user interactive + authentication session. + Note that this page must include the JavaScript which notifies of a successful + authentication (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback). + This template has no additional variables. +* `sso_auth_bad_user.html`: HTML page shown after a user-interactive authentication + session which does not map correctly onto the expected user. + When rendering, this template is given the following variables: + * `server_name`: the homeserver's name. + * `user_id_to_verify`: the MXID of the user that we are trying to + validate. +* `sso_account_deactivated.html`: HTML page shown during single sign-on if a deactivated + user (according to Synapse's database) attempts to login. + This template has no additional variables. +* `sso_error.html`: HTML page to display to users if something goes wrong during the + OpenID Connect authentication process. + When rendering, this template is given two variables: + * `error`: the technical name of the error + * `error_description`: a human-readable message for the error diff --git a/docs/usage/configuration/user_authentication/password_auth_providers.md b/docs/usage/configuration/user_authentication/password_auth_providers.md new file mode 100644 index 0000000000..dc0dfffa21 --- /dev/null +++ b/docs/usage/configuration/user_authentication/password_auth_providers.md @@ -0,0 +1,129 @@ +

+This page of the Synapse documentation is now deprecated. For up to date +documentation on setting up or writing a password auth provider module, please see +this page. +

+ +# Password auth provider modules + +Password auth providers offer a way for server administrators to +integrate their Synapse installation with an existing authentication +system. + +A password auth provider is a Python class which is dynamically loaded +into Synapse, and provides a number of methods by which it can integrate +with the authentication system. + +This document serves as a reference for those looking to implement their +own password auth providers. Additionally, here is a list of known +password auth provider module implementations: + +* [matrix-synapse-ldap3](https://github.com/matrix-org/matrix-synapse-ldap3/) +* [matrix-synapse-shared-secret-auth](https://github.com/devture/matrix-synapse-shared-secret-auth) +* [matrix-synapse-rest-password-provider](https://github.com/ma1uta/matrix-synapse-rest-password-provider) + +## Required methods + +Password auth provider classes must provide the following methods: + +* `parse_config(config)` + This method is passed the `config` object for this module from the + homeserver configuration file. + + It should perform any appropriate sanity checks on the provided + configuration, and return an object which is then passed into + `__init__`. + + This method should have the `@staticmethod` decoration. + +* `__init__(self, config, account_handler)` + + The constructor is passed the config object returned by + `parse_config`, and a `synapse.module_api.ModuleApi` object which + allows the password provider to check if accounts exist and/or create + new ones. + +## Optional methods + +Password auth provider classes may optionally provide the following methods: + +* `get_db_schema_files(self)` + + This method, if implemented, should return an Iterable of + `(name, stream)` pairs of database schema files. Each file is applied + in turn at initialisation, and a record is then made in the database + so that it is not re-applied on the next start. + +* `get_supported_login_types(self)` + + This method, if implemented, should return a `dict` mapping from a + login type identifier (such as `m.login.password`) to an iterable + giving the fields which must be provided by the user in the submission + to [the `/login` API](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login). + These fields are passed in the `login_dict` dictionary to `check_auth`. + + For example, if a password auth provider wants to implement a custom + login type of `com.example.custom_login`, where the client is expected + to pass the fields `secret1` and `secret2`, the provider should + implement this method and return the following dict: + + ```python + {"com.example.custom_login": ("secret1", "secret2")} + ``` + +* `check_auth(self, username, login_type, login_dict)` + + This method does the real work. If implemented, it + will be called for each login attempt where the login type matches one + of the keys returned by `get_supported_login_types`. + + It is passed the (possibly unqualified) `user` field provided by the client, + the login type, and a dictionary of login secrets passed by the + client. + + The method should return an `Awaitable` object, which resolves + to the canonical `@localpart:domain` user ID if authentication is + successful, and `None` if not. + + Alternatively, the `Awaitable` can resolve to a `(str, func)` tuple, in + which case the second field is a callback which will be called with + the result from the `/login` call (including `access_token`, + `device_id`, etc.) + +* `check_3pid_auth(self, medium, address, password)` + + This method, if implemented, is called when a user attempts to + register or log in with a third party identifier, such as email. It is + passed the medium (ex. "email"), an address (ex. + "") and the user's password. + + The method should return an `Awaitable` object, which resolves + to a `str` containing the user's (canonical) User id if + authentication was successful, and `None` if not. + + As with `check_auth`, the `Awaitable` may alternatively resolve to a + `(user_id, callback)` tuple. + +* `check_password(self, user_id, password)` + + This method provides a simpler interface than + `get_supported_login_types` and `check_auth` for password auth + providers that just want to provide a mechanism for validating + `m.login.password` logins. + + If implemented, it will be called to check logins with an + `m.login.password` login type. It is passed a qualified + `@localpart:domain` user id, and the password provided by the user. + + The method should return an `Awaitable` object, which resolves + to `True` if authentication is successful, and `False` if not. + +* `on_logged_out(self, user_id, device_id, access_token)` + + This method, if implemented, is called when a user logs out. It is + passed the qualified user ID, the ID of the deactivated device (if + any: access tokens are occasionally created without an associated + device ID), and the (now deactivated) access token. + + It may return an `Awaitable` object; the logout request will + wait for the `Awaitable` to complete, but the result is ignored. diff --git a/docs/usage/configuration/user_authentication/single_sign_on/openid.md b/docs/usage/configuration/user_authentication/single_sign_on/openid.md new file mode 100644 index 0000000000..c74e8bda60 --- /dev/null +++ b/docs/usage/configuration/user_authentication/single_sign_on/openid.md @@ -0,0 +1,572 @@ +# Configuring Synapse to authenticate against an OpenID Connect provider + +Synapse can be configured to use an OpenID Connect Provider (OP) for +authentication, instead of its own local password database. + +Any OP should work with Synapse, as long as it supports the authorization code +flow. There are a few options for that: + + - start a local OP. Synapse has been tested with [Hydra][hydra] and + [Dex][dex-idp]. Note that for an OP to work, it should be served under a + secure (HTTPS) origin. A certificate signed with a self-signed, locally + trusted CA should work. In that case, start Synapse with a `SSL_CERT_FILE` + environment variable set to the path of the CA. + + - set up a SaaS OP, like [Google][google-idp], [Auth0][auth0] or + [Okta][okta]. Synapse has been tested with Auth0 and Google. + +It may also be possible to use other OAuth2 providers which provide the +[authorization code grant type](https://tools.ietf.org/html/rfc6749#section-4.1), +such as [Github][github-idp]. + +[google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect +[auth0]: https://auth0.com/ +[authentik]: https://goauthentik.io/ +[lemonldap]: https://lemonldap-ng.org/ +[okta]: https://www.okta.com/ +[dex-idp]: https://github.com/dexidp/dex +[keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols +[hydra]: https://www.ory.sh/docs/hydra/ +[github-idp]: https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps + +## Preparing Synapse + +The OpenID integration in Synapse uses the +[`authlib`](https://pypi.org/project/Authlib/) library, which must be installed +as follows: + + * The relevant libraries are included in the Docker images and Debian packages + provided by `matrix.org` so no further action is needed. + + * If you installed Synapse into a virtualenv, run `/path/to/env/bin/pip + install matrix-synapse[oidc]` to install the necessary dependencies. + + * For other installation mechanisms, see the documentation provided by the + maintainer. + +To enable the OpenID integration, you should then add a section to the `oidc_providers` +setting in your configuration file (or uncomment one of the existing examples). +See [sample_config.yaml](./sample_config.yaml) for some sample settings, as well as +the text below for example configurations for specific providers. + +## Sample configs + +Here are a few configs for providers that should work with Synapse. + +### Microsoft Azure Active Directory +Azure AD can act as an OpenID Connect Provider. Register a new application under +*App registrations* in the Azure AD management console. The RedirectURI for your +application should point to your matrix server: +`[synapse public baseurl]/_synapse/client/oidc/callback` + +Go to *Certificates & secrets* and register a new client secret. Make note of your +Directory (tenant) ID as it will be used in the Azure links. +Edit your Synapse config file and change the `oidc_config` section: + +```yaml +oidc_providers: + - idp_id: microsoft + idp_name: Microsoft + issuer: "https://login.microsoftonline.com//v2.0" + client_id: "" + client_secret: "" + scopes: ["openid", "profile"] + authorization_endpoint: "https://login.microsoftonline.com//oauth2/v2.0/authorize" + token_endpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" + userinfo_endpoint: "https://graph.microsoft.com/oidc/userinfo" + + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username.split('@')[0] }}" + display_name_template: "{{ user.name }}" +``` + +### Dex + +[Dex][dex-idp] is a simple, open-source, certified OpenID Connect Provider. +Although it is designed to help building a full-blown provider with an +external database, it can be configured with static passwords in a config file. + +Follow the [Getting Started guide](https://dexidp.io/docs/getting-started/) +to install Dex. + +Edit `examples/config-dev.yaml` config file from the Dex repo to add a client: + +```yaml +staticClients: +- id: synapse + secret: secret + redirectURIs: + - '[synapse public baseurl]/_synapse/client/oidc/callback' + name: 'Synapse' +``` + +Run with `dex serve examples/config-dev.yaml`. + +Synapse config: + +```yaml +oidc_providers: + - idp_id: dex + idp_name: "My Dex server" + skip_verification: true # This is needed as Dex is served on an insecure endpoint + issuer: "http://127.0.0.1:5556/dex" + client_id: "synapse" + client_secret: "secret" + scopes: ["openid", "profile"] + user_mapping_provider: + config: + localpart_template: "{{ user.name }}" + display_name_template: "{{ user.name|capitalize }}" +``` +### Keycloak + +[Keycloak][keycloak-idp] is an opensource IdP maintained by Red Hat. + +Follow the [Getting Started Guide](https://www.keycloak.org/getting-started) to install Keycloak and set up a realm. + +1. Click `Clients` in the sidebar and click `Create` + +2. Fill in the fields as below: + +| Field | Value | +|-----------|-----------| +| Client ID | `synapse` | +| Client Protocol | `openid-connect` | + +3. Click `Save` +4. Fill in the fields as below: + +| Field | Value | +|-----------|-----------| +| Client ID | `synapse` | +| Enabled | `On` | +| Client Protocol | `openid-connect` | +| Access Type | `confidential` | +| Valid Redirect URIs | `[synapse public baseurl]/_synapse/client/oidc/callback` | + +5. Click `Save` +6. On the Credentials tab, update the fields: + +| Field | Value | +|-------|-------| +| Client Authenticator | `Client ID and Secret` | + +7. Click `Regenerate Secret` +8. Copy Secret + +```yaml +oidc_providers: + - idp_id: keycloak + idp_name: "My KeyCloak server" + issuer: "https://127.0.0.1:8443/auth/realms/{realm_name}" + client_id: "synapse" + client_secret: "copy secret generated from above" + scopes: ["openid", "profile"] + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}" + display_name_template: "{{ user.name }}" +``` +### Auth0 + +[Auth0][auth0] is a hosted SaaS IdP solution. + +1. Create a regular web application for Synapse +2. Set the Allowed Callback URLs to `[synapse public baseurl]/_synapse/client/oidc/callback` +3. Add a rule to add the `preferred_username` claim. +
+ Code sample + + ```js + function addPersistenceAttribute(user, context, callback) { + user.user_metadata = user.user_metadata || {}; + user.user_metadata.preferred_username = user.user_metadata.preferred_username || user.user_id; + context.idToken.preferred_username = user.user_metadata.preferred_username; + + auth0.users.updateUserMetadata(user.user_id, user.user_metadata) + .then(function(){ + callback(null, user, context); + }) + .catch(function(err){ + callback(err); + }); + } + ``` +
+ +Synapse config: + +```yaml +oidc_providers: + - idp_id: auth0 + idp_name: Auth0 + issuer: "https://your-tier.eu.auth0.com/" # TO BE FILLED + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + scopes: ["openid", "profile"] + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}" + display_name_template: "{{ user.name }}" +``` + +### Authentik + +[Authentik][authentik] is an open-source IdP solution. + +1. Create a provider in Authentik, with type OAuth2/OpenID. +2. The parameters are: +- Client Type: Confidential +- JWT Algorithm: RS256 +- Scopes: OpenID, Email and Profile +- RSA Key: Select any available key +- Redirect URIs: `[synapse public baseurl]/_synapse/client/oidc/callback` +3. Create an application for synapse in Authentik and link it to the provider. +4. Note the slug of your application, Client ID and Client Secret. + +Synapse config: +```yaml +oidc_providers: + - idp_id: authentik + idp_name: authentik + discover: true + issuer: "https://your.authentik.example.org/application/o/your-app-slug/" # TO BE FILLED: domain and slug + client_id: "your client id" # TO BE FILLED + client_secret: "your client secret" # TO BE FILLED + scopes: + - "openid" + - "profile" + - "email" + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}}" + display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize. +``` + +### LemonLDAP + +[LemonLDAP::NG][lemonldap] is an open-source IdP solution. + +1. Create an OpenID Connect Relying Parties in LemonLDAP::NG +2. The parameters are: +- Client ID under the basic menu of the new Relying Parties (`Options > Basic > + Client ID`) +- Client secret (`Options > Basic > Client secret`) +- JWT Algorithm: RS256 within the security menu of the new Relying Parties + (`Options > Security > ID Token signature algorithm` and `Options > Security > + Access Token signature algorithm`) +- Scopes: OpenID, Email and Profile +- Allowed redirection addresses for login (`Options > Basic > Allowed + redirection addresses for login` ) : + `[synapse public baseurl]/_synapse/client/oidc/callback` + +Synapse config: +```yaml +oidc_providers: + - idp_id: lemonldap + idp_name: lemonldap + discover: true + issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain + client_id: "your client id" # TO BE FILLED + client_secret: "your client secret" # TO BE FILLED + scopes: + - "openid" + - "profile" + - "email" + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}}" + # TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter. + display_name_template: "{{ user.preferred_username|capitalize }}" +``` + +### GitHub + +[GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but +just a regular OAuth2 provider. + +The [`/user` API endpoint](https://developer.github.com/v3/users/#get-the-authenticated-user) +can be used to retrieve information on the authenticated user. As the Synapse +login mechanism needs an attribute to uniquely identify users, and that endpoint +does not return a `sub` property, an alternative `subject_claim` has to be set. + +1. Create a new OAuth application: https://github.com/settings/applications/new. +2. Set the callback URL to `[synapse public baseurl]/_synapse/client/oidc/callback`. + +Synapse config: + +```yaml +oidc_providers: + - idp_id: github + idp_name: Github + idp_brand: "github" # optional: styling hint for clients + discover: false + issuer: "https://github.com/" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + authorization_endpoint: "https://github.com/login/oauth/authorize" + token_endpoint: "https://github.com/login/oauth/access_token" + userinfo_endpoint: "https://api.github.com/user" + scopes: ["read:user"] + user_mapping_provider: + config: + subject_claim: "id" + localpart_template: "{{ user.login }}" + display_name_template: "{{ user.name }}" +``` + +### Google + +[Google][google-idp] is an OpenID certified authentication and authorisation provider. + +1. Set up a project in the Google API Console (see + https://developers.google.com/identity/protocols/oauth2/openid-connect#appsetup). +2. Add an "OAuth Client ID" for a Web Application under "Credentials". +3. Copy the Client ID and Client Secret, and add the following to your synapse config: + ```yaml + oidc_providers: + - idp_id: google + idp_name: Google + idp_brand: "google" # optional: styling hint for clients + issuer: "https://accounts.google.com/" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + scopes: ["openid", "profile"] + user_mapping_provider: + config: + localpart_template: "{{ user.given_name|lower }}" + display_name_template: "{{ user.name }}" + ``` +4. Back in the Google console, add this Authorized redirect URI: `[synapse + public baseurl]/_synapse/client/oidc/callback`. + +### Twitch + +1. Setup a developer account on [Twitch](https://dev.twitch.tv/) +2. Obtain the OAuth 2.0 credentials by [creating an app](https://dev.twitch.tv/console/apps/) +3. Add this OAuth Redirect URL: `[synapse public baseurl]/_synapse/client/oidc/callback` + +Synapse config: + +```yaml +oidc_providers: + - idp_id: twitch + idp_name: Twitch + issuer: "https://id.twitch.tv/oauth2/" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + client_auth_method: "client_secret_post" + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}" + display_name_template: "{{ user.name }}" +``` + +### GitLab + +1. Create a [new application](https://gitlab.com/profile/applications). +2. Add the `read_user` and `openid` scopes. +3. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback` + +Synapse config: + +```yaml +oidc_providers: + - idp_id: gitlab + idp_name: Gitlab + idp_brand: "gitlab" # optional: styling hint for clients + issuer: "https://gitlab.com/" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + client_auth_method: "client_secret_post" + scopes: ["openid", "read_user"] + user_profile_method: "userinfo_endpoint" + user_mapping_provider: + config: + localpart_template: '{{ user.nickname }}' + display_name_template: '{{ user.name }}' +``` + +### Facebook + +Like Github, Facebook provide a custom OAuth2 API rather than an OIDC-compliant +one so requires a little more configuration. + +0. You will need a Facebook developer account. You can register for one + [here](https://developers.facebook.com/async/registration/). +1. On the [apps](https://developers.facebook.com/apps/) page of the developer + console, "Create App", and choose "Build Connected Experiences". +2. Once the app is created, add "Facebook Login" and choose "Web". You don't + need to go through the whole form here. +3. In the left-hand menu, open "Products"/"Facebook Login"/"Settings". + * Add `[synapse public baseurl]/_synapse/client/oidc/callback` as an OAuth Redirect + URL. +4. In the left-hand menu, open "Settings/Basic". Here you can copy the "App ID" + and "App Secret" for use below. + +Synapse config: + +```yaml + - idp_id: facebook + idp_name: Facebook + idp_brand: "facebook" # optional: styling hint for clients + discover: false + issuer: "https://facebook.com" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + scopes: ["openid", "email"] + authorization_endpoint: https://facebook.com/dialog/oauth + token_endpoint: https://graph.facebook.com/v9.0/oauth/access_token + user_profile_method: "userinfo_endpoint" + userinfo_endpoint: "https://graph.facebook.com/v9.0/me?fields=id,name,email,picture" + user_mapping_provider: + config: + subject_claim: "id" + display_name_template: "{{ user.name }}" +``` + +Relevant documents: + * https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow + * Using Facebook's Graph API: https://developers.facebook.com/docs/graph-api/using-graph-api/ + * Reference to the User endpoint: https://developers.facebook.com/docs/graph-api/reference/user + +### Gitea + +Gitea is, like Github, not an OpenID provider, but just an OAuth2 provider. + +The [`/user` API endpoint](https://try.gitea.io/api/swagger#/user/userGetCurrent) +can be used to retrieve information on the authenticated user. As the Synapse +login mechanism needs an attribute to uniquely identify users, and that endpoint +does not return a `sub` property, an alternative `subject_claim` has to be set. + +1. Create a new application. +2. Add this Callback URL: `[synapse public baseurl]/_synapse/client/oidc/callback` + +Synapse config: + +```yaml +oidc_providers: + - idp_id: gitea + idp_name: Gitea + discover: false + issuer: "https://your-gitea.com/" + client_id: "your-client-id" # TO BE FILLED + client_secret: "your-client-secret" # TO BE FILLED + client_auth_method: client_secret_post + scopes: [] # Gitea doesn't support Scopes + authorization_endpoint: "https://your-gitea.com/login/oauth/authorize" + token_endpoint: "https://your-gitea.com/login/oauth/access_token" + userinfo_endpoint: "https://your-gitea.com/api/v1/user" + user_mapping_provider: + config: + subject_claim: "id" + localpart_template: "{{ user.login }}" + display_name_template: "{{ user.full_name }}" +``` + +### XWiki + +Install [OpenID Connect Provider](https://extensions.xwiki.org/xwiki/bin/view/Extension/OpenID%20Connect/OpenID%20Connect%20Provider/) extension in your [XWiki](https://www.xwiki.org) instance. + +Synapse config: + +```yaml +oidc_providers: + - idp_id: xwiki + idp_name: "XWiki" + issuer: "https://myxwikihost/xwiki/oidc/" + client_id: "your-client-id" # TO BE FILLED + client_auth_method: none + scopes: ["openid", "profile"] + user_profile_method: "userinfo_endpoint" + user_mapping_provider: + config: + localpart_template: "{{ user.preferred_username }}" + display_name_template: "{{ user.name }}" +``` + +### Apple + +Configuring "Sign in with Apple" (SiWA) requires an Apple Developer account. + +You will need to create a new "Services ID" for SiWA, and create and download a +private key with "SiWA" enabled. + +As well as the private key file, you will need: + * Client ID: the "identifier" you gave the "Services ID" + * Team ID: a 10-character ID associated with your developer account. + * Key ID: the 10-character identifier for the key. + +https://help.apple.com/developer-account/?lang=en#/dev77c875b7e has more +documentation on setting up SiWA. + +The synapse config will look like this: + +```yaml + - idp_id: apple + idp_name: Apple + issuer: "https://appleid.apple.com" + client_id: "your-client-id" # Set to the "identifier" for your "ServicesID" + client_auth_method: "client_secret_post" + client_secret_jwt_key: + key_file: "/path/to/AuthKey_KEYIDCODE.p8" # point to your key file + jwt_header: + alg: ES256 + kid: "KEYIDCODE" # Set to the 10-char Key ID + jwt_payload: + iss: TEAMIDCODE # Set to the 10-char Team ID + scopes: ["name", "email", "openid"] + authorization_endpoint: https://appleid.apple.com/auth/authorize?response_mode=form_post + user_mapping_provider: + config: + email_template: "{{ user.email }}" +``` + +## Django OAuth Toolkit + +[django-oauth-toolkit](https://github.com/jazzband/django-oauth-toolkit) is a +Django application providing out of the box all the endpoints, data and logic +needed to add OAuth2 capabilities to your Django projects. It supports +[OpenID Connect too](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html). + +Configuration on Django's side: + +1. Add an application: https://example.com/admin/oauth2_provider/application/add/ and choose parameters like this: +* `Redirect uris`: https://synapse.example.com/_synapse/client/oidc/callback +* `Client type`: `Confidential` +* `Authorization grant type`: `Authorization code` +* `Algorithm`: `HMAC with SHA-2 256` +2. You can [customize the claims](https://django-oauth-toolkit.readthedocs.io/en/latest/oidc.html#customizing-the-oidc-responses) Django gives to synapse (optional): +
+ Code sample + + ```python + class CustomOAuth2Validator(OAuth2Validator): + + def get_additional_claims(self, request): + return { + "sub": request.user.email, + "email": request.user.email, + "first_name": request.user.first_name, + "last_name": request.user.last_name, + } + ``` +
+Your synapse config is then: + +```yaml +oidc_providers: + - idp_id: django_example + idp_name: "Django Example" + issuer: "https://example.com/o/" + client_id: "your-client-id" # CHANGE ME + client_secret: "your-client-secret" # CHANGE ME + scopes: ["openid"] + user_profile_method: "userinfo_endpoint" # needed because oauth-toolkit does not include user information in the authorization response + user_mapping_provider: + config: + localpart_template: "{{ user.email.split('@')[0] }}" + display_name_template: "{{ user.first_name }} {{ user.last_name }}" + email_template: "{{ user.email }}" +``` diff --git a/docs/usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md b/docs/usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md new file mode 100644 index 0000000000..7a407012e0 --- /dev/null +++ b/docs/usage/configuration/user_authentication/single_sign_on/sso_mapping_providers.md @@ -0,0 +1,197 @@ +# SSO Mapping Providers + +A mapping provider is a Python class (loaded via a Python module) that +works out how to map attributes of a SSO response to Matrix-specific +user attributes. Details such as user ID localpart, displayname, and even avatar +URLs are all things that can be mapped from talking to a SSO service. + +As an example, a SSO service may return the email address +"john.smith@example.com" for a user, whereas Synapse will need to figure out how +to turn that into a displayname when creating a Matrix user for this individual. +It may choose `John Smith`, or `Smith, John [Example.com]` or any number of +variations. As each Synapse configuration may want something different, this is +where SAML mapping providers come into play. + +SSO mapping providers are currently supported for OpenID and SAML SSO +configurations. Please see the details below for how to implement your own. + +It is up to the mapping provider whether the user should be assigned a predefined +Matrix ID based on the SSO attributes, or if the user should be allowed to +choose their own username. + +In the first case - where users are automatically allocated a Matrix ID - it is +the responsibility of the mapping provider to normalise the SSO attributes and +map them to a valid Matrix ID. The [specification for Matrix +IDs](https://matrix.org/docs/spec/appendices#user-identifiers) has some +information about what is considered valid. + +If the mapping provider does not assign a Matrix ID, then Synapse will +automatically serve an HTML page allowing the user to pick their own username. + +External mapping providers are provided to Synapse in the form of an external +Python module. You can retrieve this module from [PyPI](https://pypi.org) or elsewhere, +but it must be importable via Synapse (e.g. it must be in the same virtualenv +as Synapse). The Synapse config is then modified to point to the mapping provider +(and optionally provide additional configuration for it). + +## OpenID Mapping Providers + +The OpenID mapping provider can be customized by editing the +`oidc_config.user_mapping_provider.module` config option. + +`oidc_config.user_mapping_provider.config` allows you to provide custom +configuration options to the module. Check with the module's documentation for +what options it provides (if any). The options listed by default are for the +user mapping provider built in to Synapse. If using a custom module, you should +comment these options out and use those specified by the module instead. + +### Building a Custom OpenID Mapping Provider + +A custom mapping provider must specify the following methods: + +* `__init__(self, parsed_config)` + - Arguments: + - `parsed_config` - A configuration object that is the return value of the + `parse_config` method. You should set any configuration options needed by + the module here. +* `parse_config(config)` + - This method should have the `@staticmethod` decoration. + - Arguments: + - `config` - A `dict` representing the parsed content of the + `oidc_config.user_mapping_provider.config` homeserver config option. + Runs on homeserver startup. Providers should extract and validate + any option values they need here. + - Whatever is returned will be passed back to the user mapping provider module's + `__init__` method during construction. +* `get_remote_user_id(self, userinfo)` + - Arguments: + - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user + information from. + - This method must return a string, which is the unique, immutable identifier + for the user. Commonly the `sub` claim of the response. +* `map_user_attributes(self, userinfo, token, failures)` + - This method must be async. + - Arguments: + - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user + information from. + - `token` - A dictionary which includes information necessary to make + further requests to the OpenID provider. + - `failures` - An `int` that represents the amount of times the returned + mxid localpart mapping has failed. This should be used + to create a deduplicated mxid localpart which should be + returned instead. For example, if this method returns + `john.doe` as the value of `localpart` in the returned + dict, and that is already taken on the homeserver, this + method will be called again with the same parameters but + with failures=1. The method should then return a different + `localpart` value, such as `john.doe1`. + - Returns a dictionary with two keys: + - `localpart`: A string, used to generate the Matrix ID. If this is + `None`, the user is prompted to pick their own username. This is only used + during a user's first login. Once a localpart has been associated with a + remote user ID (see `get_remote_user_id`) it cannot be updated. + - `displayname`: An optional string, the display name for the user. +* `get_extra_attributes(self, userinfo, token)` + - This method must be async. + - Arguments: + - `userinfo` - A `authlib.oidc.core.claims.UserInfo` object to extract user + information from. + - `token` - A dictionary which includes information necessary to make + further requests to the OpenID provider. + - Returns a dictionary that is suitable to be serialized to JSON. This + will be returned as part of the response during a successful login. + + Note that care should be taken to not overwrite any of the parameters + usually returned as part of the [login response](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login). + +### Default OpenID Mapping Provider + +Synapse has a built-in OpenID mapping provider if a custom provider isn't +specified in the config. It is located at +[`synapse.handlers.oidc.JinjaOidcMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/oidc.py). + +## SAML Mapping Providers + +The SAML mapping provider can be customized by editing the +`saml2_config.user_mapping_provider.module` config option. + +`saml2_config.user_mapping_provider.config` allows you to provide custom +configuration options to the module. Check with the module's documentation for +what options it provides (if any). The options listed by default are for the +user mapping provider built in to Synapse. If using a custom module, you should +comment these options out and use those specified by the module instead. + +### Building a Custom SAML Mapping Provider + +A custom mapping provider must specify the following methods: + +* `__init__(self, parsed_config, module_api)` + - Arguments: + - `parsed_config` - A configuration object that is the return value of the + `parse_config` method. You should set any configuration options needed by + the module here. + - `module_api` - a `synapse.module_api.ModuleApi` object which provides the + stable API available for extension modules. +* `parse_config(config)` + - This method should have the `@staticmethod` decoration. + - Arguments: + - `config` - A `dict` representing the parsed content of the + `saml_config.user_mapping_provider.config` homeserver config option. + Runs on homeserver startup. Providers should extract and validate + any option values they need here. + - Whatever is returned will be passed back to the user mapping provider module's + `__init__` method during construction. +* `get_saml_attributes(config)` + - This method should have the `@staticmethod` decoration. + - Arguments: + - `config` - A object resulting from a call to `parse_config`. + - Returns a tuple of two sets. The first set equates to the SAML auth + response attributes that are required for the module to function, whereas + the second set consists of those attributes which can be used if available, + but are not necessary. +* `get_remote_user_id(self, saml_response, client_redirect_url)` + - Arguments: + - `saml_response` - A `saml2.response.AuthnResponse` object to extract user + information from. + - `client_redirect_url` - A string, the URL that the client will be + redirected to. + - This method must return a string, which is the unique, immutable identifier + for the user. Commonly the `uid` claim of the response. +* `saml_response_to_user_attributes(self, saml_response, failures, client_redirect_url)` + - Arguments: + - `saml_response` - A `saml2.response.AuthnResponse` object to extract user + information from. + - `failures` - An `int` that represents the amount of times the returned + mxid localpart mapping has failed. This should be used + to create a deduplicated mxid localpart which should be + returned instead. For example, if this method returns + `john.doe` as the value of `mxid_localpart` in the returned + dict, and that is already taken on the homeserver, this + method will be called again with the same parameters but + with failures=1. The method should then return a different + `mxid_localpart` value, such as `john.doe1`. + - `client_redirect_url` - A string, the URL that the client will be + redirected to. + - This method must return a dictionary, which will then be used by Synapse + to build a new user. The following keys are allowed: + * `mxid_localpart` - A string, the mxid localpart of the new user. If this is + `None`, the user is prompted to pick their own username. This is only used + during a user's first login. Once a localpart has been associated with a + remote user ID (see `get_remote_user_id`) it cannot be updated. + * `displayname` - The displayname of the new user. If not provided, will default to + the value of `mxid_localpart`. + * `emails` - A list of emails for the new user. If not provided, will + default to an empty list. + + Alternatively it can raise a `synapse.api.errors.RedirectException` to + redirect the user to another page. This is useful to prompt the user for + additional information, e.g. if you want them to provide their own username. + It is the responsibility of the mapping provider to either redirect back + to `client_redirect_url` (including any additional information) or to + complete registration using methods from the `ModuleApi`. + +### Default SAML Mapping Provider + +Synapse has a built-in SAML mapping provider if a custom provider isn't +specified in the config. It is located at +[`synapse.handlers.saml.DefaultSamlMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/saml.py). diff --git a/docs/usage/configuration/user_directory.md b/docs/usage/configuration/user_directory.md new file mode 100644 index 0000000000..c4794b04cf --- /dev/null +++ b/docs/usage/configuration/user_directory.md @@ -0,0 +1,49 @@ +User Directory API Implementation +================================= + +The user directory is currently maintained based on the 'visible' users +on this particular server - i.e. ones which your account shares a room with, or +who are present in a publicly viewable room present on the server. + +The directory info is stored in various tables, which can (typically after +DB corruption) get stale or out of sync. If this happens, for now the +solution to fix it is to use the [admin API](usage/administration/admin_api/background_updates.md#run) +and execute the job `regenerate_directory`. This should then start a background task to +flush the current tables and regenerate the directory. + +Data model +---------- + +There are five relevant tables that collectively form the "user directory". +Three of them track a master list of all the users we could search for. +The last two (collectively called the "search tables") track who can +see who. + +From all of these tables we exclude three types of local user: + - support users + - appservice users + - deactivated users + +* `user_directory`. This contains the user_id, display name and avatar we'll + return when you search the directory. + - Because there's only one directory entry per user, it's important that we only + ever put publicly visible names here. Otherwise we might leak a private + nickname or avatar used in a private room. + - Indexed on rooms. Indexed on users. + +* `user_directory_search`. To be joined to `user_directory`. It contains an extra + column that enables full text search based on user ids and display names. + Different schemas for SQLite and Postgres with different code paths to match. + - Indexed on the full text search data. Indexed on users. + +* `user_directory_stream_pos`. When the initial background update to populate + the directory is complete, we record a stream position here. This indicates + that synapse should now listen for room changes and incrementally update + the directory where necessary. + +* `users_in_public_rooms`. Contains associations between users and the public rooms they're in. + Used to determine which users are in public rooms and should be publicly visible in the directory. + +* `users_who_share_private_rooms`. Rows are triples `(L, M, room id)` where `L` + is a local user and `M` is a local or remote user. `L` and `M` should be + different, but this isn't enforced by a constraint. diff --git a/docs/usage/configuration/workers/README.md b/docs/usage/configuration/workers/README.md new file mode 100644 index 0000000000..17c8bfeef6 --- /dev/null +++ b/docs/usage/configuration/workers/README.md @@ -0,0 +1,560 @@ +# Scaling synapse via workers + +For small instances it recommended to run Synapse in the default monolith mode. +For larger instances where performance is a concern it can be helpful to split +out functionality into multiple separate python processes. These processes are +called 'workers', and are (eventually) intended to scale horizontally +independently. + +Synapse's worker support is under active development and subject to change as +we attempt to rapidly scale ever larger Synapse instances. However we are +documenting it here to help admins needing a highly scalable Synapse instance +similar to the one running `matrix.org`. + +All processes continue to share the same database instance, and as such, +workers only work with PostgreSQL-based Synapse deployments. SQLite should only +be used for demo purposes and any admin considering workers should already be +running PostgreSQL. + +See also [Matrix.org blog post](https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability) +for a higher level overview. + +## Main process/worker communication + +The processes communicate with each other via a Synapse-specific protocol called +'replication' (analogous to MySQL- or Postgres-style database replication) which +feeds streams of newly written data between processes so they can be kept in +sync with the database state. + +When configured to do so, Synapse uses a +[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication +stream between all configured Synapse processes. Additionally, processes may +make HTTP requests to each other, primarily for operations which need to wait +for a reply ─ such as sending an event. + +Redis support was added in v1.13.0 with it becoming the recommended method in +v1.18.0. It replaced the old direct TCP connections (which is deprecated as of +v1.18.0) to the main process. With Redis, rather than all the workers connecting +to the main process, all the workers and the main process connect to Redis, +which relays replication commands between processes. This can give a significant +cpu saving on the main process and will be a prerequisite for upcoming +performance improvements. + +If Redis support is enabled Synapse will use it as a shared cache, as well as a +pub/sub mechanism. + +See the [Architectural diagram](#architectural-diagram) section at the end for +a visualisation of what this looks like. + + +## Setting up workers + +A Redis server is required to manage the communication between the processes. +The Redis server should be installed following the normal procedure for your +distribution (e.g. `apt install redis-server` on Debian). It is safe to use an +existing Redis deployment if you have one. + +Once installed, check that Redis is running and accessible from the host running +Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing +a response of `+PONG`. + +The appropriate dependencies must also be installed for Synapse. If using a +virtualenv, these can be installed with: + +```sh +pip install "matrix-synapse[redis]" +``` + +Note that these dependencies are included when synapse is installed with `pip +install matrix-synapse[all]`. They are also included in the debian packages from +`matrix.org` and in the docker images at +https://hub.docker.com/r/matrixdotorg/synapse/. + +To make effective use of the workers, you will need to configure an HTTP +reverse-proxy such as nginx or haproxy, which will direct incoming requests to +the correct worker, or to the main synapse instance. See +[the reverse proxy documentation](reverse_proxy.md) for information on setting up a reverse +proxy. + +When using workers, each worker process has its own configuration file which +contains settings specific to that worker, such as the HTTP listener that it +provides (if any), logging configuration, etc. + +Normally, the worker processes are configured to read from a shared +configuration file as well as the worker-specific configuration files. This +makes it easier to keep common configuration settings synchronised across all +the processes. + +The main process is somewhat special in this respect: it does not normally +need its own configuration file and can take all of its configuration from the +shared configuration file. + + +### Shared configuration + +Normally, only a couple of changes are needed to make an existing configuration +file suitable for use with workers. First, you need to enable an "HTTP replication +listener" for the main process; and secondly, you need to enable redis-based +replication. Optionally, a shared secret can be used to authenticate HTTP +traffic between workers. For example: + + +```yaml +# extend the existing `listeners` section. This defines the ports that the +# main process will listen on. +listeners: + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] + +# Add a random shared secret to authenticate traffic. +worker_replication_secret: "" + +redis: + enabled: true +``` + +See the sample config for the full documentation of each option. + +Under **no circumstances** should the replication listener be exposed to the +public internet; it has no authentication and is unencrypted. + + +### Worker configuration + +In the config file for each worker, you must specify the type of worker +application (`worker_app`), and you should specify a unique name for the worker +(`worker_name`). The currently available worker applications are listed below. +You must also specify the HTTP replication endpoint that it should talk to on +the main synapse process. `worker_replication_host` should specify the host of +the main synapse and `worker_replication_http_port` should point to the HTTP +replication port. If the worker will handle HTTP requests then the +`worker_listeners` option should be set with a `http` listener, in the same way +as the `listeners` option in the shared config. + +For example: + +```yaml +worker_app: synapse.app.generic_worker +worker_name: worker1 + +# The replication listener on the main synapse process. +worker_replication_host: 127.0.0.1 +worker_replication_http_port: 9093 + +worker_listeners: + - type: http + port: 8083 + resources: + - names: + - client + - federation + +worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml +``` + +...is a full configuration for a generic worker instance, which will expose a +plain HTTP endpoint on port 8083 separately serving various endpoints, e.g. +`/sync`, which are listed below. + +Obviously you should configure your reverse-proxy to route the relevant +endpoints to the worker (`localhost:8083` in the above example). + + +### Running Synapse with workers + +Finally, you need to start your worker processes. This can be done with either +`synctl` or your distribution's preferred service manager such as `systemd`. We +recommend the use of `systemd` where available: for information on setting up +`systemd` to start synapse workers, see +[Systemd with Workers](systemd-with-workers). To use `synctl`, see +[Using synctl with Workers](synctl_workers.md). + + +## Available worker applications + +### `synapse.app.generic_worker` + +This worker can handle API requests matching the following regular +expressions: + + # Sync requests + ^/_matrix/client/(v2_alpha|r0|v3)/sync$ + ^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$ + ^/_matrix/client/(api/v1|r0|v3)/initialSync$ + ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ + + # Federation requests + ^/_matrix/federation/v1/event/ + ^/_matrix/federation/v1/state/ + ^/_matrix/federation/v1/state_ids/ + ^/_matrix/federation/v1/backfill/ + ^/_matrix/federation/v1/get_missing_events/ + ^/_matrix/federation/v1/publicRooms + ^/_matrix/federation/v1/query/ + ^/_matrix/federation/v1/make_join/ + ^/_matrix/federation/v1/make_leave/ + ^/_matrix/federation/v1/send_join/ + ^/_matrix/federation/v2/send_join/ + ^/_matrix/federation/v1/send_leave/ + ^/_matrix/federation/v2/send_leave/ + ^/_matrix/federation/v1/invite/ + ^/_matrix/federation/v2/invite/ + ^/_matrix/federation/v1/query_auth/ + ^/_matrix/federation/v1/event_auth/ + ^/_matrix/federation/v1/exchange_third_party_invite/ + ^/_matrix/federation/v1/user/devices/ + ^/_matrix/federation/v1/get_groups_publicised$ + ^/_matrix/key/v2/query + ^/_matrix/federation/unstable/org.matrix.msc2946/spaces/ + ^/_matrix/federation/unstable/org.matrix.msc2946/hierarchy/ + + # Inbound federation transaction request + ^/_matrix/federation/v1/send/ + + # Client API requests + ^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$ + ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/spaces$ + ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/hierarchy$ + ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/account/3pid$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/devices$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/query$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/changes$ + ^/_matrix/client/versions$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_groups$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups/ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/ + ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/search$ + + # Registration/login requests + ^/_matrix/client/(api/v1|r0|v3|unstable)/login$ + ^/_matrix/client/(r0|v3|unstable)/register$ + ^/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity$ + + # Event sending requests + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/ + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ + ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ + ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ + + +Additionally, the following REST endpoints can be handled for GET requests: + + ^/_matrix/federation/v1/groups/ + +Pagination requests can also be handled, but all requests for a given +room must be routed to the same instance. Additionally, care must be taken to +ensure that the purge history admin API is not used while pagination requests +for the room are in flight: + + ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$ + +Additionally, the following endpoints should be included if Synapse is configured +to use SSO (you only need to include the ones for whichever SSO provider you're +using): + + # for all SSO providers + ^/_matrix/client/(api/v1|r0|v3|unstable)/login/sso/redirect + ^/_synapse/client/pick_idp$ + ^/_synapse/client/pick_username + ^/_synapse/client/new_user_consent$ + ^/_synapse/client/sso_register$ + + # OpenID Connect requests. + ^/_synapse/client/oidc/callback$ + + # SAML requests. + ^/_synapse/client/saml2/authn_response$ + + # CAS requests. + ^/_matrix/client/(api/v1|r0|v3|unstable)/login/cas/ticket$ + +Ensure that all SSO logins go to a single process. +For multiple workers not handling the SSO endpoints properly, see +[#7530](https://github.com/matrix-org/synapse/issues/7530) and +[#9427](https://github.com/matrix-org/synapse/issues/9427). + +Note that a HTTP listener with `client` and `federation` resources must be +configured in the `worker_listeners` option in the worker config. + +#### Load balancing + +It is possible to run multiple instances of this worker app, with incoming requests +being load-balanced between them by the reverse-proxy. However, different endpoints +have different characteristics and so admins +may wish to run multiple groups of workers handling different endpoints so that +load balancing can be done in different ways. + +For `/sync` and `/initialSync` requests it will be more efficient if all +requests from a particular user are routed to a single instance. Extracting a +user ID from the access token or `Authorization` header is currently left as an +exercise for the reader. Admins may additionally wish to separate out `/sync` +requests that have a `since` query parameter from those that don't (and +`/initialSync`), as requests that don't are known as "initial sync" that happens +when a user logs in on a new device and can be *very* resource intensive, so +isolating these requests will stop them from interfering with other users ongoing +syncs. + +Federation and client requests can be balanced via simple round robin. + +The inbound federation transaction request `^/_matrix/federation/v1/send/` +should be balanced by source IP so that transactions from the same remote server +go to the same process. + +Registration/login requests can be handled separately purely to help ensure that +unexpected load doesn't affect new logins and sign ups. + +Finally, event sending requests can be balanced by the room ID in the URI (or +the full URI, or even just round robin), the room ID is the path component after +`/rooms/`. If there is a large bridge connected that is sending or may send lots +of events, then a dedicated set of workers can be provisioned to limit the +effects of bursts of events from that bridge on events sent by normal users. + +#### Stream writers + +Additionally, there is *experimental* support for moving writing of specific +streams (such as events) off of the main process to a particular worker. (This +is only supported with Redis-based replication.) + +Currently supported streams are `events` and `typing`. + +To enable this, the worker must have a HTTP replication listener configured, +have a `worker_name` and be listed in the `instance_map` config. For example to +move event persistence off to a dedicated worker, the shared configuration would +include: + +```yaml +instance_map: + event_persister1: + host: localhost + port: 8034 + +stream_writers: + events: event_persister1 +``` + +The `events` stream also experimentally supports having multiple writers, where +work is sharded between them by room ID. Note that you *must* restart all worker +instances when adding or removing event persisters. An example `stream_writers` +configuration with multiple writers: + +```yaml +stream_writers: + events: + - event_persister1 + - event_persister2 +``` + +#### Background tasks + +There is also *experimental* support for moving background tasks to a separate +worker. Background tasks are run periodically or started via replication. Exactly +which tasks are configured to run depends on your Synapse configuration (e.g. if +stats is enabled). + +To enable this, the worker must have a `worker_name` and can be configured to run +background tasks. For example, to move background tasks to a dedicated worker, +the shared configuration would include: + +```yaml +run_background_tasks_on: background_worker +``` + +You might also wish to investigate the `update_user_directory` and +`media_instance_running_background_jobs` settings. + +### `synapse.app.pusher` + +Handles sending push notifications to sygnal and email. Doesn't handle any +REST endpoints itself, but you should set `start_pushers: False` in the +shared configuration file to stop the main synapse sending push notifications. + +To run multiple instances at once the `pusher_instances` option should list all +pusher instances by their worker name, e.g.: + +```yaml +pusher_instances: + - pusher_worker1 + - pusher_worker2 +``` + + +### `synapse.app.appservice` + +Handles sending output traffic to Application Services. Doesn't handle any +REST endpoints itself, but you should set `notify_appservices: False` in the +shared configuration file to stop the main synapse sending appservice notifications. + +Note this worker cannot be load-balanced: only one instance should be active. + + +### `synapse.app.federation_sender` + +Handles sending federation traffic to other servers. Doesn't handle any +REST endpoints itself, but you should set `send_federation: False` in the +shared configuration file to stop the main synapse sending this traffic. + +If running multiple federation senders then you must list each +instance in the `federation_sender_instances` option by their `worker_name`. +All instances must be stopped and started when adding or removing instances. +For example: + +```yaml +federation_sender_instances: + - federation_sender1 + - federation_sender2 +``` + +### `synapse.app.media_repository` + +Handles the media repository. It can handle all endpoints starting with: + + /_matrix/media/ + +... and the following regular expressions matching media-specific administration APIs: + + ^/_synapse/admin/v1/purge_media_cache$ + ^/_synapse/admin/v1/room/.*/media.*$ + ^/_synapse/admin/v1/user/.*/media.*$ + ^/_synapse/admin/v1/media/.*$ + ^/_synapse/admin/v1/quarantine_media/.*$ + ^/_synapse/admin/v1/users/.*/media$ + +You should also set `enable_media_repo: False` in the shared configuration +file to stop the main synapse running background jobs related to managing the +media repository. Note that doing so will prevent the main process from being +able to handle the above endpoints. + +In the `media_repository` worker configuration file, configure the http listener to +expose the `media` resource. For example: + +```yaml +worker_listeners: + - type: http + port: 8085 + resources: + - names: + - media +``` + +Note that if running multiple media repositories they must be on the same server +and you must configure a single instance to run the background tasks, e.g.: + +```yaml +media_instance_running_background_jobs: "media-repository-1" +``` + +Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for both inbound client and federation requests (if they are handled separately). + +### `synapse.app.user_dir` + +Handles searches in the user directory. It can handle REST endpoints matching +the following regular expressions: + + ^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$ + +When using this worker you must also set `update_user_directory: False` in the +shared configuration file to stop the main synapse running background +jobs related to updating the user directory. + +### `synapse.app.frontend_proxy` + +Proxies some frequently-requested client endpoints to add caching and remove +load from the main synapse. It can handle REST endpoints matching the following +regular expressions: + + ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload + +If `use_presence` is False in the homeserver config, it can also handle REST +endpoints matching the following regular expressions: + + ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/[^/]+/status + +This "stub" presence handler will pass through `GET` request but make the +`PUT` effectively a no-op. + +It will proxy any requests it cannot handle to the main synapse instance. It +must therefore be configured with the location of the main instance, via +the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration +file. For example: + +```yaml +worker_main_http_uri: http://127.0.0.1:8008 +``` + +### Historical apps + +*Note:* Historically there used to be more apps, however they have been +amalgamated into a single `synapse.app.generic_worker` app. The remaining apps +are ones that do specific processing unrelated to requests, e.g. the `pusher` +that handles sending out push notifications for new events. The intention is for +all these to be folded into the `generic_worker` app and to use config to define +which processes handle the various proccessing such as push notifications. + + +## Migration from old config + +There are two main independent changes that have been made: introducing Redis +support and merging apps into `synapse.app.generic_worker`. Both these changes +are backwards compatible and so no changes to the config are required, however +server admins are encouraged to plan to migrate to Redis as the old style direct +TCP replication config is deprecated. + +To migrate to Redis add the `redis` config as above, and optionally remove the +TCP `replication` listener from master and `worker_replication_port` from worker +config. + +To migrate apps to use `synapse.app.generic_worker` simply update the +`worker_app` option in the worker configs, and where worker are started (e.g. +in systemd service files, but not required for synctl). + + +## Architectural diagram + +The following shows an example setup using Redis and a reverse proxy: + +``` + Clients & Federation + | + v + +-----------+ + | | + | Reverse | + | Proxy | + | | + +-----------+ + | | | + | | | HTTP requests + +-------------------+ | +-----------+ + | +---+ | + | | | + v v v ++--------------+ +--------------+ +--------------+ +--------------+ +| Main | | Generic | | Generic | | Event | +| Process | | Worker 1 | | Worker 2 | | Persister | ++--------------+ +--------------+ +--------------+ +--------------+ + ^ ^ | ^ | | ^ | ^ ^ + | | | | | | | | | | + | | | | | HTTP | | | | | + | +----------+<--|---|---------+ | | | | + | | +-------------|-->+----------+ | + | | | | + | | | | + v v v v +==================================================================== + Redis pub/sub channel +``` diff --git a/docs/usage/configuration/workers/synctl_workers.md b/docs/usage/configuration/workers/synctl_workers.md new file mode 100644 index 0000000000..15e37f608d --- /dev/null +++ b/docs/usage/configuration/workers/synctl_workers.md @@ -0,0 +1,36 @@ +### Using synctl with workers + +If you want to use `synctl` to manage your synapse processes, you will need to +create an an additional configuration file for the main synapse process. That +configuration should look like this: + +```yaml +worker_app: synapse.app.homeserver +``` + +Additionally, each worker app must be configured with the name of a "pid file", +to which it will write its process ID when it starts. For example, for a +synchrotron, you might write: + +```yaml +worker_pid_file: /home/matrix/synapse/worker1.pid +``` + +Finally, to actually run your worker-based synapse, you must pass synctl the `-a` +commandline option to tell it to operate on all the worker configurations found +in the given directory, e.g.: + +```sh +synctl -a $CONFIG/workers start +``` + +Currently one should always restart all workers when restarting or upgrading +synapse, unless you explicitly know it's safe not to. For instance, restarting +synapse without restarting all the synchrotrons may result in broken typing +notifications. + +To manipulate a specific worker, you pass the -w option to synctl: + +```sh +synctl -w $CONFIG/workers/worker1.yaml restart +``` diff --git a/docs/usage/federation/README.md b/docs/usage/federation/README.md new file mode 100644 index 0000000000..5107f995be --- /dev/null +++ b/docs/usage/federation/README.md @@ -0,0 +1,66 @@ +Setting up federation +===================== + +Federation is the process by which users on different servers can participate +in the same room. For this to work, those other servers must be able to contact +yours to send messages. + +The `server_name` configured in the Synapse configuration file (often +`homeserver.yaml`) defines how resources (users, rooms, etc.) will be +identified (eg: `@user:example.com`, `#room:example.com`). By default, +it is also the domain that other servers will use to try to reach your +server (via port 8448). This is easy to set up and will work provided +you set the `server_name` to match your machine's public DNS hostname. + +For this default configuration to work, you will need to listen for TLS +connections on port 8448. The preferred way to do that is by using a +reverse proxy: see [the reverse proxy documentation](reverse_proxy.md) for instructions +on how to correctly set one up. + +In some cases you might not want to run Synapse on the machine that has +the `server_name` as its public DNS hostname, or you might want federation +traffic to use a different port than 8448. For example, you might want to +have your user names look like `@user:example.com`, but you want to run +Synapse on `synapse.example.com` on port 443. This can be done using +delegation, which allows an admin to control where federation traffic should +be sent. See [the delegation documentation](delegate.md) for instructions on how to set this up. + +Once federation has been configured, you should be able to join a room over +federation. A good place to start is `#synapse:matrix.org` - a room for +Synapse admins. + +## Troubleshooting + +You can use the [federation tester](https://matrix.org/federationtester) +to check if your homeserver is configured correctly. Alternatively try the +[JSON API used by the federation tester](https://matrix.org/federationtester/api/report?server_name=DOMAIN). +Note that you'll have to modify this URL to replace `DOMAIN` with your +`server_name`. Hitting the API directly provides extra detail. + +The typical failure mode for federation is that when the server tries to join +a room, it is rejected with "401: Unauthorized". Generally this means that other +servers in the room could not access yours. (Joining a room over federation is +a complicated dance which requires connections in both directions). + +Another common problem is that people on other servers can't join rooms that +you invite them to. This can be caused by an incorrectly-configured reverse +proxy: see [the reverse proxy documentation](reverse_proxy.md) for instructions on how +to correctly configure a reverse proxy. + +### Known issues + +**HTTP `308 Permanent Redirect` redirects are not followed**: Due to missing features +in the HTTP library used by Synapse, 308 redirects are currently not followed by +federating servers, which can cause `M_UNKNOWN` or `401 Unauthorized` errors. This +may affect users who are redirecting apex-to-www (e.g. `example.com` -> `www.example.com`), +and especially users of the Kubernetes *Nginx Ingress* module, which uses 308 redirect +codes by default. For those Kubernetes users, [this Stackoverflow post](https://stackoverflow.com/a/52617528/5096871) +might be helpful. For other users, switching to a `301 Moved Permanently` code may be +an option. 308 redirect codes will be supported properly in a future +release of Synapse. + +## Running a demo federation of Synapses + +If you want to get up and running quickly with a trio of homeservers in a +private federation, there is a script in the `demo` directory. This is mainly +useful just for development purposes. See [demo/README](https://github.com/matrix-org/synapse/tree/develop/demo/). diff --git a/docs/user_directory.md b/docs/user_directory.md deleted file mode 100644 index c4794b04cf..0000000000 --- a/docs/user_directory.md +++ /dev/null @@ -1,49 +0,0 @@ -User Directory API Implementation -================================= - -The user directory is currently maintained based on the 'visible' users -on this particular server - i.e. ones which your account shares a room with, or -who are present in a publicly viewable room present on the server. - -The directory info is stored in various tables, which can (typically after -DB corruption) get stale or out of sync. If this happens, for now the -solution to fix it is to use the [admin API](usage/administration/admin_api/background_updates.md#run) -and execute the job `regenerate_directory`. This should then start a background task to -flush the current tables and regenerate the directory. - -Data model ----------- - -There are five relevant tables that collectively form the "user directory". -Three of them track a master list of all the users we could search for. -The last two (collectively called the "search tables") track who can -see who. - -From all of these tables we exclude three types of local user: - - support users - - appservice users - - deactivated users - -* `user_directory`. This contains the user_id, display name and avatar we'll - return when you search the directory. - - Because there's only one directory entry per user, it's important that we only - ever put publicly visible names here. Otherwise we might leak a private - nickname or avatar used in a private room. - - Indexed on rooms. Indexed on users. - -* `user_directory_search`. To be joined to `user_directory`. It contains an extra - column that enables full text search based on user ids and display names. - Different schemas for SQLite and Postgres with different code paths to match. - - Indexed on the full text search data. Indexed on users. - -* `user_directory_stream_pos`. When the initial background update to populate - the directory is complete, we record a stream position here. This indicates - that synapse should now listen for room changes and incrementally update - the directory where necessary. - -* `users_in_public_rooms`. Contains associations between users and the public rooms they're in. - Used to determine which users are in public rooms and should be publicly visible in the directory. - -* `users_who_share_private_rooms`. Rows are triples `(L, M, room id)` where `L` - is a local user and `M` is a local or remote user. `L` and `M` should be - different, but this isn't enforced by a constraint. diff --git a/docs/welcome_and_overview.md b/docs/welcome_and_overview.md deleted file mode 100644 index aab2d6b4f0..0000000000 --- a/docs/welcome_and_overview.md +++ /dev/null @@ -1,79 +0,0 @@ -# Introduction - -Welcome to the documentation repository for Synapse, a -[Matrix](https://matrix.org) homeserver implementation developed by the matrix.org core -team. - -## Installing and using Synapse - -This documentation covers topics for **installation**, **configuration** and -**maintainence** of your Synapse process: - -* Learn how to [install](setup/installation.md) and - [configure](usage/configuration/index.html) your own instance, perhaps with [Single - Sign-On](usage/configuration/user_authentication/index.html). - -* See how to [upgrade](upgrade.md) between Synapse versions. - -* Administer your instance using the [Admin - API](usage/administration/admin_api/index.html), installing [pluggable - modules](modules/index.html), or by accessing the [manhole](manhole.md). - -* Learn how to [read log lines](usage/administration/request_log.md), configure - [logging](usage/configuration/logging_sample_config.md) or set up [structured - logging](structured_logging.md). - -* Scale Synapse through additional [worker processes](workers.md). - -* Set up [monitoring and metrics](metrics-howto.md) to keep an eye on your - Synapse instance's performance. - -## Developing on Synapse - -Contributions are welcome! Synapse is primarily written in -[Python](https://python.org). As a developer, you may be interested in the -following documentation: - -* Read the [Contributing Guide](development/contributing_guide.md). It is meant - to walk new contributors through the process of developing and submitting a - change to the Synapse codebase (which is [hosted on - GitHub](https://github.com/matrix-org/synapse)). - -* Set up your [development - environment](development/contributing_guide.md#2-what-do-i-need), then learn - how to [lint](development/contributing_guide.md#run-the-linters) and - [test](development/contributing_guide.md#8-test-test-test) your code. - -* Look at [the issue tracker](https://github.com/matrix-org/synapse/issues) for - bugs to fix or features to add. If you're new, it may be best to start with - those labeled [good first - issue](https://github.com/matrix-org/synapse/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). - -* Understand [how Synapse is - built](development/internal_documentation/index.html), how to [migrate - database schemas](development/database_schema.md), learn about - [federation](federate.md) and how to [set up a local - federation](federate.md#running-a-demo-federation-of-synapses) for development. - -* We like to keep our `git` history clean. [Learn](development/git.md) how to - do so! - -* And finally, contribute to this documentation! The source for which is - [located here](https://github.com/matrix-org/synapse/tree/develop/docs). - -## Donating to Synapse development - -Want to help keep Synapse going but don't know how to code? Synapse is a -[Matrix.org Foundation](https://matrix.org) project. Consider becoming a -supportor on [Liberapay](https://liberapay.com/matrixdotorg), -[Patreon](https://patreon.com/matrixdotorg) or through -[PayPal](https://paypal.me/matrixdotorg) via a one-time donation. - -If you are an organisation or enterprise and would like to sponsor development, -reach out to us over email at: support (at) matrix.org - -## Reporting a security vulnerability - -If you've found a security issue in Synapse or any other Matrix.org Foundation -project, please report it to us in accordance with our [Security Disclosure -Policy](https://www.matrix.org/security-disclosure-policy/). Thank you! diff --git a/docs/workers.md b/docs/workers.md deleted file mode 100644 index 17c8bfeef6..0000000000 --- a/docs/workers.md +++ /dev/null @@ -1,560 +0,0 @@ -# Scaling synapse via workers - -For small instances it recommended to run Synapse in the default monolith mode. -For larger instances where performance is a concern it can be helpful to split -out functionality into multiple separate python processes. These processes are -called 'workers', and are (eventually) intended to scale horizontally -independently. - -Synapse's worker support is under active development and subject to change as -we attempt to rapidly scale ever larger Synapse instances. However we are -documenting it here to help admins needing a highly scalable Synapse instance -similar to the one running `matrix.org`. - -All processes continue to share the same database instance, and as such, -workers only work with PostgreSQL-based Synapse deployments. SQLite should only -be used for demo purposes and any admin considering workers should already be -running PostgreSQL. - -See also [Matrix.org blog post](https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability) -for a higher level overview. - -## Main process/worker communication - -The processes communicate with each other via a Synapse-specific protocol called -'replication' (analogous to MySQL- or Postgres-style database replication) which -feeds streams of newly written data between processes so they can be kept in -sync with the database state. - -When configured to do so, Synapse uses a -[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication -stream between all configured Synapse processes. Additionally, processes may -make HTTP requests to each other, primarily for operations which need to wait -for a reply ─ such as sending an event. - -Redis support was added in v1.13.0 with it becoming the recommended method in -v1.18.0. It replaced the old direct TCP connections (which is deprecated as of -v1.18.0) to the main process. With Redis, rather than all the workers connecting -to the main process, all the workers and the main process connect to Redis, -which relays replication commands between processes. This can give a significant -cpu saving on the main process and will be a prerequisite for upcoming -performance improvements. - -If Redis support is enabled Synapse will use it as a shared cache, as well as a -pub/sub mechanism. - -See the [Architectural diagram](#architectural-diagram) section at the end for -a visualisation of what this looks like. - - -## Setting up workers - -A Redis server is required to manage the communication between the processes. -The Redis server should be installed following the normal procedure for your -distribution (e.g. `apt install redis-server` on Debian). It is safe to use an -existing Redis deployment if you have one. - -Once installed, check that Redis is running and accessible from the host running -Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing -a response of `+PONG`. - -The appropriate dependencies must also be installed for Synapse. If using a -virtualenv, these can be installed with: - -```sh -pip install "matrix-synapse[redis]" -``` - -Note that these dependencies are included when synapse is installed with `pip -install matrix-synapse[all]`. They are also included in the debian packages from -`matrix.org` and in the docker images at -https://hub.docker.com/r/matrixdotorg/synapse/. - -To make effective use of the workers, you will need to configure an HTTP -reverse-proxy such as nginx or haproxy, which will direct incoming requests to -the correct worker, or to the main synapse instance. See -[the reverse proxy documentation](reverse_proxy.md) for information on setting up a reverse -proxy. - -When using workers, each worker process has its own configuration file which -contains settings specific to that worker, such as the HTTP listener that it -provides (if any), logging configuration, etc. - -Normally, the worker processes are configured to read from a shared -configuration file as well as the worker-specific configuration files. This -makes it easier to keep common configuration settings synchronised across all -the processes. - -The main process is somewhat special in this respect: it does not normally -need its own configuration file and can take all of its configuration from the -shared configuration file. - - -### Shared configuration - -Normally, only a couple of changes are needed to make an existing configuration -file suitable for use with workers. First, you need to enable an "HTTP replication -listener" for the main process; and secondly, you need to enable redis-based -replication. Optionally, a shared secret can be used to authenticate HTTP -traffic between workers. For example: - - -```yaml -# extend the existing `listeners` section. This defines the ports that the -# main process will listen on. -listeners: - # The HTTP replication port - - port: 9093 - bind_address: '127.0.0.1' - type: http - resources: - - names: [replication] - -# Add a random shared secret to authenticate traffic. -worker_replication_secret: "" - -redis: - enabled: true -``` - -See the sample config for the full documentation of each option. - -Under **no circumstances** should the replication listener be exposed to the -public internet; it has no authentication and is unencrypted. - - -### Worker configuration - -In the config file for each worker, you must specify the type of worker -application (`worker_app`), and you should specify a unique name for the worker -(`worker_name`). The currently available worker applications are listed below. -You must also specify the HTTP replication endpoint that it should talk to on -the main synapse process. `worker_replication_host` should specify the host of -the main synapse and `worker_replication_http_port` should point to the HTTP -replication port. If the worker will handle HTTP requests then the -`worker_listeners` option should be set with a `http` listener, in the same way -as the `listeners` option in the shared config. - -For example: - -```yaml -worker_app: synapse.app.generic_worker -worker_name: worker1 - -# The replication listener on the main synapse process. -worker_replication_host: 127.0.0.1 -worker_replication_http_port: 9093 - -worker_listeners: - - type: http - port: 8083 - resources: - - names: - - client - - federation - -worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml -``` - -...is a full configuration for a generic worker instance, which will expose a -plain HTTP endpoint on port 8083 separately serving various endpoints, e.g. -`/sync`, which are listed below. - -Obviously you should configure your reverse-proxy to route the relevant -endpoints to the worker (`localhost:8083` in the above example). - - -### Running Synapse with workers - -Finally, you need to start your worker processes. This can be done with either -`synctl` or your distribution's preferred service manager such as `systemd`. We -recommend the use of `systemd` where available: for information on setting up -`systemd` to start synapse workers, see -[Systemd with Workers](systemd-with-workers). To use `synctl`, see -[Using synctl with Workers](synctl_workers.md). - - -## Available worker applications - -### `synapse.app.generic_worker` - -This worker can handle API requests matching the following regular -expressions: - - # Sync requests - ^/_matrix/client/(v2_alpha|r0|v3)/sync$ - ^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$ - ^/_matrix/client/(api/v1|r0|v3)/initialSync$ - ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ - - # Federation requests - ^/_matrix/federation/v1/event/ - ^/_matrix/federation/v1/state/ - ^/_matrix/federation/v1/state_ids/ - ^/_matrix/federation/v1/backfill/ - ^/_matrix/federation/v1/get_missing_events/ - ^/_matrix/federation/v1/publicRooms - ^/_matrix/federation/v1/query/ - ^/_matrix/federation/v1/make_join/ - ^/_matrix/federation/v1/make_leave/ - ^/_matrix/federation/v1/send_join/ - ^/_matrix/federation/v2/send_join/ - ^/_matrix/federation/v1/send_leave/ - ^/_matrix/federation/v2/send_leave/ - ^/_matrix/federation/v1/invite/ - ^/_matrix/federation/v2/invite/ - ^/_matrix/federation/v1/query_auth/ - ^/_matrix/federation/v1/event_auth/ - ^/_matrix/federation/v1/exchange_third_party_invite/ - ^/_matrix/federation/v1/user/devices/ - ^/_matrix/federation/v1/get_groups_publicised$ - ^/_matrix/key/v2/query - ^/_matrix/federation/unstable/org.matrix.msc2946/spaces/ - ^/_matrix/federation/unstable/org.matrix.msc2946/hierarchy/ - - # Inbound federation transaction request - ^/_matrix/federation/v1/send/ - - # Client API requests - ^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$ - ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/spaces$ - ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/hierarchy$ - ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/account/3pid$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/devices$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/query$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/changes$ - ^/_matrix/client/versions$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_groups$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/search$ - - # Registration/login requests - ^/_matrix/client/(api/v1|r0|v3|unstable)/login$ - ^/_matrix/client/(r0|v3|unstable)/register$ - ^/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity$ - - # Event sending requests - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ - ^/_matrix/client/(api/v1|r0|v3|unstable)/join/ - ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/ - - -Additionally, the following REST endpoints can be handled for GET requests: - - ^/_matrix/federation/v1/groups/ - -Pagination requests can also be handled, but all requests for a given -room must be routed to the same instance. Additionally, care must be taken to -ensure that the purge history admin API is not used while pagination requests -for the room are in flight: - - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$ - -Additionally, the following endpoints should be included if Synapse is configured -to use SSO (you only need to include the ones for whichever SSO provider you're -using): - - # for all SSO providers - ^/_matrix/client/(api/v1|r0|v3|unstable)/login/sso/redirect - ^/_synapse/client/pick_idp$ - ^/_synapse/client/pick_username - ^/_synapse/client/new_user_consent$ - ^/_synapse/client/sso_register$ - - # OpenID Connect requests. - ^/_synapse/client/oidc/callback$ - - # SAML requests. - ^/_synapse/client/saml2/authn_response$ - - # CAS requests. - ^/_matrix/client/(api/v1|r0|v3|unstable)/login/cas/ticket$ - -Ensure that all SSO logins go to a single process. -For multiple workers not handling the SSO endpoints properly, see -[#7530](https://github.com/matrix-org/synapse/issues/7530) and -[#9427](https://github.com/matrix-org/synapse/issues/9427). - -Note that a HTTP listener with `client` and `federation` resources must be -configured in the `worker_listeners` option in the worker config. - -#### Load balancing - -It is possible to run multiple instances of this worker app, with incoming requests -being load-balanced between them by the reverse-proxy. However, different endpoints -have different characteristics and so admins -may wish to run multiple groups of workers handling different endpoints so that -load balancing can be done in different ways. - -For `/sync` and `/initialSync` requests it will be more efficient if all -requests from a particular user are routed to a single instance. Extracting a -user ID from the access token or `Authorization` header is currently left as an -exercise for the reader. Admins may additionally wish to separate out `/sync` -requests that have a `since` query parameter from those that don't (and -`/initialSync`), as requests that don't are known as "initial sync" that happens -when a user logs in on a new device and can be *very* resource intensive, so -isolating these requests will stop them from interfering with other users ongoing -syncs. - -Federation and client requests can be balanced via simple round robin. - -The inbound federation transaction request `^/_matrix/federation/v1/send/` -should be balanced by source IP so that transactions from the same remote server -go to the same process. - -Registration/login requests can be handled separately purely to help ensure that -unexpected load doesn't affect new logins and sign ups. - -Finally, event sending requests can be balanced by the room ID in the URI (or -the full URI, or even just round robin), the room ID is the path component after -`/rooms/`. If there is a large bridge connected that is sending or may send lots -of events, then a dedicated set of workers can be provisioned to limit the -effects of bursts of events from that bridge on events sent by normal users. - -#### Stream writers - -Additionally, there is *experimental* support for moving writing of specific -streams (such as events) off of the main process to a particular worker. (This -is only supported with Redis-based replication.) - -Currently supported streams are `events` and `typing`. - -To enable this, the worker must have a HTTP replication listener configured, -have a `worker_name` and be listed in the `instance_map` config. For example to -move event persistence off to a dedicated worker, the shared configuration would -include: - -```yaml -instance_map: - event_persister1: - host: localhost - port: 8034 - -stream_writers: - events: event_persister1 -``` - -The `events` stream also experimentally supports having multiple writers, where -work is sharded between them by room ID. Note that you *must* restart all worker -instances when adding or removing event persisters. An example `stream_writers` -configuration with multiple writers: - -```yaml -stream_writers: - events: - - event_persister1 - - event_persister2 -``` - -#### Background tasks - -There is also *experimental* support for moving background tasks to a separate -worker. Background tasks are run periodically or started via replication. Exactly -which tasks are configured to run depends on your Synapse configuration (e.g. if -stats is enabled). - -To enable this, the worker must have a `worker_name` and can be configured to run -background tasks. For example, to move background tasks to a dedicated worker, -the shared configuration would include: - -```yaml -run_background_tasks_on: background_worker -``` - -You might also wish to investigate the `update_user_directory` and -`media_instance_running_background_jobs` settings. - -### `synapse.app.pusher` - -Handles sending push notifications to sygnal and email. Doesn't handle any -REST endpoints itself, but you should set `start_pushers: False` in the -shared configuration file to stop the main synapse sending push notifications. - -To run multiple instances at once the `pusher_instances` option should list all -pusher instances by their worker name, e.g.: - -```yaml -pusher_instances: - - pusher_worker1 - - pusher_worker2 -``` - - -### `synapse.app.appservice` - -Handles sending output traffic to Application Services. Doesn't handle any -REST endpoints itself, but you should set `notify_appservices: False` in the -shared configuration file to stop the main synapse sending appservice notifications. - -Note this worker cannot be load-balanced: only one instance should be active. - - -### `synapse.app.federation_sender` - -Handles sending federation traffic to other servers. Doesn't handle any -REST endpoints itself, but you should set `send_federation: False` in the -shared configuration file to stop the main synapse sending this traffic. - -If running multiple federation senders then you must list each -instance in the `federation_sender_instances` option by their `worker_name`. -All instances must be stopped and started when adding or removing instances. -For example: - -```yaml -federation_sender_instances: - - federation_sender1 - - federation_sender2 -``` - -### `synapse.app.media_repository` - -Handles the media repository. It can handle all endpoints starting with: - - /_matrix/media/ - -... and the following regular expressions matching media-specific administration APIs: - - ^/_synapse/admin/v1/purge_media_cache$ - ^/_synapse/admin/v1/room/.*/media.*$ - ^/_synapse/admin/v1/user/.*/media.*$ - ^/_synapse/admin/v1/media/.*$ - ^/_synapse/admin/v1/quarantine_media/.*$ - ^/_synapse/admin/v1/users/.*/media$ - -You should also set `enable_media_repo: False` in the shared configuration -file to stop the main synapse running background jobs related to managing the -media repository. Note that doing so will prevent the main process from being -able to handle the above endpoints. - -In the `media_repository` worker configuration file, configure the http listener to -expose the `media` resource. For example: - -```yaml -worker_listeners: - - type: http - port: 8085 - resources: - - names: - - media -``` - -Note that if running multiple media repositories they must be on the same server -and you must configure a single instance to run the background tasks, e.g.: - -```yaml -media_instance_running_background_jobs: "media-repository-1" -``` - -Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for both inbound client and federation requests (if they are handled separately). - -### `synapse.app.user_dir` - -Handles searches in the user directory. It can handle REST endpoints matching -the following regular expressions: - - ^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$ - -When using this worker you must also set `update_user_directory: False` in the -shared configuration file to stop the main synapse running background -jobs related to updating the user directory. - -### `synapse.app.frontend_proxy` - -Proxies some frequently-requested client endpoints to add caching and remove -load from the main synapse. It can handle REST endpoints matching the following -regular expressions: - - ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload - -If `use_presence` is False in the homeserver config, it can also handle REST -endpoints matching the following regular expressions: - - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/[^/]+/status - -This "stub" presence handler will pass through `GET` request but make the -`PUT` effectively a no-op. - -It will proxy any requests it cannot handle to the main synapse instance. It -must therefore be configured with the location of the main instance, via -the `worker_main_http_uri` setting in the `frontend_proxy` worker configuration -file. For example: - -```yaml -worker_main_http_uri: http://127.0.0.1:8008 -``` - -### Historical apps - -*Note:* Historically there used to be more apps, however they have been -amalgamated into a single `synapse.app.generic_worker` app. The remaining apps -are ones that do specific processing unrelated to requests, e.g. the `pusher` -that handles sending out push notifications for new events. The intention is for -all these to be folded into the `generic_worker` app and to use config to define -which processes handle the various proccessing such as push notifications. - - -## Migration from old config - -There are two main independent changes that have been made: introducing Redis -support and merging apps into `synapse.app.generic_worker`. Both these changes -are backwards compatible and so no changes to the config are required, however -server admins are encouraged to plan to migrate to Redis as the old style direct -TCP replication config is deprecated. - -To migrate to Redis add the `redis` config as above, and optionally remove the -TCP `replication` listener from master and `worker_replication_port` from worker -config. - -To migrate apps to use `synapse.app.generic_worker` simply update the -`worker_app` option in the worker configs, and where worker are started (e.g. -in systemd service files, but not required for synctl). - - -## Architectural diagram - -The following shows an example setup using Redis and a reverse proxy: - -``` - Clients & Federation - | - v - +-----------+ - | | - | Reverse | - | Proxy | - | | - +-----------+ - | | | - | | | HTTP requests - +-------------------+ | +-----------+ - | +---+ | - | | | - v v v -+--------------+ +--------------+ +--------------+ +--------------+ -| Main | | Generic | | Generic | | Event | -| Process | | Worker 1 | | Worker 2 | | Persister | -+--------------+ +--------------+ +--------------+ +--------------+ - ^ ^ | ^ | | ^ | ^ ^ - | | | | | | | | | | - | | | | | HTTP | | | | | - | +----------+<--|---|---------+ | | | | - | | +-------------|-->+----------+ | - | | | | - | | | | - v v v v -==================================================================== - Redis pub/sub channel -``` -- cgit 1.5.1