diff --git a/latest/print.html b/latest/print.html
index 717f3b5c25..b0a7e87299 100644
--- a/latest/print.html
+++ b/latest/print.html
@@ -77,7 +77,7 @@
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
- <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/config_documentation.html">Configuration Manual</a></li><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/index.html">Single-Sign On</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/cas.html">CAS</a></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/refresh_tokens.html">Refresh Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules/index.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/writing_a_module.html">Writing a module</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/spam_checker_callbacks.html">Spam checker callbacks</a></li><li class="chapter-item expanded "><a href="modules/third_party_rules_callbacks.html">Third-party rules callbacks</a></li><li class="chapter-item expanded "><a href="modules/presence_router_callbacks.html">Presence router callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_validity_callbacks.html">Account validity callbacks</a></li><li class="chapter-item expanded "><a href="modules/password_auth_provider_callbacks.html">Password auth provider callbacks</a></li><li class="chapter-item expanded "><a href="modules/background_update_controller_callbacks.html">Background update controller callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_data_callbacks.html">Account data callbacks</a></li><li class="chapter-item expanded "><a href="modules/porting_legacy_module.html">Porting a legacy module to the new interface</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/background_updates.html">Background Updates</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/federation.html">Federation</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/monitoring/reporting_homeserver_usage_statistics.html">Reporting Homeserver Usage Statistics</a></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/understanding_synapse_through_grafana_graphs.html">Understanding Synapse Through Grafana Graphs</a></li><li class="chapter-item expanded "><a href="usage/administration/useful_sql_for_admins.html">Useful SQL for Admins</a></li><li class="chapter-item expanded "><a href="usage/administration/database_maintenance_tools.html">Database Maintenance Tools</a></li><li class="chapter-item expanded "><a href="usage/administration/state_groups.html">State Groups</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_faq.html">Admin FAQ</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/reviews.html">Reviewing Code</a></li><li class="chapter-item expanded "><a href="development/releases.html">Release Cycle</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/demo.html">Demo scripts</a></li></ol></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><a href="development/experimental_features.html">Experimental features</a></li><li class="chapter-item expanded "><a href="development/dependencies.html">Dependency management</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/synapse_architecture/cancellation.html">Cancellation</a></li><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li><li class="chapter-item expanded "><a href="other/running_synapse_on_single_board_computers.html">Running Synapse on a Single-Board Computer</a></li></ol>
+ <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/config_documentation.html">Configuration Manual</a></li><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/index.html">Single-Sign On</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/cas.html">CAS</a></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/refresh_tokens.html">Refresh Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules/index.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/writing_a_module.html">Writing a module</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/spam_checker_callbacks.html">Spam checker callbacks</a></li><li class="chapter-item expanded "><a href="modules/third_party_rules_callbacks.html">Third-party rules callbacks</a></li><li class="chapter-item expanded "><a href="modules/presence_router_callbacks.html">Presence router callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_validity_callbacks.html">Account validity callbacks</a></li><li class="chapter-item expanded "><a href="modules/password_auth_provider_callbacks.html">Password auth provider callbacks</a></li><li class="chapter-item expanded "><a href="modules/background_update_controller_callbacks.html">Background update controller callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_data_callbacks.html">Account data callbacks</a></li><li class="chapter-item expanded "><a href="modules/porting_legacy_module.html">Porting a legacy module to the new interface</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/background_updates.html">Background Updates</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/federation.html">Federation</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/monitoring/reporting_homeserver_usage_statistics.html">Reporting Homeserver Usage Statistics</a></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/monthly_active_users.html">Monthly Active Users</a></li><li class="chapter-item expanded "><a href="usage/administration/understanding_synapse_through_grafana_graphs.html">Understanding Synapse Through Grafana Graphs</a></li><li class="chapter-item expanded "><a href="usage/administration/useful_sql_for_admins.html">Useful SQL for Admins</a></li><li class="chapter-item expanded "><a href="usage/administration/database_maintenance_tools.html">Database Maintenance Tools</a></li><li class="chapter-item expanded "><a href="usage/administration/state_groups.html">State Groups</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_faq.html">Admin FAQ</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/reviews.html">Reviewing Code</a></li><li class="chapter-item expanded "><a href="development/releases.html">Release Cycle</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/demo.html">Demo scripts</a></li></ol></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><a href="development/experimental_features.html">Experimental features</a></li><li class="chapter-item expanded "><a href="development/dependencies.html">Dependency management</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/synapse_architecture/cancellation.html">Cancellation</a></li><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li><li class="chapter-item expanded "><a href="other/running_synapse_on_single_board_computers.html">Running Synapse on a Single-Board Computer</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
@@ -583,8 +583,12 @@ and <code>notif_from</code> fields filled out. You may also need to set <code>s
<p>If email is not configured, password reset, registration and notifications via
email will be disabled.</p>
<h3 id="registering-a-user"><a class="header" href="#registering-a-user">Registering a user</a></h3>
-<p>The easiest way to create a new user is to do so from a client like <a href="https://element.io/">Element</a>.</p>
-<p>Alternatively, you can do so from the command line. This can be done as follows:</p>
+<p>One way to create a new user is to do so from a client like
+<a href="https://element.io/">Element</a>. This requires registration to be enabled via
+the
+<a href="setup/../usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a>
+setting.</p>
+<p>Alternatively, you can create new users from the command line. This can be done as follows:</p>
<ol>
<li>If synapse was installed via pip, activate the virtualenv as follows (if Synapse was
installed via a prebuilt package, <code>register_new_matrix_user</code> should already be
@@ -595,7 +599,7 @@ synctl start # if not already running
</code></pre>
</li>
<li>Run the following command:
-<pre><code class="language-sh">register_new_matrix_user -c homeserver.yaml http://localhost:8008
+<pre><code class="language-sh">register_new_matrix_user -c homeserver.yaml
</code></pre>
</li>
</ol>
@@ -607,12 +611,13 @@ Confirm password:
Make admin [no]:
Success!
</code></pre>
-<p>This process uses a setting <code>registration_shared_secret</code> in
-<code>homeserver.yaml</code>, which is shared between Synapse itself and the
-<code>register_new_matrix_user</code> script. It doesn't matter what it is (a random
-value is generated by <code>--generate-config</code>), but it should be kept secret, as
-anyone with knowledge of it can register users, including admin accounts,
-on your server even if <code>enable_registration</code> is <code>false</code>.</p>
+<p>This process uses a setting
+<a href="setup/../usage/configuration/config_documentation.html#registration_shared_secret"><code>registration_shared_secret</code></a>,
+which is shared between Synapse itself and the <code>register_new_matrix_user</code>
+script. It doesn't matter what it is (a random value is generated by
+<code>--generate-config</code>), but it should be kept secret, as anyone with knowledge of
+it can register users, including admin accounts, on your server even if
+<code>enable_registration</code> is <code>false</code>.</p>
<h3 id="setting-up-a-turn-server"><a class="header" href="#setting-up-a-turn-server">Setting up a TURN server</a></h3>
<p>For reliable VoIP calls to be routed via this homeserver, you MUST configure
a TURN server. See <a href="setup/../turn-howto.html">TURN setup</a> for details.</p>
@@ -1612,6 +1617,37 @@ dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
</code></pre>
</li>
</ul>
+<h1 id="upgrading-to-v1670"><a class="header" href="#upgrading-to-v1670">Upgrading to v1.67.0</a></h1>
+<h2 id="direct-tcp-replication-is-no-longer-supported-migrate-to-redis"><a class="header" href="#direct-tcp-replication-is-no-longer-supported-migrate-to-redis">Direct TCP replication is no longer supported: migrate to Redis</a></h2>
+<p>Redis support was added in v1.13.0 with it becoming the recommended method in
+v1.18.0. It replaced the old direct TCP connections (which was deprecated as of
+v1.18.0) to the main process. With Redis, rather than all the workers connecting
+to the main process, all the workers and the main process connect to Redis,
+which relays replication commands between processes. This can give a significant
+CPU saving on the main process and is a prerequisite for upcoming
+performance improvements.</p>
+<p>To migrate to Redis add the <a href="./workers.html#shared-configuration"><code>redis</code> config</a>,
+and remove the TCP <code>replication</code> listener from config of the master and
+<code>worker_replication_port</code> from worker config. Note that a HTTP listener with a
+<code>replication</code> resource is still required.</p>
+<h2 id="minimum-version-of-poetry-is-now-v120"><a class="header" href="#minimum-version-of-poetry-is-now-v120">Minimum version of Poetry is now v1.2.0</a></h2>
+<p>The minimum supported version of poetry is now 1.2. This should only affect
+those installing from a source checkout.</p>
+<h2 id="rust-requirement-in-the-next-release"><a class="header" href="#rust-requirement-in-the-next-release">Rust requirement in the next release</a></h2>
+<p>From the next major release (v1.68.0) installing Synapse from a source checkout
+will require a recent Rust compiler. Those using packages or
+<code>pip install matrix-synapse</code> will not be affected.</p>
+<p>The simplest way of installing Rust is via <a href="https://rustup.rs/">rustup.rs</a></p>
+<h2 id="sqlite-version-requirement-in-the-next-release"><a class="header" href="#sqlite-version-requirement-in-the-next-release">SQLite version requirement in the next release</a></h2>
+<p>From the next major release (v1.68.0) Synapse will require SQLite 3.27.0 or
+higher. Synapse v1.67.0 will be the last major release supporting SQLite
+versions 3.22 to 3.26.</p>
+<p>Those using docker images or Debian packages from Matrix.org will not be
+affected. If you have installed from source, you should check the version of
+SQLite used by Python with:</p>
+<pre><code class="language-shell">python -c "import sqlite3; print(sqlite3.sqlite_version)"
+</code></pre>
+<p>If this is too old, refer to your distribution for advice on upgrading.</p>
<h1 id="upgrading-to-v1660"><a class="header" href="#upgrading-to-v1660">Upgrading to v1.66.0</a></h1>
<h2 id="delegation-of-email-validation-no-longer-supported"><a class="header" href="#delegation-of-email-validation-no-longer-supported">Delegation of email validation no longer supported</a></h2>
<p>As of this version, Synapse no longer allows the tasks of verifying email address
@@ -3496,9 +3532,6 @@ configuration.</p>
<li>
<p><code>metrics</code>: (see the docs <a href="usage/configuration/../../metrics-howto.html">here</a>),</p>
</li>
-<li>
-<p><code>replication</code>: (deprecated as of Synapse 1.18, see the docs <a href="usage/configuration/../../workers.html">here</a>).</p>
-</li>
</ul>
</li>
<li>
@@ -3660,6 +3693,7 @@ hs_disabled_message: 'Reason for why the HS is blocked'
server owner wants to limit to the number of monthly active users. When enabled and a limit is
reached the server returns a <code>ResourceLimitError</code> with error type <code>Codes.RESOURCE_LIMIT_EXCEEDED</code>.
Defaults to false. If this is enabled, a value for <code>max_mau_value</code> must also be set.</p>
+<p>See <a href="usage/configuration/../administration/monthly_active_users.html">Monthly Active Users</a> for details on how to configure MAU.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">limit_usage_by_mau: true
</code></pre>
@@ -4728,23 +4762,25 @@ should be in the form of providers.json). By default this list is empty.</p>
<p>See <a href="usage/configuration/../../CAPTCHA_SETUP.html">here</a> for full details on setting up captcha.</p>
<hr />
<h3 id="recaptcha_public_key"><a class="header" href="#recaptcha_public_key"><code>recaptcha_public_key</code></a></h3>
-<p>This homeserver's ReCAPTCHA public key. Must be specified if <code>enable_registration_captcha</code> is
-enabled.</p>
+<p>This homeserver's ReCAPTCHA public key. Must be specified if
+<a href="usage/configuration/config_documentation.html#enable_registration_captcha"><code>enable_registration_captcha</code></a> is enabled.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">recaptcha_public_key: "YOUR_PUBLIC_KEY"
</code></pre>
<hr />
<h3 id="recaptcha_private_key"><a class="header" href="#recaptcha_private_key"><code>recaptcha_private_key</code></a></h3>
-<p>This homeserver's ReCAPTCHA private key. Must be specified if <code>enable_registration_captcha</code> is
+<p>This homeserver's ReCAPTCHA private key. Must be specified if
+<a href="usage/configuration/config_documentation.html#enable_registration_captcha"><code>enable_registration_captcha</code></a> is
enabled.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">recaptcha_private_key: "YOUR_PRIVATE_KEY"
</code></pre>
<hr />
<h3 id="enable_registration_captcha"><a class="header" href="#enable_registration_captcha"><code>enable_registration_captcha</code></a></h3>
-<p>Set to true to enable ReCaptcha checks when registering, preventing signup
-unless a captcha is answered. Requires a valid ReCaptcha public/private key.
-Defaults to false.</p>
+<p>Set to <code>true</code> to require users to complete a CAPTCHA test when registering an account.
+Requires a valid ReCaptcha public/private key.
+Defaults to <code>false</code>.</p>
+<p>Note that <a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a> must also be set to allow account registration.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">enable_registration_captcha: true
</code></pre>
@@ -4796,69 +4832,34 @@ it allows users to connect to arbitrary endpoints without having first signed up
<p>Registration can be rate-limited using the parameters in the <a href="usage/configuration/config_documentation.html#ratelimiting">Ratelimiting</a> section of this manual.</p>
<hr />
<h3 id="enable_registration"><a class="header" href="#enable_registration"><code>enable_registration</code></a></h3>
-<p>Enable registration for new users. Defaults to false. It is highly recommended that if you enable registration,
-you use either captcha, email, or token-based verification to verify that new users are not bots. In order to enable registration
-without any verification, you must also set <code>enable_registration_without_verification</code> to true.</p>
+<p>Enable registration for new users. Defaults to <code>false</code>.</p>
+<p>It is highly recommended that if you enable registration, you set one or more
+or the following options, to avoid abuse of your server by "bots":</p>
+<ul>
+<li><a href="usage/configuration/config_documentation.html#enable_registration_captcha"><code>enable_registration_captcha</code></a></li>
+<li><a href="usage/configuration/config_documentation.html#registrations_require_3pid"><code>registrations_require_3pid</code></a></li>
+<li><a href="usage/configuration/config_documentation.html#registration_requires_token"><code>registration_requires_token</code></a></li>
+</ul>
+<p>(In order to enable registration without any verification, you must also set
+<a href="usage/configuration/config_documentation.html#enable_registration_without_verification"><code>enable_registration_without_verification</code></a>.)</p>
+<p>Note that even if this setting is disabled, new accounts can still be created
+via the admin API if
+<a href="usage/configuration/config_documentation.html#registration_shared_secret"><code>registration_shared_secret</code></a> is set.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">enable_registration: true
</code></pre>
<hr />
<h3 id="enable_registration_without_verification"><a class="header" href="#enable_registration_without_verification"><code>enable_registration_without_verification</code></a></h3>
<p>Enable registration without email or captcha verification. Note: this option is <em>not</em> recommended,
-as registration without verification is a known vector for spam and abuse. Defaults to false. Has no effect
-unless <code>enable_registration</code> is also enabled.</p>
+as registration without verification is a known vector for spam and abuse. Defaults to <code>false</code>. Has no effect
+unless <a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a> is also enabled.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">enable_registration_without_verification: true
</code></pre>
<hr />
-<h3 id="session_lifetime"><a class="header" href="#session_lifetime"><code>session_lifetime</code></a></h3>
-<p>Time that a user's session remains valid for, after they log in.</p>
-<p>Note that this is not currently compatible with guest logins.</p>
-<p>Note also that this is calculated at login time: changes are not applied retrospectively to users who have already
-logged in.</p>
-<p>By default, this is infinite.</p>
-<p>Example configuration:</p>
-<pre><code class="language-yaml">session_lifetime: 24h
-</code></pre>
-<hr />
-<h3 id="refresh_access_token_lifetime"><a class="header" href="#refresh_access_token_lifetime"><code>refresh_access_token_lifetime</code></a></h3>
-<p>Time that an access token remains valid for, if the session is using refresh tokens.</p>
-<p>For more information about refresh tokens, please see the <a href="usage/configuration/user_authentication/refresh_tokens.html">manual</a>.</p>
-<p>Note that this only applies to clients which advertise support for refresh tokens.</p>
-<p>Note also that this is calculated at login time and refresh time: changes are not applied to
-existing sessions until they are refreshed.</p>
-<p>By default, this is 5 minutes.</p>
-<p>Example configuration:</p>
-<pre><code class="language-yaml">refreshable_access_token_lifetime: 10m
-</code></pre>
-<hr />
-<h3 id="refresh_token_lifetime-24h"><a class="header" href="#refresh_token_lifetime-24h"><code>refresh_token_lifetime: 24h</code></a></h3>
-<p>Time that a refresh token remains valid for (provided that it is not
-exchanged for another one first).
-This option can be used to automatically log-out inactive sessions.
-Please see the manual for more information.</p>
-<p>Note also that this is calculated at login time and refresh time:
-changes are not applied to existing sessions until they are refreshed.</p>
-<p>By default, this is infinite.</p>
-<p>Example configuration:</p>
-<pre><code class="language-yaml">refresh_token_lifetime: 24h
-</code></pre>
-<hr />
-<h3 id="nonrefreshable_access_token_lifetime"><a class="header" href="#nonrefreshable_access_token_lifetime"><code>nonrefreshable_access_token_lifetime</code></a></h3>
-<p>Time that an access token remains valid for, if the session is NOT
-using refresh tokens.</p>
-<p>Please note that not all clients support refresh tokens, so setting
-this to a short value may be inconvenient for some users who will
-then be logged out frequently.</p>
-<p>Note also that this is calculated at login time: changes are not applied
-retrospectively to existing sessions for users that have already logged in.</p>
-<p>By default, this is infinite.</p>
-<p>Example configuration:</p>
-<pre><code class="language-yaml">nonrefreshable_access_token_lifetime: 24h
-</code></pre>
-<hr />
<h3 id="registrations_require_3pid"><a class="header" href="#registrations_require_3pid"><code>registrations_require_3pid</code></a></h3>
-<p>If this is set, the user must provide all of the specified types of 3PID when registering.</p>
+<p>If this is set, users must provide all of the specified types of 3PID when registering an account.</p>
+<p>Note that <a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a> must also be set to allow account registration.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">registrations_require_3pid:
- email
@@ -4894,20 +4895,36 @@ flow (overrides <code>registrations_require_3pid</code> if MSISDNs are set as re
<h3 id="registration_requires_token"><a class="header" href="#registration_requires_token"><code>registration_requires_token</code></a></h3>
<p>Require users to submit a token during registration.
Tokens can be managed using the admin <a href="usage/configuration/../administration/admin_api/registration_tokens.html">API</a>.
-Note that <code>enable_registration</code> must be set to true.
Disabling this option will not delete any tokens previously generated.
-Defaults to false. Set to true to enable.</p>
+Defaults to <code>false</code>. Set to <code>true</code> to enable.</p>
+<p>Note that <a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a> must also be set to allow account registration.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">registration_requires_token: true
</code></pre>
<hr />
<h3 id="registration_shared_secret"><a class="header" href="#registration_shared_secret"><code>registration_shared_secret</code></a></h3>
-<p>If set, allows registration of standard or admin accounts by anyone who
-has the shared secret, even if registration is otherwise disabled.</p>
+<p>If set, allows registration of standard or admin accounts by anyone who has the
+shared secret, even if <a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a> is not
+set.</p>
+<p>This is primarily intended for use with the <code>register_new_matrix_user</code> script
+(see <a href="usage/configuration/../../setup/installation.html#registering-a-user">Registering a user</a>);
+however, the interface is <a href="usage/configuration/../admin_api/register_api.html">documented</a>.</p>
+<p>See also <a href="usage/configuration/config_documentation.html#registration_shared_secret_path"><code>registration_shared_secret_path</code></a>.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">registration_shared_secret: <PRIVATE STRING>
</code></pre>
<hr />
+<h3 id="registration_shared_secret_path"><a class="header" href="#registration_shared_secret_path"><code>registration_shared_secret_path</code></a></h3>
+<p>An alternative to <a href="usage/configuration/config_documentation.html#registration_shared_secret"><code>registration_shared_secret</code></a>:
+allows the shared secret to be specified in an external file.</p>
+<p>The file should be a plain text file, containing only the shared secret.</p>
+<p>If this file does not exist, Synapse will create a new signing
+key on startup and store it in this file.</p>
+<p>Example configuration:</p>
+<pre><code class="language-yaml">registration_shared_secret_file: /path/to/secrets/file
+</code></pre>
+<p><em>Added in Synapse 1.67.0.</em></p>
+<hr />
<h3 id="bcrypt_rounds"><a class="header" href="#bcrypt_rounds"><code>bcrypt_rounds</code></a></h3>
<p>Set the number of bcrypt rounds used to generate password hash.
Larger numbers increase the work factor needed to generate the hash.
@@ -5075,6 +5092,54 @@ raise an error if the registration completes and the username conflicts.</p>
<pre><code class="language-yaml">inhibit_user_in_use_error: true
</code></pre>
<hr />
+<h2 id="user-session-management"><a class="header" href="#user-session-management">User session management</a></h2>
+<hr />
+<h3 id="session_lifetime"><a class="header" href="#session_lifetime"><code>session_lifetime</code></a></h3>
+<p>Time that a user's session remains valid for, after they log in.</p>
+<p>Note that this is not currently compatible with guest logins.</p>
+<p>Note also that this is calculated at login time: changes are not applied retrospectively to users who have already
+logged in.</p>
+<p>By default, this is infinite.</p>
+<p>Example configuration:</p>
+<pre><code class="language-yaml">session_lifetime: 24h
+</code></pre>
+<hr />
+<h3 id="refresh_access_token_lifetime"><a class="header" href="#refresh_access_token_lifetime"><code>refresh_access_token_lifetime</code></a></h3>
+<p>Time that an access token remains valid for, if the session is using refresh tokens.</p>
+<p>For more information about refresh tokens, please see the <a href="usage/configuration/user_authentication/refresh_tokens.html">manual</a>.</p>
+<p>Note that this only applies to clients which advertise support for refresh tokens.</p>
+<p>Note also that this is calculated at login time and refresh time: changes are not applied to
+existing sessions until they are refreshed.</p>
+<p>By default, this is 5 minutes.</p>
+<p>Example configuration:</p>
+<pre><code class="language-yaml">refreshable_access_token_lifetime: 10m
+</code></pre>
+<hr />
+<h3 id="refresh_token_lifetime-24h"><a class="header" href="#refresh_token_lifetime-24h"><code>refresh_token_lifetime: 24h</code></a></h3>
+<p>Time that a refresh token remains valid for (provided that it is not
+exchanged for another one first).
+This option can be used to automatically log-out inactive sessions.
+Please see the manual for more information.</p>
+<p>Note also that this is calculated at login time and refresh time:
+changes are not applied to existing sessions until they are refreshed.</p>
+<p>By default, this is infinite.</p>
+<p>Example configuration:</p>
+<pre><code class="language-yaml">refresh_token_lifetime: 24h
+</code></pre>
+<hr />
+<h3 id="nonrefreshable_access_token_lifetime"><a class="header" href="#nonrefreshable_access_token_lifetime"><code>nonrefreshable_access_token_lifetime</code></a></h3>
+<p>Time that an access token remains valid for, if the session is NOT
+using refresh tokens.</p>
+<p>Please note that not all clients support refresh tokens, so setting
+this to a short value may be inconvenient for some users who will
+then be logged out frequently.</p>
+<p>Note also that this is calculated at login time: changes are not applied
+retrospectively to existing sessions for users that have already logged in.</p>
+<p>By default, this is infinite.</p>
+<p>Example configuration:</p>
+<pre><code class="language-yaml">nonrefreshable_access_token_lifetime: 24h
+</code></pre>
+<hr />
<h2 id="metrics"><a class="header" href="#metrics">Metrics</a></h2>
<p>Config options related to metrics.</p>
<hr />
@@ -5132,7 +5197,7 @@ Defaults to https://matrix.org/report-usage-stats/push</p>
<h2 id="api-configuration"><a class="header" href="#api-configuration">API Configuration</a></h2>
<p>Config settings related to the client/server API</p>
<hr />
-<h3 id="room_prejoin_state"><a class="header" href="#room_prejoin_state"><code>room_prejoin_state:</code></a></h3>
+<h3 id="room_prejoin_state"><a class="header" href="#room_prejoin_state"><code>room_prejoin_state</code></a></h3>
<p>Controls for the state that is shared with users who receive an invite
to a room. By default, the following state event types are shared with users who
receive invites to the room:</p>
@@ -5216,7 +5281,9 @@ forms to work.</p>
<p>Config options relating to signing keys</p>
<hr />
<h3 id="signing_key_path"><a class="header" href="#signing_key_path"><code>signing_key_path</code></a></h3>
-<p>Path to the signing key to sign messages with.</p>
+<p>Path to the signing key to sign events and federation requests with.</p>
+<p><em>New in Synapse 1.67</em>: If this file does not exist, Synapse will create a new signing
+key on startup and store it in this file.</p>
<p>Example configuration:</p>
<pre><code class="language-yaml">signing_key_path: "CONFDIR/SERVERNAME.signing.key"
</code></pre>
@@ -5242,7 +5309,7 @@ are still valid. Defaults to 1d.</p>
<pre><code class="language-yaml">key_refresh_interval: 2d
</code></pre>
<hr />
-<h3 id="trusted_key_servers"><a class="header" href="#trusted_key_servers"><code>trusted_key_servers:</code></a></h3>
+<h3 id="trusted_key_servers"><a class="header" href="#trusted_key_servers"><code>trusted_key_servers</code></a></h3>
<p>The trusted servers to download signing keys from.</p>
<p>When we need to fetch a signing key, each server is tried in parallel.</p>
<p>Normally, the connection to the key server is validated via TLS certificates.
@@ -5296,14 +5363,12 @@ defaults to the server signing key.</p>
<h2 id="single-sign-on-integration"><a class="header" href="#single-sign-on-integration">Single sign-on integration</a></h2>
<p>The following settings can be used to make Synapse use a single sign-on
provider for authentication, instead of its internal password database.</p>
-<p>You will probably also want to set the following options to false to
+<p>You will probably also want to set the following options to <code>false</code> to
disable the regular login/registration flows:</p>
<ul>
-<li><code>enable_registration</code></li>
-<li><code>password_config.enabled</code></li>
+<li><a href="usage/configuration/config_documentation.html#enable_registration"><code>enable_registration</code></a></li>
+<li><a href="usage/configuration/config_documentation.html#password_config"><code>password_config.enabled</code></a></li>
</ul>
-<p>You will also want to investigate the settings under the "sso" configuration
-section below.</p>
<hr />
<h3 id="saml2_config"><a class="header" href="#saml2_config"><code>saml2_config</code></a></h3>
<p>Enable SAML2 for registration and login. Uses pysaml2. To learn more about pysaml and
@@ -7003,7 +7068,8 @@ to install Dex.</p>
<p>Set the Allowed Callback URLs to <code>[synapse public baseurl]/_synapse/client/oidc/callback</code></p>
</li>
<li>
-<p>Add a rule to add the <code>preferred_username</code> claim.</p>
+<p>Add a rule with any name to add the <code>preferred_username</code> claim.
+(See https://auth0.com/docs/customize/rules/create-rules for more information on how to create rules.)</p>
<details>
<summary>Code sample</summary>
<pre><code class="language-js">function addPersistenceAttribute(user, context, callback) {
@@ -9641,13 +9707,8 @@ sync with the database state.</p>
stream between all configured Synapse processes. Additionally, processes may
make HTTP requests to each other, primarily for operations which need to wait
for a reply ─ such as sending an event.</p>
-<p>Redis support was added in v1.13.0 with it becoming the recommended method in
-v1.18.0. It replaced the old direct TCP connections (which is deprecated as of
-v1.18.0) to the main process. With Redis, rather than all the workers connecting
-to the main process, all the workers and the main process connect to Redis,
-which relays replication commands between processes. This can give a significant
-cpu saving on the main process and will be a prerequisite for upcoming
-performance improvements.</p>
+<p>All the workers and the main process connect to Redis, which relays replication
+commands between processes.</p>
<p>If Redis support is enabled Synapse will use it as a shared cache, as well as a
pub/sub mechanism.</p>
<p>See the <a href="workers.html#architectural-diagram">Architectural diagram</a> section at the end for
@@ -9704,19 +9765,25 @@ worker_replication_secret: ""
redis:
enabled: true
</code></pre>
-<p>See the sample config for the full documentation of each option.</p>
+<p>See the <a href="usage/configuration/config_documentation.html">configuration manual</a> for the full documentation of each option.</p>
<p>Under <strong>no circumstances</strong> should the replication listener be exposed to the
-public internet; it has no authentication and is unencrypted.</p>
+public internet; replication traffic is:</p>
+<ul>
+<li>always unencrypted</li>
+<li>unauthenticated, unless <code>worker_replication_secret</code> is configured</li>
+</ul>
<h3 id="worker-configuration"><a class="header" href="#worker-configuration">Worker configuration</a></h3>
-<p>In the config file for each worker, you must specify the type of worker
-application (<code>worker_app</code>), and you should specify a unique name for the worker
-(<code>worker_name</code>). The currently available worker applications are listed below.
-You must also specify the HTTP replication endpoint that it should talk to on
-the main synapse process. <code>worker_replication_host</code> should specify the host of
-the main synapse and <code>worker_replication_http_port</code> should point to the HTTP
-replication port. If the worker will handle HTTP requests then the
-<code>worker_listeners</code> option should be set with a <code>http</code> listener, in the same way
-as the <code>listeners</code> option in the shared config.</p>
+<p>In the config file for each worker, you must specify:</p>
+<ul>
+<li>The type of worker (<code>worker_app</code>). The currently available worker applications are listed below.</li>
+<li>A unique name for the worker (<code>worker_name</code>).</li>
+<li>The HTTP replication endpoint that it should talk to on the main synapse process
+(<code>worker_replication_host</code> and <code>worker_replication_http_port</code>)</li>
+<li>If handling HTTP requests, a <code>worker_listeners</code> option with an <code>http</code>
+listener, in the same way as the <code>listeners</code> option in the shared config.</li>
+<li>If handling the <code>^/_matrix/client/v3/keys/upload</code> endpoint, the HTTP URI for
+the main process (<code>worker_main_http_uri</code>).</li>
+</ul>
<p>For example:</p>
<pre><code class="language-yaml">worker_app: synapse.app.generic_worker
worker_name: generic_worker1
@@ -9725,6 +9792,8 @@ worker_name: generic_worker1
worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093
+worker_main_http_uri: http://localhost:8008/
+
worker_listeners:
- type: http
port: 8083
@@ -9800,10 +9869,12 @@ information.</p>
^/_matrix/client/(api/v1|r0|v3|unstable)/search$
# Encryption requests
+# Note that ^/_matrix/client/(r0|v3|unstable)/keys/upload/ requires `worker_main_http_uri`
^/_matrix/client/(r0|v3|unstable)/keys/query$
^/_matrix/client/(r0|v3|unstable)/keys/changes$
^/_matrix/client/(r0|v3|unstable)/keys/claim$
^/_matrix/client/(r0|v3|unstable)/room_keys/
+^/_matrix/client/(r0|v3|unstable)/keys/upload/
# Registration/login requests
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
@@ -9894,8 +9965,7 @@ of events, then a dedicated set of workers can be provisioned to limit the
effects of bursts of events from that bridge on events sent by normal users.</p>
<h4 id="stream-writers"><a class="header" href="#stream-writers">Stream writers</a></h4>
<p>Additionally, the writing of specific streams (such as events) can be moved off
-of the main process to a particular worker.
-(This is only supported with Redis-based replication.)</p>
+of the main process to a particular worker.</p>
<p>To enable this, the worker must have a HTTP replication listener configured,
have a <code>worker_name</code> and be listed in the <code>instance_map</code> config. The same worker
can handle multiple streams, but unless otherwise documented, each stream can only
@@ -10097,40 +10167,20 @@ endpoint as long as either this worker or the main process are configured to
handle it, and are online.</p>
<p>If <code>update_user_directory</code> is set to <code>false</code>, and this worker is not running,
the above endpoint may give outdated results.</p>
-<h3 id="synapseappfrontend_proxy"><a class="header" href="#synapseappfrontend_proxy"><code>synapse.app.frontend_proxy</code></a></h3>
-<p>Proxies some frequently-requested client endpoints to add caching and remove
-load from the main synapse. It can handle REST endpoints matching the following
-regular expressions:</p>
-<pre><code>^/_matrix/client/(r0|v3|unstable)/keys/upload
-</code></pre>
-<p>If <code>use_presence</code> is False in the homeserver config, it can also handle REST
-endpoints matching the following regular expressions:</p>
-<pre><code>^/_matrix/client/(api/v1|r0|v3|unstable)/presence/[^/]+/status
-</code></pre>
-<p>This "stub" presence handler will pass through <code>GET</code> request but make the
-<code>PUT</code> effectively a no-op.</p>
-<p>It will proxy any requests it cannot handle to the main synapse instance. It
-must therefore be configured with the location of the main instance, via
-the <code>worker_main_http_uri</code> setting in the <code>frontend_proxy</code> worker configuration
-file. For example:</p>
-<pre><code class="language-yaml">worker_main_http_uri: http://127.0.0.1:8008
-</code></pre>
<h3 id="historical-apps"><a class="header" href="#historical-apps">Historical apps</a></h3>
-<p><em>Note:</em> Historically there used to be more apps, however they have been
-amalgamated into a single <code>synapse.app.generic_worker</code> app. The remaining apps
-are ones that do specific processing unrelated to requests, e.g. the <code>pusher</code>
-that handles sending out push notifications for new events. The intention is for
-all these to be folded into the <code>generic_worker</code> app and to use config to define
-which processes handle the various proccessing such as push notifications.</p>
+<p>The following used to be separate worker application types, but are now
+equivalent to <code>synapse.app.generic_worker</code>:</p>
+<ul>
+<li><code>synapse.app.client_reader</code></li>
+<li><code>synapse.app.event_creator</code></li>
+<li><code>synapse.app.federation_reader</code></li>
+<li><code>synapse.app.frontend_proxy</code></li>
+<li><code>synapse.app.synchrotron</code></li>
+</ul>
<h2 id="migration-from-old-config"><a class="header" href="#migration-from-old-config">Migration from old config</a></h2>
-<p>There are two main independent changes that have been made: introducing Redis
-support and merging apps into <code>synapse.app.generic_worker</code>. Both these changes
-are backwards compatible and so no changes to the config are required, however
-server admins are encouraged to plan to migrate to Redis as the old style direct
-TCP replication config is deprecated.</p>
-<p>To migrate to Redis add the <code>redis</code> config as above, and optionally remove the
-TCP <code>replication</code> listener from master and <code>worker_replication_port</code> from worker
-config.</p>
+<p>A main change that has occurred is the merging of worker apps into
+<code>synapse.app.generic_worker</code>. This change is backwards compatible and so no
+changes to the config are required.</p>
<p>To migrate apps to use <code>synapse.app.generic_worker</code> simply update the
<code>worker_app</code> option in the worker configs, and where worker are started (e.g.
in systemd service files, but not required for synctl).</p>
@@ -10210,6 +10260,8 @@ worker_name: generic_worker1
worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093
+worker_main_http_uri: http://localhost:8008/
+
worker_listeners:
- type: http
port: 8083
@@ -10840,9 +10892,9 @@ a purge id:</p>
non-interactive way. This is generally used for bootstrapping a Synapse
instance with administrator accounts.</p>
<p>To authenticate yourself to the server, you will need both the shared secret
-(<code>registration_shared_secret</code> in the homeserver configuration), and a
-one-time nonce. If the registration shared secret is not configured, this API
-is not enabled.</p>
+(<a href="admin_api/../configuration/config_documentation.html#registration_shared_secret"><code>registration_shared_secret</code></a>
+in the homeserver configuration), and a one-time nonce. If the registration
+shared secret is not configured, this API is not enabled.</p>
<p>To fetch the nonce, you need to request one from the API:</p>
<pre><code>> GET /_synapse/admin/v1/register
@@ -13360,7 +13412,13 @@ parts of the process.</p>
</li>
<li>
<p>Enable Synapse metrics:</p>
-<p>There are two methods of enabling metrics in Synapse.</p>
+<p>In <code>homeserver.yaml</code>, make sure <code>enable_metrics</code> is
+set to <code>True</code>.</p>
+</li>
+<li>
+<p>Enable the <code>/_synapse/metrics</code> Synapse endpoint that Prometheus uses to
+collect data:</p>
+<p>There are two methods of enabling the metrics endpoint in Synapse.</p>
<p>The first serves the metrics as a part of the usual web server and
can be enabled by adding the "metrics" resource to the existing
listener as such:</p>
@@ -13385,8 +13443,6 @@ over HTTP only, and will be available at <code>/_synapse/metrics</code>.</p>
bind_addresses:
- '0.0.0.0'
</code></pre>
-<p>For both options, you will need to ensure that <code>enable_metrics</code> is
-set to <code>True</code>.</p>
</li>
<li>
<p>Restart Synapse.</p>
@@ -13665,6 +13721,70 @@ consider using one of the following known implementations:</p>
<li><a href="https://github.com/matrix-org/panopticon">Matrix.org's Panopticon</a></li>
<li><a href="https://gitlab.com/famedly/company/devops/services/barad-dur">Famedly's Barad-dûr</a></li>
</ul>
+<div style="break-before: page; page-break-before: always;"></div><h1 id="monthly-active-users"><a class="header" href="#monthly-active-users">Monthly Active Users</a></h1>
+<p>Synapse can be configured to record the number of monthly active users (also referred to as MAU) on a given homeserver.
+For clarity's sake, MAU only tracks local users.</p>
+<p>Please note that the metrics recorded by the <a href="usage/administration/../../usage/administration/monitoring/reporting_homeserver_usage_statistics.html">Homeserver Usage Stats</a>
+are calculated differently. The <code>monthly_active_users</code> from the usage stats does not take into account any
+of the rules below, and counts any users who have made a request to the homeserver in the last 30 days.</p>
+<p>See the <a href="usage/administration/../../usage/configuration/config_documentation.html#limit_usage_by_mau">configuration manual</a> for details on how to configure MAU.</p>
+<h2 id="calculating-active-users"><a class="header" href="#calculating-active-users">Calculating active users</a></h2>
+<p>Individual user activity is measured in active days. If a user performs an action, the exact time of that action is then recorded. When
+calculating the MAU figure, any users with a recorded action in the last 30 days are considered part of the cohort. Days are measured
+as a rolling window from the current system time to 30 days ago.</p>
+<p>So for example, if Synapse were to calculate the active users on the 15th July at 13:25, it would include any activity from 15th June 13:25 onwards.</p>
+<p>A user is <strong>never</strong> considered active if they are either:</p>
+<ul>
+<li>Part of the trial day cohort (described below)</li>
+<li>Owned by an application service.
+<ul>
+<li>Note: This <strong>only</strong> covers users that are part of an application service <code>namespaces.users</code> registration. The namespace
+must also be marked as <code>exclusive</code>.</li>
+</ul>
+</li>
+</ul>
+<p>Otherwise, any request to Synapse will mark the user as active. Please note that registration will not mark a user as active <em>unless</em>
+they register with a 3pid that is included in the config field <code>mau_limits_reserved_threepids</code>.</p>
+<p>The Prometheus metric for MAU is refreshed every 5 minutes.</p>
+<p>Once an hour, Synapse checks to see if any users are inactive (with only activity timestamps later than 30 days). These users
+are removed from the active users cohort. If they then become active, they are immediately restored to the cohort.</p>
+<p>It is important to note that <strong>deactivated</strong> users are not immediately removed from the pool of active users, but as these users won't
+perform actions they will eventually be removed from the cohort.</p>
+<h3 id="trial-days"><a class="header" href="#trial-days">Trial days</a></h3>
+<p>If the config option <code>mau_trial_days</code> is set, a user must have been active this many days <strong>after</strong> registration to be active. A user is in the
+trial period if their registration timestamp (also known as the <code>creation_ts</code>) is less than <code>mau_trial_days</code> old.</p>
+<p>As an example, if <code>mau_trial_days</code> is set to <code>3</code> and a user is active <strong>after</strong> 3 days (72 hours from registration time) then they will be counted as active.</p>
+<p>The <code>mau_appservice_trial_days</code> config further extends this rule by applying different durations depending on the <code>appservice_id</code> of the user.
+Users registered by an application service will be recorded with an <code>appservice_id</code> matching the <code>id</code> key in the registration file for that service.</p>
+<h2 id="limiting-usage-of-the-homeserver-when-the-maximum-mau-is-reached"><a class="header" href="#limiting-usage-of-the-homeserver-when-the-maximum-mau-is-reached">Limiting usage of the homeserver when the maximum MAU is reached</a></h2>
+<p>If both config options <code>limit_usage_by_mau</code> and <code>max_mau_value</code> is set, and the current MAU value exceeds the maximum value, the
+homeserver will begin to block some actions.</p>
+<p>Individual users matching <strong>any</strong> of the below criteria never have their actions blocked:</p>
+<ul>
+<li>Considered part of the cohort of MAU users.</li>
+<li>Considered part of the trial period.</li>
+<li>Registered as a <code>support</code> user.</li>
+<li>Application service users if <code>track_appservice_user_ips</code> is NOT set.</li>
+</ul>
+<p>Please not that server admins are <strong>not</strong> exempt from blocking.</p>
+<p>The following actions are blocked when the MAU limit is exceeded:</p>
+<ul>
+<li>Logging in</li>
+<li>Sending events</li>
+<li>Creating rooms</li>
+<li>Syncing</li>
+</ul>
+<p>Registration is also blocked for all new signups <em>unless</em> the user is registering with a threepid included in the <code>mau_limits_reserved_threepids</code>
+config value.</p>
+<p>When a request is blocked, the response will have the <code>errcode</code> <code>M_RESOURCE_LIMIT_EXCEEDED</code>.</p>
+<h2 id="metrics-1"><a class="header" href="#metrics-1">Metrics</a></h2>
+<p>Synapse records several different prometheus metrics for MAU.</p>
+<p><code>synapse_admin_mau:current</code> records the current MAU figure for native (non-application-service) users.</p>
+<p><code>synapse_admin_mau:max</code> records the maximum MAU as dictated by the <code>max_mau_value</code> config value.</p>
+<p><code>synapse_admin_mau_current_mau_by_service</code> records the current MAU including application service users. The label <code>app_service</code> can be used
+to filter by a specific service ID. This <em>also</em> includes non-application-service users under <code>app_service=native</code> .</p>
+<p><code>synapse_admin_mau:registered_reserved_users</code> records the number of users specified in <code>mau_limits_reserved_threepids</code> which have
+registered accounts on the homeserver.</p>
<div style="break-before: page; page-break-before: always;"></div><h2 id="understanding-synapse-through-grafana-graphs"><a class="header" href="#understanding-synapse-through-grafana-graphs">Understanding Synapse through Grafana graphs</a></h2>
<p>It is possible to monitor much of the internal state of Synapse using <a href="https://prometheus.io">Prometheus</a>
metrics and <a href="https://grafana.com/">Grafana</a>.
@@ -14109,6 +14229,7 @@ pipx install poetry
</code></pre>
<p>but see poetry's <a href="https://python-poetry.org/docs/#installation">installation instructions</a>
for other installation methods.</p>
+<p>Synapse requires Poetry version 1.2.0 or later.</p>
<p>Next, open a terminal and install dependencies as follows:</p>
<pre><code class="language-sh">cd path/where/you/have/cloned/the/repository
poetry install --extras all
@@ -15053,6 +15174,22 @@ default value is the <strong>string</strong> <code>"FALSE"</code> - wh
in Python, evaluates to <code>True</code>.</p>
</li>
</ul>
+<h2 id="event_id-global-uniqueness"><a class="header" href="#event_id-global-uniqueness"><code>event_id</code> global uniqueness</a></h2>
+<p>In room versions <code>1</code> and <code>2</code> it's possible to end up with two events with the
+same <code>event_id</code> (in the same or different rooms). After room version <code>3</code>, that
+can only happen with a hash collision, which we basically hope will never
+happen.</p>
+<p>There are several places in Synapse and even Matrix APIs like <a href="https://spec.matrix.org/v1.1/server-server-api/#get_matrixfederationv1eventeventid"><code>GET /_matrix/federation/v1/event/{eventId}</code></a>
+where we assume that event IDs are globally unique.</p>
+<p>But hash collisions are still possible, and by treating event IDs as room
+scoped, we can reduce the possibility of a hash collision. When scoping
+<code>event_id</code> in the database schema, it should be also accompanied by <code>room_id</code>
+(<code>PRIMARY KEY (room_id, event_id)</code>) and lookups should be done through the pair
+<code>(room_id, event_id)</code>.</p>
+<p>There has been a lot of debate on this in places like
+https://github.com/matrix-org/matrix-spec-proposals/issues/2779 and
+<a href="https://github.com/matrix-org/matrix-spec-proposals/pull/2848">MSC2848</a> which
+has no resolution yet (as of 2022-09-01).</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="implementing-experimental-features-in-synapse"><a class="header" href="#implementing-experimental-features-in-synapse">Implementing experimental features in Synapse</a></h1>
<p>It can be desirable to implement "experimental" features which are disabled by
default and must be explicitly enabled via the Synapse configuration. This is
@@ -15263,11 +15400,7 @@ doesn't require poetry. (It's what we use in CI too). However, you could try
<code>poetry build</code> too.</p>
<h1 id="troubleshooting-4"><a class="header" href="#troubleshooting-4">Troubleshooting</a></h1>
<h2 id="check-the-version-of-poetry-with-poetry---version"><a class="header" href="#check-the-version-of-poetry-with-poetry---version">Check the version of poetry with <code>poetry --version</code>.</a></h2>
-<p>At the time of writing, the 1.2 series is beta only. We have seen some examples
-where the lockfiles generated by 1.2 prereleasese aren't interpreted correctly
-by poetry 1.1.x. For now, use poetry 1.1.14, which includes a critical
-<a href="https://github.com/python-poetry/poetry/pull/5973">change</a> needed to remain
-<a href="https://github.com/pypi/warehouse/pull/11775">compatible with PyPI</a>.</p>
+<p>The minimum version of poetry supported by Synapse is 1.2.</p>
<p>It can also be useful to check the version of <code>poetry-core</code> in use. If you've
installed <code>poetry</code> with <code>pipx</code>, try <code>pipx runpip poetry list | grep poetry-core</code>.</p>
<h2 id="clear-caches-poetry-cache-clear---all-pypi"><a class="header" href="#clear-caches-poetry-cache-clear---all-pypi">Clear caches: <code>poetry cache clear --all pypi</code>.</a></h2>
@@ -16370,13 +16503,42 @@ used if we don't have a chain cover index for the room (e.g. because we're in
the process of indexing it).</p>
<h2 id="chain-cover-index"><a class="header" href="#chain-cover-index">Chain Cover Index</a></h2>
<p>Synapse computes auth chain differences by pre-computing a "chain cover" index
-for the auth chain in a room, allowing efficient reachability queries like "is
-event A in the auth chain of event B". This is done by assigning every event a
-<em>chain ID</em> and <em>sequence number</em> (e.g. <code>(5,3)</code>), and having a map of <em>links</em>
-between chains (e.g. <code>(5,3) -> (2,4)</code>) such that A is reachable by B (i.e. <code>A</code>
-is in the auth chain of <code>B</code>) if and only if either:</p>
+for the auth chain in a room, allowing us to efficiently make reachability queries
+like "is event <code>A</code> in the auth chain of event <code>B</code>?". We could do this with an index
+that tracks all pairs <code>(A, B)</code> such that <code>A</code> is in the auth chain of <code>B</code>. However, this
+would be prohibitively large, scaling poorly as the room accumulates more state
+events.</p>
+<p>Instead, we break down the graph into <em>chains</em>. A chain is a subset of a DAG
+with the following property: for any pair of events <code>E</code> and <code>F</code> in the chain,
+the chain contains a path <code>E -> F</code> or a path <code>F -> E</code>. This forces a chain to be
+linear (without forks), e.g. <code>E -> F -> G -> ... -> H</code>. Each event in the chain
+is given a <em>sequence number</em> local to that chain. The oldest event <code>E</code> in the
+chain has sequence number 1. If <code>E</code> has a child <code>F</code> in the chain, then <code>F</code> has
+sequence number 2. If <code>E</code> has a grandchild <code>G</code> in the chain, then <code>G</code> has
+sequence number 3; and so on.</p>
+<p>Synapse ensures that each persisted event belongs to exactly one chain, and
+tracks how the chains are connected to one another. This allows us to
+efficiently answer reachability queries. Doing so uses less storage than
+tracking reachability on an event-by-event basis, particularly when we have
+fewer and longer chains. See</p>
+<blockquote>
+<p>Jagadish, H. (1990). <a href="https://doi.org/10.1145/99935.99944">A compression technique to materialize transitive closure</a>.
+<em>ACM Transactions on Database Systems (TODS)</em>, 15*(4)*, 558-598.</p>
+</blockquote>
+<p>for the original idea or</p>
+<blockquote>
+<p>Y. Chen, Y. Chen, <a href="https://doi.org/10.1109/ICDE.2008.4497498">An efficient algorithm for answering graph
+reachability queries</a>,
+in: 2008 IEEE 24th International Conference on Data Engineering, April 2008,
+pp. 893–902. (PDF available via <a href="https://scholar.google.com/scholar?q=Y.%20Chen,%20Y.%20Chen,%20An%20efficient%20algorithm%20for%20answering%20graph%20reachability%20queries,%20in:%202008%20IEEE%2024th%20International%20Conference%20on%20Data%20Engineering,%20April%202008,%20pp.%20893902.">Google Scholar</a>.)</p>
+</blockquote>
+<p>for a more modern take.</p>
+<p>In practical terms, the chain cover assigns every event a
+<em>chain ID</em> and <em>sequence number</em> (e.g. <code>(5,3)</code>), and maintains a map of <em>links</em>
+between events in chains (e.g. <code>(5,3) -> (2,4)</code>) such that <code>A</code> is reachable by <code>B</code>
+(i.e. <code>A</code> is in the auth chain of <code>B</code>) if and only if either:</p>
<ol>
-<li>A and B have the same chain ID and <code>A</code>'s sequence number is less than <code>B</code>'s
+<li><code>A</code> and <code>B</code> have the same chain ID and <code>A</code>'s sequence number is less than <code>B</code>'s
sequence number; or</li>
<li>there is a link <code>L</code> between <code>B</code>'s chain ID and <code>A</code>'s chain ID such that
<code>L.start_seq_no</code> <= <code>B.seq_no</code> and <code>A.seq_no</code> <= <code>L.end_seq_no</code>.</li>
@@ -16385,8 +16547,9 @@ sequence number; or</li>
each chain to every other reachable chain (the transitive closure of the links
graph), and one where we remove redundant links (the transitive reduction of the
links graph) e.g. if we have chains <code>C3 -> C2 -> C1</code> then the link <code>C3 -> C1</code>
-would not be stored. Synapse uses the former implementations so that it doesn't
-need to recurse to test reachability between chains.</p>
+would not be stored. Synapse uses the former implementation so that it doesn't
+need to recurse to test reachability between chains. This trades-off extra storage
+in order to save CPU cycles and DB queries.</p>
<h3 id="example-6"><a class="header" href="#example-6">Example</a></h3>
<p>An example auth graph would look like the following, where chains have been
formed based on type/state_key and are denoted by colour and are labelled with
|
