summary refs log tree commit diff
path: root/latest/print.html
diff options
context:
space:
mode:
Diffstat (limited to 'latest/print.html')
-rw-r--r--latest/print.html549
1 files changed, 283 insertions, 266 deletions
diff --git a/latest/print.html b/latest/print.html

index 7c49231aac..4c4d3b1b0e 100644 --- a/latest/print.html +++ b/latest/print.html
@@ -4,55 +4,31 @@ <!-- Book generated using mdBook --> <meta charset="UTF-8"> <title>Synapse</title> - <meta name="robots" content="noindex" /> - - - - <!-- Custom HTML head --> - - - <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="theme-color" content="#ffffff" /> - <link rel="icon" href="favicon.svg"> - - <link rel="shortcut icon" href="favicon.png"> - <link rel="stylesheet" href="css/variables.css"> <link rel="stylesheet" href="css/general.css"> <link rel="stylesheet" href="css/chrome.css"> - <link rel="stylesheet" href="css/print.css" media="print"> - - <!-- Fonts --> <link rel="stylesheet" href="FontAwesome/css/font-awesome.css"> - <link rel="stylesheet" href="fonts/fonts.css"> - - <!-- Highlight.js Stylesheets --> <link rel="stylesheet" href="highlight.css"> <link rel="stylesheet" href="tomorrow-night.css"> <link rel="stylesheet" href="ayu-highlight.css"> <!-- Custom theme stylesheets --> - <link rel="stylesheet" href="docs/website_files/table-of-contents.css"> - <link rel="stylesheet" href="docs/website_files/remove-nav-buttons.css"> - <link rel="stylesheet" href="docs/website_files/indent-section-headers.css"> - - - </head> <body> <!-- Provide site root to javascript --> @@ -101,7 +77,7 @@ <nav id="sidebar" class="sidebar" aria-label="Table of contents"> <div class="sidebar-scrollbox"> - <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/index.html">Single-Sign On</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/cas.html">CAS</a></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/refresh_tokens.html">Refresh Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="development/url_previews.html">URL Previews</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules/index.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/writing_a_module.html">Writing a module</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/spam_checker_callbacks.html">Spam checker callbacks</a></li><li class="chapter-item expanded "><a href="modules/third_party_rules_callbacks.html">Third-party rules callbacks</a></li><li class="chapter-item expanded "><a href="modules/presence_router_callbacks.html">Presence router callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_validity_callbacks.html">Account validity callbacks</a></li><li class="chapter-item expanded "><a href="modules/password_auth_provider_callbacks.html">Password auth provider callbacks</a></li><li class="chapter-item expanded "><a href="modules/background_update_controller_callbacks.html">Background update controller callbacks</a></li><li class="chapter-item expanded "><a href="modules/porting_legacy_module.html">Porting a legacy module to the new interface</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/background_updates.html">Background Updates</a></li><li class="chapter-item expanded "><a href="admin_api/delete_group.html">Delete Group</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/federation.html">Federation</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li class="chapter-item expanded "><a href="usage/administration/understanding_synapse_through_grafana_graphs.html">Understanding Synapse Through Grafana Graphs</a></li><li class="chapter-item expanded "><a href="usage/administration/useful_sql_for_admins.html">Useful SQL for Admins</a></li><li class="chapter-item expanded "><a href="usage/administration/database_maintenance_tools.html">Database Maintenance Tools</a></li><li class="chapter-item expanded "><a href="usage/administration/state_groups.html">State Groups</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_faq.html">Admin FAQ</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/releases.html">Release Cycle</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/demo.html">Demo scripts</a></li></ol></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><a href="development/experimental_features.html">Experimental features</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li><li class="chapter-item expanded "><a href="other/running_synapse_on_single_board_computers.html">Running Synapse on a Single-Board Computer</a></li></ol> + <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/index.html">Single-Sign On</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/single_sign_on/cas.html">CAS</a></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/refresh_tokens.html">Refresh Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="development/url_previews.html">URL Previews</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules/index.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/writing_a_module.html">Writing a module</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/spam_checker_callbacks.html">Spam checker callbacks</a></li><li class="chapter-item expanded "><a href="modules/third_party_rules_callbacks.html">Third-party rules callbacks</a></li><li class="chapter-item expanded "><a href="modules/presence_router_callbacks.html">Presence router callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_validity_callbacks.html">Account validity callbacks</a></li><li class="chapter-item expanded "><a href="modules/password_auth_provider_callbacks.html">Password auth provider callbacks</a></li><li class="chapter-item expanded "><a href="modules/background_update_controller_callbacks.html">Background update controller callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_data_callbacks.html">Account data callbacks</a></li><li class="chapter-item expanded "><a href="modules/porting_legacy_module.html">Porting a legacy module to the new interface</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/background_updates.html">Background Updates</a></li><li class="chapter-item expanded "><a href="admin_api/delete_group.html">Delete Group</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/federation.html">Federation</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li class="chapter-item expanded "><a href="usage/administration/understanding_synapse_through_grafana_graphs.html">Understanding Synapse Through Grafana Graphs</a></li><li class="chapter-item expanded "><a href="usage/administration/useful_sql_for_admins.html">Useful SQL for Admins</a></li><li class="chapter-item expanded "><a href="usage/administration/database_maintenance_tools.html">Database Maintenance Tools</a></li><li class="chapter-item expanded "><a href="usage/administration/state_groups.html">State Groups</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_faq.html">Admin FAQ</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/releases.html">Release Cycle</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/demo.html">Demo scripts</a></li></ol></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><a href="development/experimental_features.html">Experimental features</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li><li class="chapter-item expanded "><a href="other/running_synapse_on_single_board_computers.html">Running Synapse on a Single-Board Computer</a></li></ol> </div> <div id="sidebar-resize-handle" class="sidebar-resize-handle"></div> </nav> @@ -109,7 +85,6 @@ <div id="page-wrapper" class="page-wrapper"> <div class="page"> - <div id="menu-bar-hover-placeholder"></div> <div id="menu-bar" class="menu-bar sticky bordered"> <div class="left-buttons"> @@ -126,32 +101,23 @@ <li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li> <li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li> </ul> - <button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar"> <i class="fa fa-search"></i> </button> - </div> <h1 class="menu-title">Synapse</h1> <div class="right-buttons"> - <a href="print.html" title="Print this book" aria-label="Print this book"> <i id="print-button" class="fa fa-print"></i> </a> - - <a href="https://github.com/matrix-org/synapse" title="Git repository" aria-label="Git repository"> <i id="git-repository-button" class="fa fa-github"></i> </a> - - - </div> </div> - <div id="search-wrapper" class="hidden"> <form id="searchbar-outer" class="searchbar-outer"> <input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header"> @@ -162,8 +128,6 @@ </ul> </div> </div> - - <!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM --> <script type="text/javascript"> document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible'); @@ -180,7 +144,7 @@ <nav class="pagetoc"></nav> </div> - <div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="introduction"><a class="header" href="#introduction">Introduction</a></h1> + <div style="break-before: page; page-break-before: always;"></div><h1 id="introduction"><a class="header" href="#introduction">Introduction</a></h1> <p>Welcome to the documentation repository for Synapse, a <a href="https://matrix.org">Matrix</a> homeserver implementation developed by the matrix.org core team.</p> @@ -265,7 +229,7 @@ reach out to us over email at: support (at) matrix.org</p> <p>If you've found a security issue in Synapse or any other Matrix.org Foundation project, please report it to us in accordance with our <a href="https://www.matrix.org/security-disclosure-policy/">Security Disclosure Policy</a>. Thank you!</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="installation-instructions"><a class="header" href="#installation-instructions">Installation Instructions</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="installation-instructions"><a class="header" href="#installation-instructions">Installation Instructions</a></h1> <h2 id="choosing-your-server-name"><a class="header" href="#choosing-your-server-name">Choosing your server name</a></h2> <p>It is important to choose the name for your server before you install Synapse, because it cannot be changed later.</p> @@ -668,7 +632,7 @@ failing, e.g.:</p> </code></pre> <p>If you have any other problems, feel free to ask in <a href="https://matrix.to/#/#synapse:matrix.org">#synapse:matrix.org</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="using-postgres"><a class="header" href="#using-postgres">Using Postgres</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="using-postgres"><a class="header" href="#using-postgres">Using Postgres</a></h1> <p>Synapse supports PostgreSQL versions 10 or later.</p> <h2 id="install-postgres-client-libraries"><a class="header" href="#install-postgres-client-libraries">Install postgres client libraries</a></h2> <p>Synapse will require the python postgres client library in order to @@ -869,7 +833,7 @@ downgraded and then upgraded again.</p> <p>To fix the issue shut down Synapse (including any and all workers) and run the SQL command included in the error message. Once done Synapse should start successfully.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="using-a-reverse-proxy-with-synapse"><a class="header" href="#using-a-reverse-proxy-with-synapse">Using a reverse proxy with Synapse</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="using-a-reverse-proxy-with-synapse"><a class="header" href="#using-a-reverse-proxy-with-synapse">Using a reverse proxy with Synapse</a></h1> <p>It is recommended to put a reverse proxy such as <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html">nginx</a>, <a href="https://httpd.apache.org/docs/current/mod/mod_proxy_http.html">Apache</a>, @@ -1095,7 +1059,7 @@ Each configured HTTP listener has a <code>/health</code> endpoint which always r <code>/_synapse/admin</code>. These require authentication through an access token of an admin user. However as access to these endpoints grants the caller a lot of power, we do not recommend exposing them to the public internet without good reason.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="using-a-forward-proxy-with-synapse"><a class="header" href="#using-a-forward-proxy-with-synapse">Using a forward proxy with Synapse</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="using-a-forward-proxy-with-synapse"><a class="header" href="#using-a-forward-proxy-with-synapse">Using a forward proxy with Synapse</a></h1> <p>You can use Synapse with a forward or outbound proxy. An example of when this is necessary is in corporate environments behind a DMZ (demilitarized zone). Synapse supports routing outbound HTTP(S) requests via a proxy. Only HTTP(S) @@ -1170,7 +1134,7 @@ in Synapse can be deactivated.</p> <a href="setup/../usage/configuration/homeserver_sample_config.html">homserver.yaml</a>.</p> <pre><code class="language-yaml">use_insecure_ssl_client_just_for_testing_do_not_use: true </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="overview-1"><a class="header" href="#overview-1">Overview</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="overview-1"><a class="header" href="#overview-1">Overview</a></h1> <p>This document explains how to enable VoIP relaying on your homeserver with TURN.</p> <p>The synapse Matrix homeserver supports integration with TURN server via the @@ -1490,7 +1454,7 @@ Matrix clients!</p> entry in the results.</p> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="delegation-of-incoming-federation-traffic"><a class="header" href="#delegation-of-incoming-federation-traffic">Delegation of incoming federation traffic</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="delegation-of-incoming-federation-traffic"><a class="header" href="#delegation-of-incoming-federation-traffic">Delegation of incoming federation traffic</a></h1> <p>In the following documentation, we use the term <code>server_name</code> to refer to that setting in your homeserver configuration file. It appears at the ends of user ids, and tells other homeservers where they can find your server.</p> @@ -1562,7 +1526,7 @@ find it using delegation.</p> idea, since it saves handling TLS traffic in Synapse. See <a href="reverse_proxy.html">the reverse proxy documentation</a> for information on setting up a reverse proxy.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="upgrading-synapse"><a class="header" href="#upgrading-synapse">Upgrading Synapse</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="upgrading-synapse"><a class="header" href="#upgrading-synapse">Upgrading Synapse</a></h1> <p>Before upgrading check if any special steps are required to upgrade from the version you currently have installed to the current version of Synapse. The extra instructions that may be required are listed later in @@ -1640,6 +1604,48 @@ dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb </code></pre> </li> </ul> +<h1 id="upgrading-to-v1570"><a class="header" href="#upgrading-to-v1570">Upgrading to v1.57.0</a></h1> +<h2 id="changes-to-database-schema-for-application-services"><a class="header" href="#changes-to-database-schema-for-application-services">Changes to database schema for application services</a></h2> +<p>Synapse v1.57.0 includes a <a href="https://github.com/matrix-org/synapse/pull/12209">change</a> to the +way transaction IDs are managed for application services. If your deployment uses a dedicated +worker for application service traffic, <strong>it must be stopped</strong> when the database is upgraded +(which normally happens when the main process is upgraded), to ensure the change is made safely +without any risk of reusing transaction IDs.</p> +<p>Deployments which do not use separate worker processes can be upgraded as normal. Similarly, +deployments where no application services are in use can be upgraded as normal.</p> +<details> +<summary><b>Recovering from an incorrect upgrade</b></summary> +<p>If the database schema is upgraded <em>without</em> stopping the worker responsible +for AS traffic, then the following error may be given when attempting to start +a Synapse worker or master process:</p> +<pre><code>********************************************************************************** + Error during initialisation: + + Postgres sequence 'application_services_txn_id_seq' is inconsistent with associated + table 'application_services_txns'. This can happen if Synapse has been downgraded and + then upgraded again, or due to a bad migration. + + To fix this error, shut down Synapse (including any and all workers) + and run the following SQL: + + SELECT setval('application_services_txn_id_seq', ( + SELECT GREATEST(MAX(txn_id), 0) FROM application_services_txns + )); + + See docs/postgres.md for more information. + + There may be more information in the logs. +********************************************************************************** +</code></pre> +<p>This error may also be seen if Synapse is <em>downgraded</em> to an earlier version, +and then upgraded again to v1.57.0 or later.</p> +<p>In either case:</p> +<ol> +<li>Ensure that the worker responsible for AS traffic is stopped.</li> +<li>Run the SQL command given in the error message via <code>psql</code>.</li> +</ol> +<p>Synapse should then start correctly.</p> +</details> <h1 id="upgrading-to-v1560"><a class="header" href="#upgrading-to-v1560">Upgrading to v1.56.0</a></h1> <h2 id="open-registration-without-verification-is-now-disabled-by-default"><a class="header" href="#open-registration-without-verification-is-now-disabled-by-default">Open registration without verification is now disabled by default</a></h2> <p>Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config @@ -1662,15 +1668,15 @@ for more information and instructions on how to fix a database with incorrect va <a href="https://github.com/matrix-org/synapse/pull/12140">has been made</a> an <a href="https://packaging.python.org/en/latest/specifications/entry-points/">entry point</a> and no longer exists at the root of Synapse's source tree. If you wish to use -<code>synctl</code> to manage your homeserver, you should invoke <code>synctl</code> directly, e.g. -<code>synctl start</code> instead of <code>./synctl start</code> or <code>/path/to/synctl start</code>. </p> +<code>synctl</code> to manage your homeserver, you should invoke <code>synctl</code> directly, e.g. +<code>synctl start</code> instead of <code>./synctl start</code> or <code>/path/to/synctl start</code>.</p> <p>You will need to ensure <code>synctl</code> is on your <code>PATH</code>.</p> <ul> <li>This is automatically the case when using <a href="https://packages.matrix.org/debian/">Debian packages</a> or <a href="https://hub.docker.com/r/matrixdotorg/synapse">docker images</a> provided by Matrix.org.</li> -<li>When installing from a wheel, sdist, or PyPI, a <code>synctl</code> executable is added +<li>When installing from a wheel, sdist, or PyPI, a <code>synctl</code> executable is added to your Python installation's <code>bin</code>. This should be on your <code>PATH</code> automatically, though you might need to activate a virtual environment depending on how you installed Synapse.</li> @@ -1684,7 +1690,7 @@ please upgrade Mjolnir to version 1.3.2 or later before upgrading Synapse.</p> <p>This release removes support for the <code>structured: true</code> logging configuration which was deprecated in Synapse v1.23.0. If your logging configuration contains <code>structured: true</code> then it should be modified based on the -<a href="structured_logging.html">structured logging documentation</a>.</p> +<a href="https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration">structured logging documentation</a>.</p> <h1 id="upgrading-to-v1530"><a class="header" href="#upgrading-to-v1530">Upgrading to v1.53.0</a></h1> <h2 id="dropping-support-for-webclient-listeners-and-non-https-web_client_location"><a class="header" href="#dropping-support-for-webclient-listeners-and-non-https-web_client_location">Dropping support for <code>webclient</code> listeners and non-HTTP(S) <code>web_client_location</code></a></h2> <p>Per the deprecation notice in Synapse v1.51.0, listeners of type <code>webclient</code> @@ -1694,8 +1700,8 @@ configuration error. Since the <code>webclient</code> listener is no longer supp setting only applies to the root path <code>/</code> of Synapse's web server and no longer the <code>/_matrix/client/</code> path.</p> <h2 id="stablisation-of-msc3231"><a class="header" href="#stablisation-of-msc3231">Stablisation of MSC3231</a></h2> -<p>The unstable validity-check endpoint for the -<a href="https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv1registermloginregistration_tokenvalidity">Registration Tokens</a> +<p>The unstable validity-check endpoint for the +<a href="https://spec.matrix.org/v1.2/client-server-api/#get_matrixclientv1registermloginregistration_tokenvalidity">Registration Tokens</a> feature has been stabilised and moved from:</p> <p><code>/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity</code></p> <p>to:</p> @@ -1703,9 +1709,9 @@ feature has been stabilised and moved from:</p> <p>Please update any relevant reverse proxy or firewall configurations appropriately.</p> <h2 id="time-based-cache-expiry-is-now-enabled-by-default"><a class="header" href="#time-based-cache-expiry-is-now-enabled-by-default">Time-based cache expiry is now enabled by default</a></h2> <p>Formerly, entries in the cache were not evicted regardless of whether they were accessed after storing. -This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed. -To change the default behavior, go to the <code>caches</code> section of the config and change the <code>expire_caches</code> and -<code>cache_entry_ttl</code> flags as necessary. Please note that these flags replace the <code>expiry_time</code> flag in the config.<br /> +This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed. +To change the default behavior, go to the <code>caches</code> section of the config and change the <code>expire_caches</code> and +<code>cache_entry_ttl</code> flags as necessary. Please note that these flags replace the <code>expiry_time</code> flag in the config. The <code>expiry_time</code> flag will still continue to work, but it has been deprecated and will be removed in the future.</p> <h2 id="deprecation-of-capability-orgmatrixmsc3283"><a class="header" href="#deprecation-of-capability-orgmatrixmsc3283">Deprecation of <code>capability</code> <code>org.matrix.msc3283.*</code></a></h2> <p>The <code>capabilities</code> of MSC3283 from the REST API <code>/_matrix/client/r0/capabilities</code> @@ -2150,7 +2156,7 @@ lock down external access to the Admin API endpoints.</p> <p>This release deprecates use of the <code>structured: true</code> logging configuration for structured logging. If your logging configuration contains <code>structured: true</code> then it should be modified based on the -<a href="structured_logging.html">structured logging documentation</a>.</p> +<a href="https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration">structured logging documentation</a>.</p> <p>The <code>structured</code> and <code>drains</code> logging options are now deprecated and should be replaced by standard logging configuration of <code>handlers</code> and <code>formatters</code>.</p> @@ -2894,7 +2900,7 @@ longer to restart than usual as it reinitializes the database.</p> using room aliases or by being reinvited. Alternatively, if any other homeserver sends a message to a room that the homeserver was previously in the local HS will automatically rejoin the room.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="setting-up-federation"><a class="header" href="#setting-up-federation">Setting up federation</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="setting-up-federation"><a class="header" href="#setting-up-federation">Setting up federation</a></h1> <p>Federation is the process by which users on different servers can participate in the same room. For this to work, those other servers must be able to contact yours to send messages.</p> @@ -2947,10 +2953,10 @@ release of Synapse.</p> private federation, there is a script in the <code>demo</code> directory. This is mainly useful just for development purposes. See <a href="https://matrix-org.github.io/synapse/develop/development/demo.html">demo scripts</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="configuration-1"><a class="header" href="#configuration-1">Configuration</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="configuration-1"><a class="header" href="#configuration-1">Configuration</a></h1> <p>This section contains information on tweaking Synapse via the various options in the configuration file. A configuration file should have been generated when you <a href="usage/configuration/../../setup/installation.html">installed Synapse</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="homeserver-sample-configuration-file"><a class="header" href="#homeserver-sample-configuration-file">Homeserver Sample Configuration File</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="homeserver-sample-configuration-file"><a class="header" href="#homeserver-sample-configuration-file">Homeserver Sample Configuration File</a></h1> <p>Below is a sample homeserver configuration file. The homeserver configuration file can be tweaked to change the behaviour of your homeserver. A restart of the server is generally required to apply any changes made to this file.</p> @@ -3499,6 +3505,15 @@ templates: # #custom_template_directory: /path/to/custom/templates/ +# List of rooms to exclude from sync responses. This is useful for server +# administrators wishing to group users into a room without these users being able +# to see it from their client. +# +# By default, no room is excluded. +# +#exclude_rooms_from_sync: +# - !foo:example.com + # Message retention policy at the server level. # @@ -5742,7 +5757,7 @@ background_updates: # #default_batch_size: 50 </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="logging-sample-configuration-file"><a class="header" href="#logging-sample-configuration-file">Logging Sample Configuration File</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="logging-sample-configuration-file"><a class="header" href="#logging-sample-configuration-file">Logging Sample Configuration File</a></h1> <p>Below is a sample logging configuration file. This file can be tweaked to control how your homeserver will output logs. A restart of the server is generally required to apply any changes made to this file. The value of the <code>log_config</code> option in your homeserver @@ -5835,7 +5850,7 @@ root: disable_existing_loggers: false </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="structured-logging"><a class="header" href="#structured-logging">Structured Logging</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="structured-logging"><a class="header" href="#structured-logging">Structured Logging</a></h1> <p>A structured logging system can be useful when your logs are destined for a machine to parse and process. By maintaining its machine-readable characteristics, it enables more efficient searching and aggregations when consumed by software @@ -5904,76 +5919,7 @@ loggers: <p>The above logging config will set Synapse as 'INFO' logging level by default, with the SQL layer at 'WARNING', and will log JSON formatted messages to a remote endpoint at 10.1.2.3:9999.</p> -<h2 id="upgrading-from-legacy-structured-logging-configuration"><a class="header" href="#upgrading-from-legacy-structured-logging-configuration">Upgrading from legacy structured logging configuration</a></h2> -<p>Versions of Synapse prior to v1.54.0 automatically converted the legacy -structured logging configuration, which was deprecated in v1.23.0, to the standard -library logging configuration.</p> -<p>The following reference can be used to update your configuration. Based on the -drain <code>type</code>, we can pick a new handler:</p> -<ol> -<li>For a type of <code>console</code>, <code>console_json</code>, or <code>console_json_terse</code>: a handler -with a class of <code>logging.StreamHandler</code> and a <code>stream</code> of <code>ext://sys.stdout</code> -or <code>ext://sys.stderr</code> should be used.</li> -<li>For a type of <code>file</code> or <code>file_json</code>: a handler of <code>logging.FileHandler</code> with -a location of the file path should be used.</li> -<li>For a type of <code>network_json_terse</code>: a handler of <code>synapse.logging.RemoteHandler</code> -with the host and port should be used.</li> -</ol> -<p>Then based on the drain <code>type</code> we can pick a new formatter:</p> -<ol> -<li>For a type of <code>console</code> or <code>file</code> no formatter is necessary.</li> -<li>For a type of <code>console_json</code> or <code>file_json</code>: a formatter of -<code>synapse.logging.JsonFormatter</code> should be used.</li> -<li>For a type of <code>console_json_terse</code> or <code>network_json_terse</code>: a formatter of -<code>synapse.logging.TerseJsonFormatter</code> should be used.</li> -</ol> -<p>For each new handler and formatter they should be added to the logging configuration -and then assigned to either a logger or the root logger.</p> -<p>An example legacy configuration:</p> -<pre><code class="language-yaml">structured: true - -loggers: - synapse: - level: INFO - synapse.storage.SQL: - level: WARNING - -drains: - console: - type: console - location: stdout - file: - type: file_json - location: homeserver.log -</code></pre> -<p>Would be converted into a new configuration:</p> -<pre><code class="language-yaml">version: 1 - -formatters: - json: - class: synapse.logging.JsonFormatter - -handlers: - console: - class: logging.StreamHandler - stream: ext://sys.stdout - file: - class: logging.FileHandler - formatter: json - filename: homeserver.log - -loggers: - synapse: - level: INFO - handlers: [console, file] - synapse.storage.SQL: - level: WARNING -</code></pre> -<p>The new logging configuration is a bit more verbose, but significantly more -flexible. It allows for configuration that were not previously possible, such as -sending plain logs over the network, or using different handlers for different -modules.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="templates"><a class="header" href="#templates">Templates</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="templates"><a class="header" href="#templates">Templates</a></h1> <p>Synapse uses parametrised templates to generate the content of emails it sends and webpages it shows to users.</p> <p>By default, Synapse will use the templates listed <a href="https://github.com/matrix-org/synapse/tree/master/synapse/res/templates">here</a>. @@ -6255,7 +6201,7 @@ When rendering, this template is given two variables: </ul> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="user-authentication"><a class="header" href="#user-authentication">User Authentication</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="user-authentication"><a class="header" href="#user-authentication">User Authentication</a></h1> <p>Synapse supports multiple methods of authenticating users, either out-of-the-box or through custom pluggable authentication modules.</p> <p>Included in Synapse is support for authenticating users via:</p> @@ -6268,11 +6214,11 @@ authentication modules.</p> </ul> <p>Synapse can additionally be extended to support custom authentication schemes through optional &quot;password auth provider&quot; modules.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="single-sign-on"><a class="header" href="#single-sign-on">Single Sign-On</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="single-sign-on"><a class="header" href="#single-sign-on">Single Sign-On</a></h1> <p>Synapse supports single sign-on through the SAML, Open ID Connect or CAS protocols. LDAP and other login methods are supported through first and third-party password auth provider modules.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="configuring-synapse-to-authenticate-against-an-openid-connect-provider"><a class="header" href="#configuring-synapse-to-authenticate-against-an-openid-connect-provider">Configuring Synapse to authenticate against an OpenID Connect provider</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="configuring-synapse-to-authenticate-against-an-openid-connect-provider"><a class="header" href="#configuring-synapse-to-authenticate-against-an-openid-connect-provider">Configuring Synapse to authenticate against an OpenID Connect provider</a></h1> <p>Synapse can be configured to use an OpenID Connect Provider (OP) for authentication, instead of its own local password database.</p> <p>Any OP should work with Synapse, as long as it supports the authorization code @@ -6791,19 +6737,19 @@ needed to add OAuth2 capabilities to your Django projects. It supports display_name_template: &quot;{{ user.first_name }} {{ user.last_name }}&quot; email_template: &quot;{{ user.email }}&quot; </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="saml"><a class="header" href="#saml">SAML</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="saml"><a class="header" href="#saml">SAML</a></h1> <p>Synapse supports authenticating users via the <a href="https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language">Security Assertion Markup Language</a> (SAML) protocol natively.</p> <p>Please see the <code>saml2_config</code> and <code>sso</code> sections of the <a href="usage/configuration/user_authentication/single_sign_on/../../../configuration/homeserver_sample_config.html">Synapse configuration file</a> for more details.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="cas"><a class="header" href="#cas">CAS</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="cas"><a class="header" href="#cas">CAS</a></h1> <p>Synapse supports authenticating users via the <a href="https://en.wikipedia.org/wiki/Central_Authentication_Service">Central Authentication Service protocol</a> (CAS) natively.</p> <p>Please see the <code>cas_config</code> and <code>sso</code> sections of the <a href="usage/configuration/user_authentication/single_sign_on/../../../configuration/homeserver_sample_config.html">Synapse configuration file</a> for more details.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="sso-mapping-providers"><a class="header" href="#sso-mapping-providers">SSO Mapping Providers</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="sso-mapping-providers"><a class="header" href="#sso-mapping-providers">SSO Mapping Providers</a></h1> <p>A mapping provider is a Python class (loaded via a Python module) that works out how to map attributes of a SSO response to Matrix-specific user attributes. Details such as user ID localpart, displayname, and even avatar @@ -7052,7 +6998,7 @@ complete registration using methods from the <code>ModuleApi</code>.</p> <p>Synapse has a built-in SAML mapping provider if a custom provider isn't specified in the config. It is located at <a href="https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/saml.py"><code>synapse.handlers.saml.DefaultSamlMappingProvider</code></a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 style="color:red"> +<div style="break-before: page; page-break-before: always;"></div><h2 style="color:red"> This page of the Synapse documentation is now deprecated. For up to date documentation on setting up or writing a password auth provider module, please see <a href="modules/index.html">this page</a>. @@ -7166,7 +7112,7 @@ device ID), and the (now deactivated) access token.</p> wait for the <code>Awaitable</code> to complete, but the result is ignored.</p> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="jwt-login-type"><a class="header" href="#jwt-login-type">JWT Login Type</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="jwt-login-type"><a class="header" href="#jwt-login-type">JWT Login Type</a></h1> <p>Synapse comes with a non-standard login type to support <a href="https://en.wikipedia.org/wiki/JSON_Web_Token">JSON Web Tokens</a>. In general the documentation for @@ -7256,7 +7202,7 @@ eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.Ag71GT8v01UO3w80 </li> </ol> <p>You should now be able to use the returned access token to query the client API.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="refresh-tokens"><a class="header" href="#refresh-tokens">Refresh Tokens</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="refresh-tokens"><a class="header" href="#refresh-tokens">Refresh Tokens</a></h1> <p>Synapse supports refresh tokens since version 1.49 (some earlier versions had support for an earlier, experimental draft of <a href="https://github.com/matrix-org/matrix-doc/blob/main/proposals/2918-refreshtokens.md#msc2918-refresh-tokens">MSC2918</a> which is not compatible).</p> <h2 id="background-and-motivation"><a class="header" href="#background-and-motivation">Background and motivation</a></h2> <p>Synapse users' sessions are identified by <strong>access tokens</strong>; access tokens are @@ -7378,7 +7324,7 @@ above requirements will still hold.</em></p> <li><code>refresh_token_lifetime</code> should be set to <code>L</code>; and</li> <li><code>refreshable_access_token_lifetime</code> should be set to <code>L - S</code>.</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="overview-2"><a class="header" href="#overview-2">Overview</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="overview-2"><a class="header" href="#overview-2">Overview</a></h1> <p>A captcha can be enabled on your homeserver to help prevent bots from registering accounts. Synapse currently uses Google's reCAPTCHA service which requires API keys from Google.</p> @@ -7413,7 +7359,7 @@ CAPTCHA is sent. If the client is connecting through a proxy or load balancer, it may be required to use the <code>X-Forwarded-For</code> (XFF) header instead of the origin IP address. This can be configured using the <code>x_forwarded</code> directive in the listeners section of the <code>homeserver.yaml</code> configuration file.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="registering-an-application-service"><a class="header" href="#registering-an-application-service">Registering an Application Service</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="registering-an-application-service"><a class="header" href="#registering-an-application-service">Registering an Application Service</a></h1> <p>The registration of new application services depends on the homeserver used. In synapse, you need to create a new configuration file for your AS and add it to the list specified under the <code>app_service_config_files</code> config @@ -7439,7 +7385,7 @@ namespaces: <p><code>exclusive</code>: If enabled, only this application service is allowed to register users in its namespace(s). <code>group_id</code>: All users of this application service are dynamically joined to this group. This is useful for e.g user organisation or flairs.</p> <p>See the <a href="https://matrix.org/docs/spec/application_service/unstable.html">spec</a> for further details on how application services work.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="server-notices"><a class="header" href="#server-notices">Server Notices</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="server-notices"><a class="header" href="#server-notices">Server Notices</a></h1> <p>'Server Notices' are a new feature introduced in Synapse 0.30. They provide a channel whereby server administrators can send messages to users on the server.</p> <p>They are used as part of communication of the server polices (see @@ -7483,7 +7429,7 @@ displayname and avatar of the Server Notices user.</p> <h2 id="sending-notices"><a class="header" href="#sending-notices">Sending notices</a></h2> <p>To send server notices to users you can use the <a href="admin_api/server_notices.html">admin_api</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="support-in-synapse-for-tracking-agreement-to-server-terms-and-conditions"><a class="header" href="#support-in-synapse-for-tracking-agreement-to-server-terms-and-conditions">Support in Synapse for tracking agreement to server terms and conditions</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="support-in-synapse-for-tracking-agreement-to-server-terms-and-conditions"><a class="header" href="#support-in-synapse-for-tracking-agreement-to-server-terms-and-conditions">Support in Synapse for tracking agreement to server terms and conditions</a></h1> <p>Synapse 0.30 introduces support for tracking whether users have agreed to the terms and conditions set by the administrator of a server - and blocking access to the server until they have.</p> @@ -7644,7 +7590,7 @@ consent uri for that user.</p> <p>ensure that <code>public_baseurl</code> is set in <code>homeserver.yaml</code>, and gives the base URI that clients use to connect to the server. (It is used to construct <code>consent_uri</code> in the error.)</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="url-previews-1"><a class="header" href="#url-previews-1">URL Previews</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="url-previews-1"><a class="header" href="#url-previews-1">URL Previews</a></h1> <p>The <code>GET /_matrix/media/r0/preview_url</code> endpoint provides a generic preview API for URLs which outputs <a href="https://ogp.me/">Open Graph</a> responses (with some Matrix specific additions).</p> @@ -7728,7 +7674,7 @@ provider and saves the local media metadata.</li> <p>The in-memory cache expires after 1 hour.</p> <p>Expired entries in the database cache (and their associated media files) are deleted every 10 seconds. The default expiration time is 1 hour from download.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="user-directory-api-implementation"><a class="header" href="#user-directory-api-implementation">User Directory API Implementation</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="user-directory-api-implementation"><a class="header" href="#user-directory-api-implementation">User Directory API Implementation</a></h1> <p>The user directory is currently maintained based on the 'visible' users on this particular server - i.e. ones which your account shares a room with, or who are present in a publicly viewable room present on the server.</p> @@ -7783,7 +7729,7 @@ is a local user and <code>M</code> is a local or remote user. <code>L</code> and different, but this isn't enforced by a constraint.</p> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="message-retention-policies"><a class="header" href="#message-retention-policies">Message retention policies</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="message-retention-policies"><a class="header" href="#message-retention-policies">Message retention policies</a></h1> <p>Synapse admins can enable support for message retention policies on their homeserver. Message retention policies exist at a room level, follow the semantics described in @@ -7939,7 +7885,7 @@ space, it will start writing new data into where the purged data was.</p> operating system, the server admin needs to run <code>VACUUM FULL;</code> (or <code>VACUUM;</code> for SQLite databases) on Synapse's database (see the related <a href="https://www.postgresql.org/docs/current/sql-vacuum.html">PostgreSQL documentation</a>).</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="modules"><a class="header" href="#modules">Modules</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="modules"><a class="header" href="#modules">Modules</a></h1> <p>Synapse supports extending its functionality by configuring external modules.</p> <p><strong>Note</strong>: When using third-party modules, you effectively allow someone else to run custom code on your Synapse homeserver. Server admins are encouraged to verify the @@ -7981,7 +7927,7 @@ another part of Synapse's configuration file.</p> <li>presence router</li> <li>password auth providers</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="writing-a-module"><a class="header" href="#writing-a-module">Writing a module</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="writing-a-module"><a class="header" href="#writing-a-module">Writing a module</a></h1> <p>A module is a Python class that uses Synapse's module API to interact with the homeserver. It can register callbacks that Synapse will call on specific operations, as well as web resources to attach to Synapse's web server.</p> @@ -8005,7 +7951,7 @@ multiple modules implement it.</p> <h2 id="handling-the-modules-configuration"><a class="header" href="#handling-the-modules-configuration">Handling the module's configuration</a></h2> <p>A module can implement the following static method:</p> <pre><code class="language-python">@staticmethod -def parse_config(config: dict) -&gt; dict +def parse_config(config: dict) -&gt; Any </code></pre> <p>This method is given a dictionary resulting from parsing the YAML configuration for the module. It may modify it (for example by parsing durations expressed as strings (e.g. @@ -8042,7 +7988,7 @@ the callback name as the argument name and the function as its value. A <code>register_[...]_callbacks</code> method exists for each category.</p> <p>Callbacks for each category can be found on their respective page of the <a href="https://matrix-org.github.io/synapse">Synapse documentation website</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="spam-checker-callbacks"><a class="header" href="#spam-checker-callbacks">Spam checker callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="spam-checker-callbacks"><a class="header" href="#spam-checker-callbacks">Spam checker callbacks</a></h1> <p>Spam checker callbacks allow module developers to implement spam mitigation actions for Synapse instances. Spam checker callbacks can be registered using the module API's <code>register_spam_checker_callbacks</code> method.</p> @@ -8266,7 +8212,7 @@ class ListSpamChecker: async def check_event_for_spam(self, event: &quot;synapse.events.EventBase&quot;) -&gt; Union[bool, str]: return event.sender not in self.evil_users </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="third-party-rules-callbacks"><a class="header" href="#third-party-rules-callbacks">Third party rules callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="third-party-rules-callbacks"><a class="header" href="#third-party-rules-callbacks">Third party rules callbacks</a></h1> <p>Third party rules callbacks allow module developers to add extra checks to verify the validity of incoming events. Third party event rules callbacks can be registered using the module API's <code>register_third_party_rules_callbacks</code> method.</p> @@ -8445,6 +8391,17 @@ a server admin (and <code>False</code> otherwise). This latter <code>by_admin</c if the user is being reactivated, as this operation can only be performed through the admin API.</p> <p>If multiple modules implement this callback, Synapse runs them all in order.</p> +<h3 id="on_threepid_bind"><a class="header" href="#on_threepid_bind"><code>on_threepid_bind</code></a></h3> +<p><em>First introduced in Synapse v1.56.0</em></p> +<pre><code class="language-python">async def on_threepid_bind(user_id: str, medium: str, address: str) -&gt; None: +</code></pre> +<p>Called after creating an association between a local user and a third-party identifier +(email address, phone number). The module is given the Matrix ID of the user the +association is for, as well as the medium (<code>email</code> or <code>msisdn</code>) and address of the +third-party identifier.</p> +<p>Note that this callback is <em>not</em> called after a successful association on an <em>identity +server</em>.</p> +<p>If multiple modules implement this callback, Synapse runs them all in order.</p> <h2 id="example-1"><a class="header" href="#example-1">Example</a></h2> <p>The example below is a module that implements the third-party rules callback <code>check_event_allowed</code> to censor incoming messages as dictated by a third-party service.</p> @@ -8475,7 +8432,7 @@ class EventCensorer: event_dict[&quot;content&quot;] = new_event_content return event_dict </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="presence-router-callbacks"><a class="header" href="#presence-router-callbacks">Presence router callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="presence-router-callbacks"><a class="header" href="#presence-router-callbacks">Presence router callbacks</a></h1> <p>Presence router callbacks allow module developers to specify additional users (local or remote) to receive certain presence updates from local users. Presence router callbacks can be registered using the module API's <code>register_presence_router_callbacks</code> method.</p> @@ -8557,7 +8514,7 @@ class CustomPresenceRouter: return set() </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="account-validity-callbacks"><a class="header" href="#account-validity-callbacks">Account validity callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="account-validity-callbacks"><a class="header" href="#account-validity-callbacks">Account validity callbacks</a></h1> <p>Account validity callbacks allow module developers to add extra steps to verify the validity on an account, i.e. see if a user can be granted access to their account on the Synapse instance. Account validity callbacks can be registered using the module API's @@ -8586,7 +8543,7 @@ any of the subsequent implementations of this callback.</p> operations to keep track of them. (e.g. add them to a database table). The user is represented by their Matrix user ID.</p> <p>If multiple modules implement this callback, Synapse runs them all in order.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="password-auth-provider-callbacks"><a class="header" href="#password-auth-provider-callbacks">Password auth provider callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="password-auth-provider-callbacks"><a class="header" href="#password-auth-provider-callbacks">Password auth provider callbacks</a></h1> <p>Password auth providers offer a way for server administrators to integrate their Synapse installation with an external authentication system. The callbacks can be registered by using the Module API's <code>register_password_auth_provider_callbacks</code> method.</p> @@ -8821,7 +8778,7 @@ class MyAuthProvider: if self.credentials.get(username) == login_dict.get(&quot;password&quot;): return self.api.get_qualified_user_id(username) </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="background-update-controller-callbacks"><a class="header" href="#background-update-controller-callbacks">Background update controller callbacks</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="background-update-controller-callbacks"><a class="header" href="#background-update-controller-callbacks">Background update controller callbacks</a></h1> <p>Background update controller callbacks allow module developers to control (e.g. rate-limit) how database background updates are run. A database background update is an operation Synapse runs on its database in the background after it starts. It's usually used to run @@ -8868,7 +8825,98 @@ of the database. The module must return an integer representing the minimum numb elements to process in this iteration. This number must be at least 1, and is used to ensure that progress is always made.</p> <p>If this callback is not defined, Synapse will use a default value of 100.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="porting-an-existing-module-that-uses-the-old-interface"><a class="header" href="#porting-an-existing-module-that-uses-the-old-interface">Porting an existing module that uses the old interface</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="account-data-callbacks"><a class="header" href="#account-data-callbacks">Account data callbacks</a></h1> +<p>Account data callbacks allow module developers to react to changes of the account data +of local users. Account data callbacks can be registered using the module API's +<code>register_account_data_callbacks</code> method.</p> +<h2 id="callbacks-4"><a class="header" href="#callbacks-4">Callbacks</a></h2> +<p>The available account data callbacks are:</p> +<h3 id="on_account_data_updated"><a class="header" href="#on_account_data_updated"><code>on_account_data_updated</code></a></h3> +<p><em>First introduced in Synapse v1.57.0</em></p> +<pre><code class="language-python">async def on_account_data_updated( + user_id: str, + room_id: Optional[str], + account_data_type: str, + content: &quot;synapse.module_api.JsonDict&quot;, +) -&gt; None: +</code></pre> +<p>Called after user's account data has been updated. The module is given the +Matrix ID of the user whose account data is changing, the room ID the data is associated +with, the type associated with the change, as well as the new content. If the account +data is not associated with a specific room, then the room ID is <code>None</code>.</p> +<p>This callback is triggered when new account data is added or when the data associated with +a given type (and optionally room) changes. This includes deletion, since in Matrix, +deleting account data consists of replacing the data associated with a given type +(and optionally room) with an empty dictionary (<code>{}</code>).</p> +<p>Note that this doesn't trigger when changing the tags associated with a room, as these are +processed separately by Synapse.</p> +<p>If multiple modules implement this callback, Synapse runs them all in order.</p> +<h2 id="example-4"><a class="header" href="#example-4">Example</a></h2> +<p>The example below is a module that implements the <code>on_account_data_updated</code> callback, and +sends an event to an audit room when a user changes their account data.</p> +<pre><code class="language-python">import json +import attr +from typing import Any, Dict, Optional + +from synapse.module_api import JsonDict, ModuleApi +from synapse.module_api.errors import ConfigError + + +@attr.s(auto_attribs=True) +class CustomAccountDataConfig: + audit_room: str + sender: str + + +class CustomAccountDataModule: + def __init__(self, config: CustomAccountDataConfig, api: ModuleApi): + self.api = api + self.config = config + + self.api.register_account_data_callbacks( + on_account_data_updated=self.log_new_account_data, + ) + + @staticmethod + def parse_config(config: Dict[str, Any]) -&gt; CustomAccountDataConfig: + def check_in_config(param: str): + if param not in config: + raise ConfigError(f&quot;'{param}' is required&quot;) + + check_in_config(&quot;audit_room&quot;) + check_in_config(&quot;sender&quot;) + + return CustomAccountDataConfig( + audit_room=config[&quot;audit_room&quot;], + sender=config[&quot;sender&quot;], + ) + + async def log_new_account_data( + self, + user_id: str, + room_id: Optional[str], + account_data_type: str, + content: JsonDict, + ) -&gt; None: + content_raw = json.dumps(content) + msg_content = f&quot;{user_id} has changed their account data for type {account_data_type} to: {content_raw}&quot; + + if room_id is not None: + msg_content += f&quot; (in room {room_id})&quot; + + await self.api.create_and_send_event_into_room( + { + &quot;room_id&quot;: self.config.audit_room, + &quot;sender&quot;: self.config.sender, + &quot;type&quot;: &quot;m.room.message&quot;, + &quot;content&quot;: { + &quot;msgtype&quot;: &quot;m.text&quot;, + &quot;body&quot;: msg_content + } + } + ) +</code></pre> +<div style="break-before: page; page-break-before: always;"></div><h1 id="porting-an-existing-module-that-uses-the-old-interface"><a class="header" href="#porting-an-existing-module-that-uses-the-old-interface">Porting an existing module that uses the old interface</a></h1> <p>In order to port a module that uses Synapse's old module interface, its author needs to:</p> <ul> <li>ensure the module's callbacks are all asynchronous.</li> @@ -8885,7 +8933,7 @@ changes to the database should now be made by the module using the module API cl <p>The module's author should also update any example in the module's configuration to only use the new <code>modules</code> section in Synapse's configuration file (see <a href="modules/index.html#using-modules">this section</a> for more info).</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="scaling-synapse-via-workers"><a class="header" href="#scaling-synapse-via-workers">Scaling synapse via workers</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="scaling-synapse-via-workers"><a class="header" href="#scaling-synapse-via-workers">Scaling synapse via workers</a></h1> <p>For small instances it recommended to run Synapse in the default monolith mode. For larger instances where performance is a concern it can be helpful to split out functionality into multiple separate python processes. These processes are @@ -8907,7 +8955,7 @@ for a higher level overview.</p> feeds streams of newly written data between processes so they can be kept in sync with the database state.</p> <p>When configured to do so, Synapse uses a -<a href="https://redis.io/topics/pubsub">Redis pub/sub channel</a> to send the replication +<a href="https://redis.io/docs/manual/pubsub/">Redis pub/sub channel</a> to send the replication stream between all configured Synapse processes. Additionally, processes may make HTTP requests to each other, primarily for operations which need to wait for a reply ─ such as sending an event.</p> @@ -9379,7 +9427,7 @@ in systemd service files, but not required for synctl).</p> ==================================================================== Redis pub/sub channel </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h3 id="using-synctl-with-workers"><a class="header" href="#using-synctl-with-workers">Using synctl with workers</a></h3> +<div style="break-before: page; page-break-before: always;"></div><h3 id="using-synctl-with-workers"><a class="header" href="#using-synctl-with-workers">Using synctl with workers</a></h3> <p>If you want to use <code>synctl</code> to manage your synapse processes, you will need to create an an additional configuration file for the main synapse process. That configuration should look like this:</p> @@ -9402,7 +9450,7 @@ notifications.</p> <p>To manipulate a specific worker, you pass the -w option to synctl:</p> <pre><code class="language-sh">synctl -w $CONFIG/workers/worker1.yaml restart </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="setting-up-synapse-with-workers-and-systemd"><a class="header" href="#setting-up-synapse-with-workers-and-systemd">Setting up Synapse with Workers and Systemd</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="setting-up-synapse-with-workers-and-systemd"><a class="header" href="#setting-up-synapse-with-workers-and-systemd">Setting up Synapse with Workers and Systemd</a></h1> <p>This is a setup for managing synapse with systemd, including support for managing workers. It provides a <code>matrix-synapse</code> service for the master, as well as a <code>matrix-synapse-worker@</code> service template for any workers you @@ -9494,14 +9542,14 @@ systemctl restart matrix-synapse.target </code></pre> <p>In order to see their effect, you may run <code>systemd-analyze security matrix-synapse.service</code> before and after applying the hardening options to see the changes being applied at a glance.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="administration"><a class="header" href="#administration">Administration</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="administration"><a class="header" href="#administration">Administration</a></h1> <p>This section contains information on managing your Synapse homeserver. This includes:</p> <ul> <li>Managing users, rooms and media via the Admin API.</li> <li>Setting up metrics and monitoring to give you insight into your homeserver's health.</li> <li>Configuring structured logging.</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="the-admin-api"><a class="header" href="#the-admin-api">The Admin API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="the-admin-api"><a class="header" href="#the-admin-api">The Admin API</a></h1> <h2 id="authenticate-as-a-server-admin"><a class="header" href="#authenticate-as-a-server-admin">Authenticate as a server admin</a></h2> <p>Many of the API calls in the admin api will require an <code>access_token</code> for a server admin. (Note that a server admin is distinct from a room admin.)</p> @@ -9519,7 +9567,7 @@ providing the token as either a query parameter or a request header. To add it a </code></pre> <p>For more details on access tokens in Matrix, please refer to the complete <a href="https://matrix.org/docs/spec/client_server/r0.6.1#using-access-tokens">matrix spec documentation</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="account-validity-api"><a class="header" href="#account-validity-api">Account validity API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="account-validity-api"><a class="header" href="#account-validity-api">Account validity API</a></h1> <p>This API allows a server administrator to manage the validity of an account. To use it, you must enable the account validity feature (under <code>account_validity</code>) in Synapse's configuration.</p> @@ -9548,7 +9596,7 @@ milliseconds since epoch:</p> &quot;expiration_ts&quot;: 0 } </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="background-updates-api"><a class="header" href="#background-updates-api">Background Updates API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="background-updates-api"><a class="header" href="#background-updates-api">Background Updates API</a></h1> <p>This API allows a server administrator to manage the background updates being run against the database.</p> <h2 id="status"><a class="header" href="#status">Status</a></h2> @@ -9618,7 +9666,7 @@ background updates which won't be cancelled once started.</p> </ul> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="delete-a-local-group"><a class="header" href="#delete-a-local-group">Delete a local group</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="delete-a-local-group"><a class="header" href="#delete-a-local-group">Delete a local group</a></h1> <p>This API lets a server admin delete a local group. Doing so will kick all users out of the group so that their clients will correctly handle the group being deleted.</p> @@ -9627,7 +9675,7 @@ for a server admin: see <a href="admin_api/../usage/administration/admin_api">Ad <p>The API is:</p> <pre><code>POST /_synapse/admin/v1/delete_group/&lt;group_id&gt; </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="show-reported-events"><a class="header" href="#show-reported-events">Show reported events</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="show-reported-events"><a class="header" href="#show-reported-events">Show reported events</a></h1> <p>This API returns information about reported events.</p> <p>To use it, you will need to authenticate by providing an <code>access_token</code> for a server admin: see <a href="admin_api/../usage/administration/admin_api">Admin API</a>.</p> @@ -9780,7 +9828,7 @@ was reported.</li> have a canonical alias set.</li> <li><code>event_json</code>: object - Details of the original event that was reported.</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="querying-media"><a class="header" href="#querying-media">Querying media</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="querying-media"><a class="header" href="#querying-media">Querying media</a></h1> <p>These APIs allow extracting media information from the homeserver.</p> <p>Details about the format of the <code>media_id</code> and storage of the media in the file system are documented under <a href="admin_api/../media_repository.html">media repository</a>.</p> @@ -9993,7 +10041,7 @@ All cached media that was last accessed before this timestamp will be removed.</ </ul> <p>If the user re-requests purged remote media, synapse will re-request the media from the originating server.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="purge-history-api"><a class="header" href="#purge-history-api">Purge History API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="purge-history-api"><a class="header" href="#purge-history-api">Purge History API</a></h1> <p>The purge history API allows server admins to purge historic events from their database, reclaiming disk space.</p> <p>Depending on the amount of history being purged a call to the API may take @@ -10042,7 +10090,7 @@ a purge id:</p> <p>To reclaim the disk space and return it to the operating system, you need to run <code>VACUUM FULL;</code> on the database.</p> <p><a href="https://www.postgresql.org/docs/current/sql-vacuum.html">https://www.postgresql.org/docs/current/sql-vacuum.html</a></p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="shared-secret-registration"><a class="header" href="#shared-secret-registration">Shared-Secret Registration</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="shared-secret-registration"><a class="header" href="#shared-secret-registration">Shared-Secret Registration</a></h1> <p>This API allows for the creation of users in an administrative and non-interactive way. This is generally used for bootstrapping a Synapse instance with administrator accounts.</p> @@ -10103,7 +10151,7 @@ def generate_mac(nonce, user, password, admin=False, user_type=None): return mac.hexdigest() </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="registration-tokens"><a class="header" href="#registration-tokens">Registration Tokens</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="registration-tokens"><a class="header" href="#registration-tokens">Registration Tokens</a></h1> <p>This API allows you to manage tokens which can be used to authenticate registration requests, as proposed in <a href="https://github.com/matrix-org/matrix-doc/blob/main/proposals/3231-token-authenticated-registration.md">MSC3231</a>. @@ -10341,7 +10389,7 @@ the <a href="https://matrix.org/docs/spec/client_server/r0.6.1#api-standards">Ma &quot;error&quot;: &quot;No such registration token: 1234&quot; } </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="edit-room-membership-api"><a class="header" href="#edit-room-membership-api">Edit Room Membership API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="edit-room-membership-api"><a class="header" href="#edit-room-membership-api">Edit Room Membership API</a></h1> <p>This API allows an administrator to join an user account with a given <code>user_id</code> to a room with a given <code>room_id_or_alias</code>. You can only modify the membership of local users. The server administrator must be in the room and have permission to @@ -10367,7 +10415,7 @@ for a server admin: see <a href="admin_api/../usage/administration/admin_api">Ad &quot;room_id&quot;: &quot;!636q39766251:server.com&quot; } </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="list-room-api"><a class="header" href="#list-room-api">List Room API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="list-room-api"><a class="header" href="#list-room-api">List Room API</a></h1> <p>The List Room admin API allows server admins to get a list of rooms on their server. There are various parameters available that allow for filtering and sorting the returned list. This API supports pagination.</p> @@ -11176,7 +11224,7 @@ that were deleted.</p> ] } </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="server-notices-1"><a class="header" href="#server-notices-1">Server Notices</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="server-notices-1"><a class="header" href="#server-notices-1">Server Notices</a></h1> <p>The API to send notices is as follows:</p> <pre><code>POST /_synapse/admin/v1/send_server_notice </code></pre> @@ -11207,7 +11255,7 @@ ignored in the same way as with <code>PUT /_matrix/client/r0/rooms/{roomId}/send </code></pre> <p>Note that server notices must be enabled in <code>homeserver.yaml</code> before this API can be used. See <a href="admin_api/../server_notices.html">the server notices documentation</a> for more information.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="users-media-usage-statistics"><a class="header" href="#users-media-usage-statistics">Users' media usage statistics</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="users-media-usage-statistics"><a class="header" href="#users-media-usage-statistics">Users' media usage statistics</a></h1> <p>Returns information about all local media usage of users. Gives the possibility to filter them by time and user.</p> <p>To use it, you will need to authenticate by providing an <code>access_token</code> @@ -11283,7 +11331,7 @@ about the user and their local media. Objects contain the following fields: <li><code>next_token</code> - integer - Opaque value used for pagination. See above.</li> <li><code>total</code> - integer - Total number of users after filtering.</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="user-admin-api"><a class="header" href="#user-admin-api">User Admin API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="user-admin-api"><a class="header" href="#user-admin-api">User Admin API</a></h1> <p>To use it, you will need to authenticate by providing an <code>access_token</code> for a server admin: see <a href="admin_api/../usage/administration/admin_api">Admin API</a>.</p> <h2 id="query-user-account"><a class="header" href="#query-user-account">Query User Account</a></h2> @@ -12262,7 +12310,7 @@ for more information.</p> </code></pre> <p>The request and response format is the same as the <a href="https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available">/_matrix/client/r0/register/available</a> API.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="version-api"><a class="header" href="#version-api">Version API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="version-api"><a class="header" href="#version-api">Version API</a></h1> <p>This API returns the running Synapse version and the Python version on which Synapse is being run. This is useful when a Synapse instance is behind a proxy that does not forward the 'Server' header (which also @@ -12276,7 +12324,7 @@ contains Synapse version information).</p> &quot;python_version&quot;: &quot;3.7.8&quot; } </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="federation-api"><a class="header" href="#federation-api">Federation API</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="federation-api"><a class="header" href="#federation-api">Federation API</a></h1> <p>This API allows a server administrator to manage Synapse's federation with other homeservers.</p> <p>Note: This API is new, experimental and &quot;subject to change&quot;.</p> <h2 id="list-of-destinations"><a class="header" href="#list-of-destinations">List of destinations</a></h2> @@ -12448,7 +12496,7 @@ even if the API already returns the http status 200.</p> <ul> <li><code>destination</code> - Name of the remote server.</li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="using-the-synapse-manhole"><a class="header" href="#using-the-synapse-manhole">Using the synapse manhole</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="using-the-synapse-manhole"><a class="header" href="#using-the-synapse-manhole">Using the synapse manhole</a></h1> <p>The &quot;manhole&quot; allows server administrators to access a Python shell on a running Synapse installation. This is a very powerful mechanism for administration and debugging.</p> @@ -12515,7 +12563,7 @@ parts of the process.</p> &gt;&gt;&gt; defer.ensureDeferred(hs.get_datastores().main.get_event('$1416420717069yeQaw:matrix.org')) &lt;Deferred at 0x7ff253fc6998 current result: &lt;FrozenEvent event_id='$1416420717069yeQaw:matrix.org', type='m.room.create', state_key=''&gt;&gt; </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="how-to-monitor-synapse-metrics-using-prometheus"><a class="header" href="#how-to-monitor-synapse-metrics-using-prometheus">How to monitor Synapse metrics using Prometheus</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="how-to-monitor-synapse-metrics-using-prometheus"><a class="header" href="#how-to-monitor-synapse-metrics-using-prometheus">How to monitor Synapse metrics using Prometheus</a></h1> <ol> <li> <p>Install Prometheus:</p> @@ -12755,7 +12803,7 @@ renamed.</p> <tr><td>python_twisted_reactor_pending_calls</td><td>reactor_pending_calls</td></tr> <tr><td>python_twisted_reactor_tick_time</td><td>reactor_tick_time</td></tr> </tbody></table> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 id="understanding-synapse-through-grafana-graphs"><a class="header" href="#understanding-synapse-through-grafana-graphs">Understanding Synapse through Grafana graphs</a></h2> +<div style="break-before: page; page-break-before: always;"></div><h2 id="understanding-synapse-through-grafana-graphs"><a class="header" href="#understanding-synapse-through-grafana-graphs">Understanding Synapse through Grafana graphs</a></h2> <p>It is possible to monitor much of the internal state of Synapse using <a href="https://prometheus.io">Prometheus</a> metrics and <a href="https://grafana.com/">Grafana</a>. A guide for configuring Synapse to provide metrics is available <a href="usage/administration/../../metrics-howto.html">here</a> @@ -12805,7 +12853,7 @@ indicator of problems, and a symptom of other problems though, so check other gr tried everything you can, it may just be a lack of system resources. Consider adding more CPU and RAM, and make use of <a href="usage/administration/../../workers.html">worker mode</a> to make use of multiple CPU cores / multiple machines for your homeserver.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 id="some-useful-sql-queries-for-synapse-admins"><a class="header" href="#some-useful-sql-queries-for-synapse-admins">Some useful SQL queries for Synapse Admins</a></h2> +<div style="break-before: page; page-break-before: always;"></div><h2 id="some-useful-sql-queries-for-synapse-admins"><a class="header" href="#some-useful-sql-queries-for-synapse-admins">Some useful SQL queries for Synapse Admins</a></h2> <h2 id="size-of-full-matrix-db"><a class="header" href="#size-of-full-matrix-db">Size of full matrix db</a></h2> <p><code>SELECT pg_size_pretty( pg_database_size( 'matrix' ) );</code></p> <h3 id="result-example"><a class="header" href="#result-example">Result example:</a></h3> @@ -12939,19 +12987,19 @@ WHERE room_id IN (WHERE room_id IN ( '!YTvKGNlinIzlkMTVRl:matrix.org' ) </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><p>This blog post by Victor Berger explains how to use many of the tools listed on this page: https://levans.fr/shrink-synapse-database.html</p> +<div style="break-before: page; page-break-before: always;"></div><p>This blog post by Victor Berger explains how to use many of the tools listed on this page: https://levans.fr/shrink-synapse-database.html</p> <h1 id="list-of-useful-tools-and-scripts-for-maintenance-synapse-database"><a class="header" href="#list-of-useful-tools-and-scripts-for-maintenance-synapse-database">List of useful tools and scripts for maintenance Synapse database:</a></h1> -<h2 id="a-hrefusageadministrationadmin_apimedia_admin_apihtmlpurge-remote-media-apipurge-remote-media-apia"><a class="header" href="#a-hrefusageadministrationadmin_apimedia_admin_apihtmlpurge-remote-media-apipurge-remote-media-apia"><a href="usage/administration/../../admin_api/media_admin_api.html#purge-remote-media-api">Purge Remote Media API</a></a></h2> +<h2 id="purge-remote-media-api-1"><a class="header" href="#purge-remote-media-api-1"><a href="usage/administration/../../admin_api/media_admin_api.html#purge-remote-media-api">Purge Remote Media API</a></a></h2> <p>The purge remote media API allows server admins to purge old cached remote media.</p> -<h2 id="a-hrefusageadministrationadmin_apimedia_admin_apihtmldelete-local-mediapurge-local-media-apia"><a class="header" href="#a-hrefusageadministrationadmin_apimedia_admin_apihtmldelete-local-mediapurge-local-media-apia"><a href="usage/administration/../../admin_api/media_admin_api.html#delete-local-media">Purge Local Media API</a></a></h2> +<h2 id="purge-local-media-api"><a class="header" href="#purge-local-media-api"><a href="usage/administration/../../admin_api/media_admin_api.html#delete-local-media">Purge Local Media API</a></a></h2> <p>This API deletes the <em>local</em> media from the disk of your own server.</p> -<h2 id="a-hrefusageadministrationadmin_apipurge_history_apihtmlpurge-history-apia"><a class="header" href="#a-hrefusageadministrationadmin_apipurge_history_apihtmlpurge-history-apia"><a href="usage/administration/../../admin_api/purge_history_api.html">Purge History API</a></a></h2> +<h2 id="purge-history-api-1"><a class="header" href="#purge-history-api-1"><a href="usage/administration/../../admin_api/purge_history_api.html">Purge History API</a></a></h2> <p>The purge history API allows server admins to purge historic events from their database, reclaiming disk space.</p> -<h2 id="a-hrefhttpsgithubcommatrix-orgrust-synapse-compress-statesynapse-compress-statea"><a class="header" href="#a-hrefhttpsgithubcommatrix-orgrust-synapse-compress-statesynapse-compress-statea"><a href="https://github.com/matrix-org/rust-synapse-compress-state">synapse-compress-state</a></a></h2> +<h2 id="synapse-compress-state"><a class="header" href="#synapse-compress-state"><a href="https://github.com/matrix-org/rust-synapse-compress-state">synapse-compress-state</a></a></h2> <p>Tool for compressing (deduplicating) <code>state_groups_state</code> table.</p> -<h2 id="a-hrefusageadministrationuseful_sql_for_adminshtmlsql-for-analyzing-synapse-postgresql-database-statsa"><a class="header" href="#a-hrefusageadministrationuseful_sql_for_adminshtmlsql-for-analyzing-synapse-postgresql-database-statsa"><a href="usage/administration/useful_sql_for_admins.html">SQL for analyzing Synapse PostgreSQL database stats</a></a></h2> +<h2 id="sql-for-analyzing-synapse-postgresql-database-stats"><a class="header" href="#sql-for-analyzing-synapse-postgresql-database-stats"><a href="usage/administration/useful_sql_for_admins.html">SQL for analyzing Synapse PostgreSQL database stats</a></a></h2> <p>Some easy SQL that reports useful stats about your Synapse database.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="how-do-state-groups-work"><a class="header" href="#how-do-state-groups-work">How do State Groups work?</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="how-do-state-groups-work"><a class="header" href="#how-do-state-groups-work">How do State Groups work?</a></h1> <p>As a general rule, I encourage people who want to understand the deepest darkest secrets of the database schema to drop by #synapse-dev:matrix.org and ask questions.</p> <p>However, one question that comes up frequently is that of how &quot;state groups&quot; work, and why the <code>state_groups_state</code> table gets so big, so here's an attempt to answer that question.</p> <p>We need to be able to relatively quickly calculate the state of a room at any point in that room's history. In other words, we need to know the state of the room at each event in that room. This is done as follows:</p> @@ -12964,7 +13012,7 @@ WHERE room_id IN (WHERE room_id IN ( <p>There are various reasons that we can end up creating many more state groups than we need: see https://github.com/matrix-org/synapse/issues/3364 for more details.</p> <h2 id="compression-tool"><a class="header" href="#compression-tool">Compression tool</a></h2> <p>There is a tool at https://github.com/matrix-org/rust-synapse-compress-state which can compress the <code>state_groups_state</code> on a room by-room basis (essentially, it reduces the number of &quot;full&quot; state groups). This can result in dramatic reductions of the storage used.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="request-log-format"><a class="header" href="#request-log-format">Request log format</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="request-log-format"><a class="header" href="#request-log-format">Request log format</a></h1> <p>HTTP request logs are written by synapse (see <a href="usage/administration/../synapse/http/site.py"><code>site.py</code></a> for details).</p> <p>See the following for how to decode the dense data available from the default logging configuration.</p> <pre><code>2020-10-01 12:00:00,000 - synapse.access.http.8008 - 311 - INFO - PUT-1000- 192.168.0.1 - 8008 - {another-matrix-server.com} Processed request: 0.100sec/-0.000sec (0.000sec, 0.000sec) (0.001sec/0.090sec/3) 11B !200 &quot;PUT /_matrix/federation/v1/send/1600000000000 HTTP/1.1&quot; &quot;Synapse/1.20.1&quot; [0 dbevts] @@ -12999,7 +13047,7 @@ the same data, but only the first request will report time/transactions in <code>KKKK</code>/<code>LLLL</code>/<code>MMMM</code>/<code>NNNN</code>/<code>OOOO</code> - the others will be awaiting the first query to return a response and will simultaneously return with the first request, but with very small processing times.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 id="admin-faq"><a class="header" href="#admin-faq">Admin FAQ</a></h2> +<div style="break-before: page; page-break-before: always;"></div><h2 id="admin-faq"><a class="header" href="#admin-faq">Admin FAQ</a></h2> <h2 id="how-do-i-become-a-server-admin"><a class="header" href="#how-do-i-become-a-server-admin">How do I become a server admin?</a></h2> <p>If your server already has an admin account you should use the user admin API to promote other accounts to become admins. See <a href="usage/administration/../../admin_api/user_admin_api.html#Change-whether-a-user-is-a-server-administrator-or-not">User Admin API</a></p> <p>If you don't have any admin accounts yet you won't be able to use the admin API so you'll have to edit the database manually. Manually editing the database is generally not recommended so once you have an admin account, use the admin APIs to make further changes.</p> @@ -13061,7 +13109,7 @@ LIMIT 10; </code></pre> <p>You can also use the <a href="usage/administration/../../admin_api/rooms.html#list-room-api">List Room API</a> and <code>order_by</code> <code>state_events</code>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="contributing"><a class="header" href="#contributing">Contributing</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="contributing"><a class="header" href="#contributing">Contributing</a></h1> <p>This document aims to get you started with contributing to Synapse!</p> <h1 id="1-who-can-contribute-to-synapse"><a class="header" href="#1-who-can-contribute-to-synapse">1. Who can contribute to Synapse?</a></h1> <p>Everyone is welcome to contribute code to <a href="https://github.com/matrix-org">matrix.org @@ -13198,9 +13246,10 @@ useful to reproduce this locally.</p> following environment variables matching your configuration:</p> <ul> <li><code>SYNAPSE_POSTGRES</code> to anything nonempty</li> -<li><code>SYNAPSE_POSTGRES_HOST</code></li> -<li><code>SYNAPSE_POSTGRES_USER</code></li> -<li><code>SYNAPSE_POSTGRES_PASSWORD</code></li> +<li><code>SYNAPSE_POSTGRES_HOST</code> (optional if it's the default: UNIX socket)</li> +<li><code>SYNAPSE_POSTGRES_PORT</code> (optional if it's the default: 5432)</li> +<li><code>SYNAPSE_POSTGRES_USER</code> (optional if using a UNIX socket)</li> +<li><code>SYNAPSE_POSTGRES_PASSWORD</code> (optional if using a UNIX socket)</li> </ul> <p>For example:</p> <pre><code class="language-shell">export SYNAPSE_POSTGRES=1 @@ -13209,20 +13258,11 @@ export SYNAPSE_POSTGRES_USER=postgres export SYNAPSE_POSTGRES_PASSWORD=mydevenvpassword trial </code></pre> -<h4 id="prebuilt-container"><a class="header" href="#prebuilt-container">Prebuilt container</a></h4> -<p>Since configuring PostgreSQL can be fiddly, we can make use of a pre-made -Docker container to set up PostgreSQL and run our tests for us. To do so, run</p> -<pre><code class="language-shell">scripts-dev/test_postgresql.sh -</code></pre> -<p>Any extra arguments to the script will be passed to <code>tox</code> and then to <code>trial</code>, -so we can run a specific test in this container with e.g.</p> -<pre><code class="language-shell">scripts-dev/test_postgresql.sh tests.replication.test_sharded_event_persister.EventPersisterShardTestCase -</code></pre> -<p>The container creates a folder in your Synapse checkout called -<code>.tox-pg-container</code> and uses this as a tox environment. The output of any -<code>trial</code> runs goes into <code>_trial_temp</code> in your synapse source directory — the same -as running <code>trial</code> directly on your host machine.</p> -<h2 id="run-the-integration-tests-a-hrefhttpsgithubcommatrix-orgsytestsytesta"><a class="header" href="#run-the-integration-tests-a-hrefhttpsgithubcommatrix-orgsytestsytesta">Run the integration tests (<a href="https://github.com/matrix-org/sytest">Sytest</a>).</a></h2> +<p>You don't need to specify the host, user, port or password if your Postgres +server is set to authenticate you over the UNIX socket (i.e. if the <code>psql</code> command +works without further arguments).</p> +<p>Your Postgres account needs to be able to create databases.</p> +<h2 id="run-the-integration-tests-sytest"><a class="header" href="#run-the-integration-tests-sytest">Run the integration tests (<a href="https://github.com/matrix-org/sytest">Sytest</a>).</a></h2> <p>The integration tests are a more comprehensive suite of tests. They run a full version of Synapse, including your changes, to check if anything was broken. They are slower than the unit tests but will @@ -13231,8 +13271,14 @@ typically catch more errors.</p> configuration:</p> <pre><code class="language-sh">$ docker run --rm -it -v /path/where/you/have/cloned/the/repository\:/src:ro -v /path/to/where/you/want/logs\:/logs matrixdotorg/sytest-synapse:buster </code></pre> -<p>This configuration should generally cover your needs. For more details about other configurations, see <a href="https://github.com/matrix-org/sytest/blob/develop/docker/README.md">documentation in the SyTest repo</a>.</p> -<h2 id="run-the-integration-tests-a-hrefhttpsgithubcommatrix-orgcomplementcomplementa"><a class="header" href="#run-the-integration-tests-a-hrefhttpsgithubcommatrix-orgcomplementcomplementa">Run the integration tests (<a href="https://github.com/matrix-org/complement">Complement</a>).</a></h2> +<p>(Note that the paths must be full paths! You could also write <code>$(realpath relative/path)</code> if needed.)</p> +<p>This configuration should generally cover your needs.</p> +<ul> +<li>To run with Postgres, supply the <code>-e POSTGRES=1 -e MULTI_POSTGRES=1</code> environment flags.</li> +<li>To run with Synapse in worker mode, supply the <code>-e WORKERS=1 -e REDIS=1</code> environment flags (in addition to the Postgres flags).</li> +</ul> +<p>For more details about other configurations, see the <a href="https://github.com/matrix-org/sytest/blob/develop/docker/README.md">Docker-specific documentation in the SyTest repo</a>.</p> +<h2 id="run-the-integration-tests-complement"><a class="header" href="#run-the-integration-tests-complement">Run the integration tests (<a href="https://github.com/matrix-org/complement">Complement</a>).</a></h2> <p><a href="https://github.com/matrix-org/complement">Complement</a> is a suite of black box tests that can be run on any homeserver implementation. It can also be thought of as end-to-end (e2e) tests.</p> <p>It's often nice to develop on Synapse and write Complement tests at the same time. Here is how to run your local Synapse checkout against your local Complement checkout.</p> @@ -13435,7 +13481,7 @@ matrix together all the fragmented communication technologies out there we are reliant on contributions and collaboration from the community to do so. So please get involved - and we hope you have as much fun hacking on Matrix as we do!</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="code-style"><a class="header" href="#code-style">Code Style</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="code-style"><a class="header" href="#code-style">Code Style</a></h1> <h2 id="formatting-tools"><a class="header" href="#formatting-tools">Formatting tools</a></h2> <p>The Synapse codebase uses a number of code formatting tools in order to quickly and automatically check for formatting (and sometimes logical) @@ -13452,7 +13498,7 @@ as an opinionated code formatter, ensuring all comitted code is properly formatted.</p> <p>Have <code>black</code> auto-format your code (it shouldn't change any functionality) with:</p> -<pre><code class="language-sh">black . --exclude=&quot;\.tox|build|env&quot; +<pre><code class="language-sh">black . </code></pre> </li> <li> @@ -13460,7 +13506,7 @@ functionality) with:</p> <p><code>flake8</code> is a code checking tool. We require code to pass <code>flake8</code> before being merged into the codebase.</p> <p>Check all application and test code with:</p> -<pre><code class="language-sh">flake8 synapse tests +<pre><code class="language-sh">flake8 . </code></pre> </li> <li> @@ -13468,9 +13514,8 @@ before being merged into the codebase.</p> <p><code>isort</code> ensures imports are nicely formatted, and can suggest and auto-fix issues such as double-importing.</p> <p>Auto-fix imports with:</p> -<pre><code class="language-sh">isort -rc synapse tests +<pre><code class="language-sh">isort . </code></pre> -<p><code>-rc</code> means to recursively search the given directories.</p> </li> </ul> <p>It's worth noting that modern IDEs and text editors can run these tools @@ -13613,7 +13658,7 @@ frobber: and is maintained by a script, <code>scripts-dev/generate_sample_config.sh</code>. Making sure that the output from this script matches the desired format is left as an exercise for the reader!</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="synapse-release-cycle"><a class="header" href="#synapse-release-cycle">Synapse Release Cycle</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="synapse-release-cycle"><a class="header" href="#synapse-release-cycle">Synapse Release Cycle</a></h1> <p>Releases of Synapse follow a two week release cycle with new releases usually occurring on Tuesdays:</p> <ul> @@ -13643,7 +13688,7 @@ be held to be released together.</p> <p>In some cases, a pre-disclosure of a security release will be issued as a notice to Synapse operators that there is an upcoming security release. These can be found in the <a href="https://matrix.org/blog/category/security">security category of the Matrix blog</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="some-notes-on-how-we-use-git"><a class="header" href="#some-notes-on-how-we-use-git">Some notes on how we use git</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="some-notes-on-how-we-use-git"><a class="header" href="#some-notes-on-how-we-use-git">Some notes on how we use git</a></h1> <h2 id="on-keeping-the-commit-history-clean"><a class="header" href="#on-keeping-the-commit-history-clean">On keeping the commit history clean</a></h2> <p>In an ideal world, our git commit history would be a linear progression of commits each of which contains a single change building on what came @@ -13745,7 +13790,7 @@ that our active branches are ordered thus, from more-stable to less-stable:</p> <ul> <li><code>master</code> (tracks our last release).</li> <li><code>release-vX.Y</code> (the branch where we prepare the next release)<sup - id="a3"><a href="development/git.html#f3">3</a></sup>.</li> +id="a3"><a href="development/git.html#f3">3</a></sup>.</li> <li>PR branches which are targeting the release.</li> <li><code>develop</code> (our &quot;mainline&quot; branch containing our bleeding-edge).</li> <li>regular PR branches.</li> @@ -13764,7 +13809,7 @@ most intuitive name. <a href="development/git.html#a1">^</a></p> <p><b id="f3">[3]</b>: Very, very occasionally (I think this has happened once in the history of Synapse), we've had two releases in flight at once. Obviously, <code>release-v1.2</code> is more-stable than <code>release-v1.3</code>. <a href="development/git.html#a3">^</a></p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="synapse-demo-setup"><a class="header" href="#synapse-demo-setup">Synapse demo setup</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="synapse-demo-setup"><a class="header" href="#synapse-demo-setup">Synapse demo setup</a></h1> <p><strong>DO NOT USE THESE DEMO SERVERS IN PRODUCTION</strong></p> <p>Requires you to have a <a href="https://matrix-org.github.io/synapse/develop/development/contributing_guide.html#4-install-the-dependencies">Synapse development environment setup</a>.</p> <p>The demo setup allows running three federation Synapse servers, with server @@ -13798,7 +13843,7 @@ it won't work if your HS already has a room with that name.</p> <p>To start a completely new set of servers, run:</p> <pre><code class="language-sh">./demo/stop.sh; ./demo/clean.sh &amp;&amp; ./demo/start.sh </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="opentracing"><a class="header" href="#opentracing">OpenTracing</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="opentracing"><a class="header" href="#opentracing">OpenTracing</a></h1> <h2 id="background"><a class="header" href="#background">Background</a></h2> <p>OpenTracing is a semi-standard being adopted by a number of distributed tracing platforms. It is a common api for facilitating vendor-agnostic @@ -13871,7 +13916,7 @@ logged to OpenTracing's logs.</li> <h2 id="configuring-jaeger"><a class="header" href="#configuring-jaeger">Configuring Jaeger</a></h2> <p>Sampling strategies can be set as in this document: <a href="https://www.jaegertracing.io/docs/latest/sampling/">https://www.jaegertracing.io/docs/latest/sampling/</a>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="synapse-database-schema-files"><a class="header" href="#synapse-database-schema-files">Synapse database schema files</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="synapse-database-schema-files"><a class="header" href="#synapse-database-schema-files">Synapse database schema files</a></h1> <p>Synapse's database schema is stored in the <code>synapse.storage.schema</code> module.</p> <h2 id="logical-databases"><a class="header" href="#logical-databases">Logical databases</a></h2> <p>Synapse supports splitting its datastore across multiple physical databases (which can @@ -14033,7 +14078,7 @@ default value is the <strong>string</strong> <code>&quot;FALSE&quot;</code> - wh in Python, evaluates to <code>True</code>.</p> </li> </ul> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="implementing-experimental-features-in-synapse"><a class="header" href="#implementing-experimental-features-in-synapse">Implementing experimental features in Synapse</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="implementing-experimental-features-in-synapse"><a class="header" href="#implementing-experimental-features-in-synapse">Implementing experimental features in Synapse</a></h1> <p>It can be desirable to implement &quot;experimental&quot; features which are disabled by default and must be explicitly enabled via the Synapse configuration. This is applicable for features which:</p> @@ -14066,7 +14111,7 @@ but one should be used if unsure.</p> <p>New experimental configuration flags should be added under the <code>experimental</code> configuration key (see the <code>synapse.config.experimental</code> file) and either explain (briefly) what is being enabled, or include the MSC number.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="log-contexts"><a class="header" href="#log-contexts">Log Contexts</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="log-contexts"><a class="header" href="#log-contexts">Log Contexts</a></h1> <p>To help track the processing of individual requests, synapse uses a '<code>log context</code>' to track which request it is handling at any given moment. This is done via a thread-local variable; a <code>logging.Filter</code> is @@ -14362,7 +14407,7 @@ chain are dropped. Dropping the the reference to an awaitable you're supposed to be awaiting is bad practice, so this doesn't actually happen too much. Unfortunately, when it does happen, it will lead to leaked logcontexts which are incredibly hard to track down.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="replication-architecture"><a class="header" href="#replication-architecture">Replication Architecture</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="replication-architecture"><a class="header" href="#replication-architecture">Replication Architecture</a></h1> <h2 id="motivation"><a class="header" href="#motivation">Motivation</a></h2> <p>We'd like to be able to split some of the work that synapse does into multiple python processes. In theory multiple synapse processes could @@ -14390,7 +14435,7 @@ minimal.</p> <p>There are read-only version of the synapse storage layer in <code>synapse/replication/slave/storage</code> that use the response of the replication API to invalidate their caches.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="tcp-replication"><a class="header" href="#tcp-replication">TCP Replication</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="tcp-replication"><a class="header" href="#tcp-replication">TCP Replication</a></h1> <h2 id="motivation-1"><a class="header" href="#motivation-1">Motivation</a></h2> <p>Previously the workers used an HTTP long poll mechanism to get updates from the master, which had the problem of causing a lot of duplicate @@ -14493,7 +14538,7 @@ connection errors.</p> received for each stream so that on reconneciton it can start streaming from the correct place. Note: not all RDATA have valid tokens due to batching. See <code>RdataCommand</code> for more details.</p> -<h3 id="example-4"><a class="header" href="#example-4">Example</a></h3> +<h3 id="example-5"><a class="header" href="#example-5">Example</a></h3> <p>An example iteraction is shown below. Each line is prefixed with '&gt;' or '&lt;' to indicate which side is sending, these are <em>not</em> included on the wire:</p> @@ -14585,7 +14630,7 @@ workers understand to mean to expand to invalidate the correct caches.</p> <li><code>cs_cache_fake</code> ─ invalidates caches that depend on the current state</li> </ol> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="internal-documentation"><a class="header" href="#internal-documentation">Internal Documentation</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="internal-documentation"><a class="header" href="#internal-documentation">Internal Documentation</a></h1> <p>This section covers implementation documentation for various parts of Synapse.</p> <p>If a developer is planning to make a change to a feature of Synapse, it can be useful for general documentation of how that feature is implemented to be available. This saves the @@ -14594,7 +14639,7 @@ code.</p> <p>Documentation that would be more useful for the perspective of a system administrator, rather than a developer who's intending to change to code, should instead be placed under the Usage section of the documentation.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="how-to-test-saml-as-a-developer-without-a-server"><a class="header" href="#how-to-test-saml-as-a-developer-without-a-server">How to test SAML as a developer without a server</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="how-to-test-saml-as-a-developer-without-a-server"><a class="header" href="#how-to-test-saml-as-a-developer-without-a-server">How to test SAML as a developer without a server</a></h1> <p>https://fujifish.github.io/samling/samling.html (https://github.com/fujifish/samling) is a great resource for being able to tinker with the SAML options within Synapse without needing to deploy and configure a complicated software stack.</p> <p>To make Synapse (and therefore Element) use it:</p> @@ -14632,7 +14677,7 @@ The response must also be signed.</li> <p>If you try and repeat this process, you may be automatically logged in using the information you gave previously. To fix this, open your developer console (<code>F12</code> or <code>Ctrl+Shift+I</code>) while on the samling page and clear the site data. In Chrome, this will be a button on the Application tab.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="how-to-test-cas-as-a-developer-without-a-server"><a class="header" href="#how-to-test-cas-as-a-developer-without-a-server">How to test CAS as a developer without a server</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="how-to-test-cas-as-a-developer-without-a-server"><a class="header" href="#how-to-test-cas-as-a-developer-without-a-server">How to test CAS as a developer without a server</a></h1> <p>The <a href="https://github.com/jbittel/django-mama-cas">django-mama-cas</a> project is an easy to run CAS implementation built on top of Django.</p> <h2 id="prerequisites"><a class="header" href="#prerequisites">Prerequisites</a></h2> @@ -14694,7 +14739,7 @@ and that the CAS server is on port 8000, both on localhost.</p> <li>http://localhost:8000/admin/</li> <li>Click &quot;logout&quot; in the top right.</li> </ol> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="room-dag-concepts"><a class="header" href="#room-dag-concepts">Room DAG concepts</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="room-dag-concepts"><a class="header" href="#room-dag-concepts">Room DAG concepts</a></h1> <h2 id="edges"><a class="header" href="#edges">Edges</a></h2> <p>The word &quot;edge&quot; comes from graph theory lingo. An edge is just a connection between two events. In Synapse, we connect events by specifying their @@ -14724,7 +14769,8 @@ yet correlated to the DAG.</p> <p>Outliers typically arise when we fetch the auth chain or state for a given event. When that happens, we just grab the events in the state/auth chain, without calculating the state at those events, or backfilling their -<code>prev_events</code>.</p> +<code>prev_events</code>. Since we don't have the state at any events fetched in that +way, we mark them as outliers.</p> <p>So, typically, we won't have the <code>prev_events</code> of an <code>outlier</code> in the database, (though it's entirely possible that we <em>might</em> have them for some other reason). Other things that make outliers different from regular events:</p> @@ -14782,7 +14828,7 @@ mappings of <code>event_id -&gt; state_group</code> and <code>state_group -&gt; <h3 id="stage-group-edges"><a class="header" href="#stage-group-edges">Stage group edges</a></h3> <p>TODO: <code>state_group_edges</code> is a further optimization... notes from @Azrenbeth, https://pastebin.com/seUGVGeT</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="auth-chain-difference-algorithm"><a class="header" href="#auth-chain-difference-algorithm">Auth Chain Difference Algorithm</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="auth-chain-difference-algorithm"><a class="header" href="#auth-chain-difference-algorithm">Auth Chain Difference Algorithm</a></h1> <p>The auth chain difference algorithm is used by V2 state resolution, where a naive implementation can be a significant source of CPU and DB usage.</p> <h3 id="definitions"><a class="header" href="#definitions">Definitions</a></h3> @@ -14825,7 +14871,7 @@ graph), and one where we remove redundant links (the transitive reduction of the links graph) e.g. if we have chains <code>C3 -&gt; C2 -&gt; C1</code> then the link <code>C3 -&gt; C1</code> would not be stored. Synapse uses the former implementations so that it doesn't need to recurse to test reachability between chains.</p> -<h3 id="example-5"><a class="header" href="#example-5">Example</a></h3> +<h3 id="example-6"><a class="header" href="#example-6">Example</a></h3> <p>An example auth graph would look like the following, where chains have been formed based on type/state_key and are denoted by colour and are labelled with <code>(chain ID, sequence number)</code>. Links are denoted by the arrows (links in grey @@ -14873,7 +14919,7 @@ level).</li> </ol> <p>So the final result is: Bob's second join <code>(2,2)</code>, the second power level <code>(3,2)</code> and both of Alice's joins <code>(4,2)</code> &amp; <code>(4,3)</code>.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="media-repository"><a class="header" href="#media-repository">Media Repository</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="media-repository"><a class="header" href="#media-repository">Media Repository</a></h1> <p><em>Synapse implementation-specific details for the media repository</em></p> <p>The media repository</p> <ul> @@ -14936,7 +14982,7 @@ date and <code>aaaaaaaaaaaaaaaa</code> is a random sequence of 16 case-sensitive <pre><code>url_cache/yyyy-mm-dd/aaaaaaaaaaaaaaaa url_cache_thumbnails/yyyy-mm-dd/aaaaaaaaaaaaaaaa/128-96-image-jpeg-scale </code></pre> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="room-and-user-statistics"><a class="header" href="#room-and-user-statistics">Room and User Statistics</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="room-and-user-statistics"><a class="header" href="#room-and-user-statistics">Room and User Statistics</a></h1> <p>Synapse maintains room and user statistics in various tables. These can be used for administrative purposes but are also used when generating the public room directory.</p> @@ -14951,7 +14997,7 @@ table. Each subject can have only one.</li> <h3 id="overview-4"><a class="header" href="#overview-4">Overview</a></h3> <p>Stats correspond to the present values. Current rows contain the most up-to-date statistics for a room. Each subject can only have one entry.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="deprecation-policy-for-platform-dependencies"><a class="header" href="#deprecation-policy-for-platform-dependencies">Deprecation Policy for Platform Dependencies</a></h1> +<div style="break-before: page; page-break-before: always;"></div><h1 id="deprecation-policy-for-platform-dependencies"><a class="header" href="#deprecation-policy-for-platform-dependencies">Deprecation Policy for Platform Dependencies</a></h1> <p>Synapse has a number of platform dependencies, including Python and PostgreSQL. This document outlines the policy towards which versions we support, and when we drop support for versions in the future.</p> @@ -14972,7 +15018,7 @@ long process.</p> <p>By following the upstream support life cycles Synapse can ensure that its dependencies continue to get security patches, while not requiring system admins to constantly update their platform dependencies to the latest versions.</p> -<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 id="summary-of-performance-impact-of-running-on-resource-constrained-devices-such-as-sbcs"><a class="header" href="#summary-of-performance-impact-of-running-on-resource-constrained-devices-such-as-sbcs">Summary of performance impact of running on resource constrained devices such as SBCs</a></h2> +<div style="break-before: page; page-break-before: always;"></div><h2 id="summary-of-performance-impact-of-running-on-resource-constrained-devices-such-as-sbcs"><a class="header" href="#summary-of-performance-impact-of-running-on-resource-constrained-devices-such-as-sbcs">Summary of performance impact of running on resource constrained devices such as SBCs</a></h2> <p>I've been running my homeserver on a cubietruck at home now for some time and am often replying to statements like &quot;you need loads of ram to join large rooms&quot; with &quot;it works fine for me&quot;. I thought it might be useful to curate a summary of the issues you're likely to run into to help as a scaling-down guide, maybe highlight these for development work or end up as documentation. It seems that once you get up to about 4x1.5GHz arm64 4GiB these issues are no longer a problem.</p> <ul> <li><strong>Platform</strong>: 2x1GHz armhf 2GiB ram <a href="https://wiki.debian.org/CheapServerBoxHardware">Single-board computers</a>, SSD, postgres.</li> @@ -15033,61 +15079,32 @@ admin@homeserver:~$ sudo --user postgres psql matrix-synapse --command 'select c <nav class="nav-wrapper" aria-label="Page navigation"> <!-- Mobile navigation buttons --> - - - - <div style="clear: both"></div> </nav> </div> </div> <nav class="nav-wide-wrapper" aria-label="Page navigation"> - - - </nav> </div> - - - - - - - <script type="text/javascript"> window.playground_copyable = true; </script> - - - - - <script src="elasticlunr.min.js" type="text/javascript" charset="utf-8"></script> <script src="mark.min.js" type="text/javascript" charset="utf-8"></script> <script src="searcher.js" type="text/javascript" charset="utf-8"></script> - - <script src="clipboard.min.js" type="text/javascript" charset="utf-8"></script> <script src="highlight.js" type="text/javascript" charset="utf-8"></script> <script src="book.js" type="text/javascript" charset="utf-8"></script> <!-- Custom JS scripts --> - <script type="text/javascript" src="docs/website_files/table-of-contents.js"></script> - - - - <script type="text/javascript"> window.addEventListener('load', function() { window.setTimeout(window.print, 100); }); </script> - - - </body> </html> \ No newline at end of file