diff --git a/latest/print.html b/latest/print.html
index 85964f0f41..dc2b03ddb4 100644
--- a/latest/print.html
+++ b/latest/print.html
@@ -101,7 +101,7 @@
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
- <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded "><a href="MSC1711_certificates_FAQ.html">Upgrading from pre-Synapse 1.0</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single-Sign On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><div>SAML</div></li><li class="chapter-item expanded "><div>CAS</div></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="url_previews.html">URL Previews</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Third Party Rules</div></li><li class="chapter-item expanded "><a href="spam_checker.html">Spam Checker</a></li><li class="chapter-item expanded "><a href="presence_router_module.html">Presence Router</a></li><li class="chapter-item expanded "><div>Media Storage Providers</div></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="admin_api/delete_group.html">Delete Group</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li></ol>
+ <ol class="chapter"><li class="chapter-item expanded affix "><li class="part-title">Introduction</li><li class="chapter-item expanded "><a href="welcome_and_overview.html">Welcome and Overview</a></li><li class="chapter-item expanded affix "><li class="part-title">Setup</li><li class="chapter-item expanded "><a href="setup/installation.html">Installation</a></li><li class="chapter-item expanded "><a href="postgres.html">Using Postgres</a></li><li class="chapter-item expanded "><a href="reverse_proxy.html">Configuring a Reverse Proxy</a></li><li class="chapter-item expanded "><a href="setup/forward_proxy.html">Configuring a Forward/Outbound Proxy</a></li><li class="chapter-item expanded "><a href="turn-howto.html">Configuring a Turn Server</a></li><li class="chapter-item expanded "><a href="delegate.html">Delegation</a></li><li class="chapter-item expanded affix "><li class="part-title">Upgrading</li><li class="chapter-item expanded "><a href="upgrade.html">Upgrading between Synapse Versions</a></li><li class="chapter-item expanded "><a href="MSC1711_certificates_FAQ.html">Upgrading from pre-Synapse 1.0</a></li><li class="chapter-item expanded affix "><li class="part-title">Usage</li><li class="chapter-item expanded "><a href="federate.html">Federation</a></li><li class="chapter-item expanded "><a href="usage/configuration/index.html">Configuration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/configuration/homeserver_sample_config.html">Homeserver Sample Config File</a></li><li class="chapter-item expanded "><a href="usage/configuration/logging_sample_config.html">Logging Sample Config File</a></li><li class="chapter-item expanded "><a href="structured_logging.html">Structured Logging</a></li><li class="chapter-item expanded "><a href="templates.html">Templates</a></li><li class="chapter-item expanded "><a href="usage/configuration/user_authentication/index.html">User Authentication</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single-Sign On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="openid.html">OpenID Connect</a></li><li class="chapter-item expanded "><div>SAML</div></li><li class="chapter-item expanded "><div>CAS</div></li><li class="chapter-item expanded "><a href="sso_mapping_providers.html">SSO Mapping Providers</a></li></ol></li><li class="chapter-item expanded "><a href="password_auth_providers.html">Password Auth Providers</a></li><li class="chapter-item expanded "><a href="jwt.html">JSON Web Tokens</a></li></ol></li><li class="chapter-item expanded "><a href="CAPTCHA_SETUP.html">Registration Captcha</a></li><li class="chapter-item expanded "><a href="application_services.html">Application Services</a></li><li class="chapter-item expanded "><a href="server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="consent_tracking.html">Consent Tracking</a></li><li class="chapter-item expanded "><a href="development/url_previews.html">URL Previews</a></li><li class="chapter-item expanded "><a href="user_directory.html">User Directory</a></li><li class="chapter-item expanded "><a href="message_retention_policies.html">Message Retention Policies</a></li><li class="chapter-item expanded "><a href="modules/index.html">Pluggable Modules</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/writing_a_module.html">Writing a module</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="modules/spam_checker_callbacks.html">Spam checker callbacks</a></li><li class="chapter-item expanded "><a href="modules/third_party_rules_callbacks.html">Third-party rules callbacks</a></li><li class="chapter-item expanded "><a href="modules/presence_router_callbacks.html">Presence router callbacks</a></li><li class="chapter-item expanded "><a href="modules/account_validity_callbacks.html">Account validity callbacks</a></li><li class="chapter-item expanded "><a href="modules/porting_legacy_module.html">Porting a legacy module to the new interface</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="workers.html">Workers</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="synctl_workers.html">Using synctl with Workers</a></li><li class="chapter-item expanded "><a href="systemd-with-workers/index.html">Systemd</a></li></ol></li></ol></li><li class="chapter-item expanded "><a href="usage/administration/index.html">Administration</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="usage/administration/admin_api/index.html">Admin API</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="admin_api/account_validity.html">Account Validity</a></li><li class="chapter-item expanded "><a href="admin_api/delete_group.html">Delete Group</a></li><li class="chapter-item expanded "><a href="admin_api/event_reports.html">Event Reports</a></li><li class="chapter-item expanded "><a href="admin_api/media_admin_api.html">Media</a></li><li class="chapter-item expanded "><a href="admin_api/purge_history_api.html">Purge History</a></li><li class="chapter-item expanded "><a href="admin_api/register_api.html">Register Users</a></li><li class="chapter-item expanded "><a href="usage/administration/admin_api/registration_tokens.html">Registration Tokens</a></li><li class="chapter-item expanded "><a href="admin_api/room_membership.html">Manipulate Room Membership</a></li><li class="chapter-item expanded "><a href="admin_api/rooms.html">Rooms</a></li><li class="chapter-item expanded "><a href="admin_api/server_notices.html">Server Notices</a></li><li class="chapter-item expanded "><a href="admin_api/statistics.html">Statistics</a></li><li class="chapter-item expanded "><a href="admin_api/user_admin_api.html">Users</a></li><li class="chapter-item expanded "><a href="admin_api/version_api.html">Server Version</a></li></ol></li><li class="chapter-item expanded "><a href="manhole.html">Manhole</a></li><li class="chapter-item expanded "><a href="metrics-howto.html">Monitoring</a></li><li class="chapter-item expanded "><a href="usage/administration/request_log.html">Request log format</a></li><li class="chapter-item expanded "><div>Scripts</div></li></ol></li><li class="chapter-item expanded "><li class="part-title">Development</li><li class="chapter-item expanded "><a href="development/contributing_guide.html">Contributing Guide</a></li><li class="chapter-item expanded "><a href="code_style.html">Code Style</a></li><li class="chapter-item expanded "><a href="development/git.html">Git Usage</a></li><li class="chapter-item expanded "><div>Testing</div></li><li class="chapter-item expanded "><a href="opentracing.html">OpenTracing</a></li><li class="chapter-item expanded "><a href="development/database_schema.html">Database Schemas</a></li><li class="chapter-item expanded "><div>Synapse Architecture</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="log_contexts.html">Log Contexts</a></li><li class="chapter-item expanded "><a href="replication.html">Replication</a></li><li class="chapter-item expanded "><a href="tcp_replication.html">TCP Replication</a></li></ol></li><li class="chapter-item expanded "><a href="development/internal_documentation/index.html">Internal Documentation</a></li><li><ol class="section"><li class="chapter-item expanded "><div>Single Sign-On</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="development/saml.html">SAML</a></li><li class="chapter-item expanded "><a href="development/cas.html">CAS</a></li></ol></li><li class="chapter-item expanded "><a href="development/room-dag-concepts.html">Room DAG concepts</a></li><li class="chapter-item expanded "><div>State Resolution</div></li><li><ol class="section"><li class="chapter-item expanded "><a href="auth_chain_difference_algorithm.html">The Auth Chain Difference Algorithm</a></li></ol></li><li class="chapter-item expanded "><a href="media_repository.html">Media Repository</a></li><li class="chapter-item expanded "><a href="room_and_user_statistics.html">Room and User Statistics</a></li></ol></li><li class="chapter-item expanded "><div>Scripts</div></li><li class="chapter-item expanded affix "><li class="part-title">Other</li><li class="chapter-item expanded "><a href="deprecation_policy.html">Dependency Deprecation Policy</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
</nav>
@@ -1487,6 +1487,12 @@ dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
</code></pre>
</li>
</ul>
+<h1 id="upgrading-to-v1430"><a class="header" href="#upgrading-to-v1430">Upgrading to v1.43.0</a></h1>
+<h2 id="the-spaces-summary-apis-can-now-be-handled-by-workers"><a class="header" href="#the-spaces-summary-apis-can-now-be-handled-by-workers">The spaces summary APIs can now be handled by workers</a></h2>
+<p>The <a href="https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications">available worker applications documentation</a>
+has been updated to reflect that calls to the <code>/spaces</code>, <code>/hierarchy</code>, and
+<code>/summary</code> endpoints can now be routed to workers for both client API and
+federation requests.</p>
<h1 id="upgrading-to-v1420"><a class="header" href="#upgrading-to-v1420">Upgrading to v1.42.0</a></h1>
<h2 id="removal-of-old-room-admin-api"><a class="header" href="#removal-of-old-room-admin-api">Removal of old Room Admin API</a></h2>
<p>The following admin APIs were deprecated in <a href="https://github.com/matrix-org/synapse/blob/v1.25.0/CHANGES.md#removal-warning">Synapse 1.25</a>
@@ -3274,6 +3280,24 @@ listeners:
# bind_addresses: ['::1', '127.0.0.1']
# type: manhole
+# Connection settings for the manhole
+#
+manhole_settings:
+ # The username for the manhole. This defaults to 'matrix'.
+ #
+ #username: manhole
+
+ # The password for the manhole. This defaults to 'rabbithole'.
+ #
+ #password: mypassword
+
+ # The private and public SSH key pair used to encrypt the manhole traffic.
+ # If these are left unset, then hardcoded and non-secret keys are used,
+ # which could allow traffic to be intercepted if sent over a public network.
+ #
+ #ssh_priv_key_path: CONFDIR/id_rsa
+ #ssh_pub_key_path: CONFDIR/id_rsa.pub
+
# Forward extremities can build up in a room due to networking delays between
# homeservers. Once this happens in a large room, calculation of the state of
# that room can become quite expensive. To mitigate this, once the number of
@@ -4014,6 +4038,27 @@ url_preview_accept_language:
# - en
+# oEmbed allows for easier embedding content from a website. It can be
+# used for generating URLs previews of services which support it.
+#
+oembed:
+ # A default list of oEmbed providers is included with Synapse.
+ #
+ # Uncomment the following to disable using these default oEmbed URLs.
+ # Defaults to 'false'.
+ #
+ #disable_default_providers: true
+
+ # Additional files with oEmbed configuration (each should be in the
+ # form of providers.json).
+ #
+ # By default, this list is empty (so only the default providers.json
+ # is used).
+ #
+ #additional_providers:
+ # - oembed/my_providers.json
+
+
## Captcha ##
# See docs/CAPTCHA_SETUP.md for full details of configuring this.
@@ -4986,7 +5031,7 @@ password_config:
#
#require_lowercase: true
- # Whether a password must contain at least one lowercase letter.
+ # Whether a password must contain at least one uppercase letter.
# Defaults to 'false'.
#
#require_uppercase: true
@@ -7225,29 +7270,10 @@ consent uri for that user.</p>
URI that clients use to connect to the server. (It is used to construct
<code>consent_uri</code> in the error.)</p>
<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="url-previews-1"><a class="header" href="#url-previews-1">URL Previews</a></h1>
-<p>Design notes on a URL previewing service for Matrix:</p>
-<p>Options are:</p>
-<ol>
-<li>Have an AS which listens for URLs, downloads them, and inserts an event that describes their metadata.</li>
-</ol>
-<ul>
-<li>Pros:
-<ul>
-<li>Decouples the implementation entirely from Synapse.</li>
-<li>Uses existing Matrix events & content repo to store the metadata.</li>
-</ul>
-</li>
-<li>Cons:
-<ul>
-<li>Which AS should provide this service for a room, and why should you trust it?</li>
-<li>Doesn't work well with E2E; you'd have to cut the AS into every room</li>
-<li>the AS would end up subscribing to every room anyway.</li>
-</ul>
-</li>
-</ul>
-<ol start="2">
-<li>Have a generic preview API (nothing to do with Matrix) that provides a previewing service:</li>
-</ol>
+<p>The <code>GET /_matrix/media/r0/preview_url</code> endpoint provides a generic preview API
+for URLs which outputs <a href="https://ogp.me/">Open Graph</a> responses (with some Matrix
+specific additions).</p>
+<p>This does have trade-offs compared to other designs:</p>
<ul>
<li>Pros:
<ul>
@@ -7256,84 +7282,58 @@ URI that clients use to connect to the server. (It is used to construct
</li>
<li>Cons:
<ul>
-<li>If each HS provides one of these independently, all the HSes in a room may needlessly DoS the target URI</li>
-<li>We need somewhere to store the URL metadata rather than just using Matrix itself</li>
-<li>We can't piggyback on matrix to distribute the metadata between HSes.</li>
+<li>If each homeserver provides one of these independently, all the HSes in a
+room may needlessly DoS the target URI</li>
+<li>The URL metadata must be stored somewhere, rather than just using Matrix
+itself to store the media.</li>
+<li>Matrix cannot be used to distribute the metadata between homeservers.</li>
</ul>
</li>
</ul>
-<ol start="3">
-<li>Make the synapse of the sending user responsible for spidering the URL and inserting an event asynchronously which describes the metadata.</li>
+<p>When Synapse is asked to preview a URL it does the following:</p>
+<ol>
+<li>Checks against a URL blacklist (defined as <code>url_preview_url_blacklist</code> in the
+config).</li>
+<li>Checks the in-memory cache by URLs and returns the result if it exists. (This
+is also used to de-duplicate processing of multiple in-flight requests at once.)</li>
+<li>Kicks off a background process to generate a preview:
+<ol>
+<li>Checks the database cache by URL and timestamp and returns the result if it
+has not expired and was successful (a 2xx return code).</li>
+<li>Checks if the URL matches an oEmbed pattern. If it does, fetch the oEmbed
+response. If this is an image, replace the URL to fetch and continue. If
+if it is HTML content, use the HTML as the document and continue.</li>
+<li>If it doesn't match an oEmbed pattern, downloads the URL and stores it
+into a file via the media storage provider and saves the local media
+metadata.</li>
+<li>If the media is an image:
+<ol>
+<li>Generates thumbnails.</li>
+<li>Generates an Open Graph response based on image properties.</li>
</ol>
-<ul>
-<li>Pros:
-<ul>
-<li>Works transparently for all clients</li>
-<li>Piggy-backs nicely on using Matrix for distributing the metadata.</li>
-<li>No confusion as to which AS</li>
-</ul>
-</li>
-<li>Cons:
-<ul>
-<li>Doesn't work with E2E</li>
-<li>We might want to decouple the implementation of the spider from the HS, given spider behaviour can be quite complicated and evolve much more rapidly than the HS. It's more like a bot than a core part of the server.</li>
-</ul>
</li>
-</ul>
-<ol start="4">
-<li>Make the sending client use the preview API and insert the event itself when successful.</li>
+<li>If the media is HTML:
+<ol>
+<li>Decodes the HTML via the stored file.</li>
+<li>Generates an Open Graph response from the HTML.</li>
+<li>If an image exists in the Open Graph response:
+<ol>
+<li>Downloads the URL and stores it into a file via the media storage
+provider and saves the local media metadata.</li>
+<li>Generates thumbnails.</li>
+<li>Updates the Open Graph response based on image properties.</li>
</ol>
-<ul>
-<li>Pros:
-<ul>
-<li>Works well with E2E</li>
-<li>No custom server functionality</li>
-<li>Lets the client customise the preview that they send (like on FB)</li>
-</ul>
-</li>
-<li>Cons:
-<ul>
-<li>Entirely specific to the sending client, whereas it'd be nice if /any/ URL was correctly previewed if clients support it.</li>
-</ul>
</li>
-</ul>
-<ol start="5">
-<li>Have the option of specifying a shared (centralised) previewing service used by a room, to avoid all the different HSes in the room DoSing the target.</li>
</ol>
-<p>Best solution is probably a combination of both 2 and 4.</p>
-<ul>
-<li>Sending clients do their best to create and send a preview at the point of sending the message, perhaps delaying the message until the preview is computed? (This also lets the user validate the preview before sending)</li>
-<li>Receiving clients have the option of going and creating their own preview if one doesn't arrive soon enough (or if the original sender didn't create one)</li>
-</ul>
-<p>This is a bit magical though in that the preview could come from two entirely different sources - the sending HS or your local one. However, this can always be exposed to users: "Generate your own URL previews if none are available?"</p>
-<p>This is tantamount also to senders calculating their own thumbnails for sending in advance of the main content - we are trusting the sender not to lie about the content in the thumbnail. Whereas currently thumbnails are calculated by the receiving homeserver to avoid this attack.</p>
-<p>However, this kind of phishing attack does exist whether we let senders pick their thumbnails or not, in that a malicious sender can send normal text messages around the attachment claiming it to be legitimate. We could rely on (future) reputation/abuse management to punish users who phish (be it with bogus metadata or bogus descriptions). Bogus metadata is particularly bad though, especially if it's avoidable.</p>
-<p>As a first cut, let's do #2 and have the receiver hit the API to calculate its own previews (as it does currently for image thumbnails). We can then extend/optimise this to option 4 as a special extra if needed.</p>
-<h2 id="api"><a class="header" href="#api">API</a></h2>
-<pre><code>GET /_matrix/media/r0/preview_url?url=http://wherever.com
-200 OK
-{
- "og:type" : "article"
- "og:url" : "https://twitter.com/matrixdotorg/status/684074366691356672"
- "og:title" : "Matrix on Twitter"
- "og:image" : "https://pbs.twimg.com/profile_images/500400952029888512/yI0qtFi7_400x400.png"
- "og:description" : "“Synapse 0.12 is out! Lots of polishing, performance &amp;amp; bugfixes: /sync API, /r0 prefix, fulltext search, 3PID invites https://t.co/5alhXLLEGP”"
- "og:site_name" : "Twitter"
-}
-</code></pre>
-<ul>
-<li>Downloads the URL
-<ul>
-<li>If HTML, just stores it in RAM and parses it for OG meta tags
-<ul>
-<li>Download any media OG meta tags to the media repo, and refer to them in the OG via mxc:// URIs.</li>
-</ul>
</li>
-<li>If a media filetype we know we can thumbnail: store it on disk, and hand it to the thumbnailer. Generate OG meta tags from the thumbnailer contents.</li>
-<li>Otherwise, don't bother downloading further.</li>
-</ul>
+<li>Stores the result in the database cache.</li>
+</ol>
</li>
-</ul>
+<li>Returns the result.</li>
+</ol>
+<p>The in-memory cache expires after 1 hour.</p>
+<p>Expired entries in the database cache (and their associated media files) are
+deleted every 10 seconds. The default expiration time is 1 hour from download.</p>
<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="user-directory-api-implementation"><a class="header" href="#user-directory-api-implementation">User Directory API Implementation</a></h1>
<p>The user directory is currently maintained based on the 'visible' users
on this particular server - i.e. ones which your account shares a room with, or
@@ -7343,6 +7343,52 @@ DB corruption) get stale or out of sync. If this happens, for now the
solution to fix it is to execute the SQL <a href="https://github.com/matrix-org/synapse/blob/master/synapse/storage/schema/main/delta/53/user_dir_populate.sql">here</a>
and then restart synapse. This should then start a background task to
flush the current tables and regenerate the directory.</p>
+<h2 id="data-model"><a class="header" href="#data-model">Data model</a></h2>
+<p>There are five relevant tables that collectively form the "user directory".
+Three of them track a master list of all the users we could search for.
+The last two (collectively called the "search tables") track who can
+see who.</p>
+<p>From all of these tables we exclude three types of local user:</p>
+<ul>
+<li>support users</li>
+<li>appservice users</li>
+<li>deactivated users</li>
+</ul>
+<ul>
+<li>
+<p><code>user_directory</code>. This contains the user_id, display name and avatar we'll
+return when you search the directory.</p>
+<ul>
+<li>Because there's only one directory entry per user, it's important that we only
+ever put publicly visible names here. Otherwise we might leak a private
+nickname or avatar used in a private room.</li>
+<li>Indexed on rooms. Indexed on users.</li>
+</ul>
+</li>
+<li>
+<p><code>user_directory_search</code>. To be joined to <code>user_directory</code>. It contains an extra
+column that enables full text search based on user ids and display names.
+Different schemas for SQLite and Postgres with different code paths to match.</p>
+<ul>
+<li>Indexed on the full text search data. Indexed on users.</li>
+</ul>
+</li>
+<li>
+<p><code>user_directory_stream_pos</code>. When the initial background update to populate
+the directory is complete, we record a stream position here. This indicates
+that synapse should now listen for room changes and incrementally update
+the directory where necessary.</p>
+</li>
+<li>
+<p><code>users_in_public_rooms</code>. Contains associations between users and the public rooms they're in.
+Used to determine which users are in public rooms and should be publicly visible in the directory.</p>
+</li>
+<li>
+<p><code>users_who_share_private_rooms</code>. Rows are triples <code>(L, M, room id)</code> where <code>L</code>
+is a local user and <code>M</code> is a local or remote user. <code>L</code> and <code>M</code> should be
+different, but this isn't enforced by a constraint.</p>
+</li>
+</ul>
<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="message-retention-policies"><a class="header" href="#message-retention-policies">Message retention policies</a></h1>
<p>Synapse admins can enable support for message retention policies on
their homeserver. Message retention policies exist at a room level,
@@ -7518,9 +7564,14 @@ provenance of the modules they use on their homeserver and make sure the modules
running malicious code on their instance.</p>
<p>Also note that we are currently in the process of migrating module interfaces to this
system. While some interfaces might be compatible with it, others still require
-configuring modules in another part of Synapse's configuration file. Currently, only the
-spam checker interface is compatible with this new system.</p>
-<h2 id="writing-a-module"><a class="header" href="#writing-a-module">Writing a module</a></h2>
+configuring modules in another part of Synapse's configuration file.</p>
+<p>Currently, only the following pre-existing interfaces are compatible with this new system:</p>
+<ul>
+<li>spam checker</li>
+<li>third-party rules</li>
+<li>presence router</li>
+</ul>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="writing-a-module"><a class="header" href="#writing-a-module">Writing a module</a></h1>
<p>A module is a Python class that uses Synapse's module API to interact with the
homeserver. It can register callbacks that Synapse will call on specific operations, as
well as web resources to attach to Synapse's web server.</p>
@@ -7530,7 +7581,7 @@ either the output of the module's <code>parse_config</code> static method (see b
configuration associated with the module in Synapse's configuration file.</p>
<p>See the documentation for the <code>ModuleApi</code> class
<a href="https://github.com/matrix-org/synapse/blob/master/synapse/module_api/__init__.py">here</a>.</p>
-<h3 id="handling-the-modules-configuration"><a class="header" href="#handling-the-modules-configuration">Handling the module's configuration</a></h3>
+<h2 id="handling-the-modules-configuration"><a class="header" href="#handling-the-modules-configuration">Handling the module's configuration</a></h2>
<p>A module can implement the following static method:</p>
<pre><code class="language-python">@staticmethod
def parse_config(config: dict) -> dict
@@ -7540,7 +7591,7 @@ module. It may modify it (for example by parsing durations expressed as strings
"5d") into milliseconds, etc.), and return the modified dictionary. It may also verify
that the configuration is correct, and raise an instance of
<code>synapse.module_api.errors.ConfigError</code> if not.</p>
-<h3 id="registering-a-web-resource"><a class="header" href="#registering-a-web-resource">Registering a web resource</a></h3>
+<h2 id="registering-a-web-resource"><a class="header" href="#registering-a-web-resource">Registering a web resource</a></h2>
<p>Modules can register web resources onto Synapse's web server using the following module
API method:</p>
<pre><code class="language-python">def ModuleApi.register_web_resource(path: str, resource: IResource) -> None
@@ -7558,7 +7609,7 @@ interface (such as <a href="https://twistedmatrix.com/documents/current/api/twis
register a resource for the same path, the module that appears first in Synapse's
configuration file takes priority.</p>
<p>Modules <strong>must</strong> register their web resources in their <code>__init__</code> method.</p>
-<h3 id="registering-a-callback"><a class="header" href="#registering-a-callback">Registering a callback</a></h3>
+<h2 id="registering-a-callback"><a class="header" href="#registering-a-callback">Registering a callback</a></h2>
<p>Modules can use Synapse's module API to register callbacks. Callbacks are functions that
Synapse will call when performing specific actions. Callbacks must be asynchronous, and
are split in categories. A single module may implement callbacks from multiple categories,
@@ -7567,38 +7618,46 @@ callbacks for.</p>
<p>Modules can register callbacks using one of the module API's <code>register_[...]_callbacks</code>
methods. The callback functions are passed to these methods as keyword arguments, with
the callback name as the argument name and the function as its value. This is demonstrated
-in the example below. A <code>register_[...]_callbacks</code> method exists for each module type
-documented in this section.</p>
-<h4 id="spam-checker-callbacks"><a class="header" href="#spam-checker-callbacks">Spam checker callbacks</a></h4>
+in the example below. A <code>register_[...]_callbacks</code> method exists for each category.</p>
+<p>Callbacks for each category can be found on their respective page of the
+<a href="https://matrix-org.github.io/synapse">Synapse documentation website</a>.</p>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="spam-checker-callbacks"><a class="header" href="#spam-checker-callbacks">Spam checker callbacks</a></h1>
<p>Spam checker callbacks allow module developers to implement spam mitigation actions for
Synapse instances. Spam checker callbacks can be registered using the module API's
<code>register_spam_checker_callbacks</code> method.</p>
+<h2 id="callbacks"><a class="header" href="#callbacks">Callbacks</a></h2>
<p>The available spam checker callbacks are:</p>
+<h3 id="check_event_for_spam"><a class="header" href="#check_event_for_spam"><code>check_event_for_spam</code></a></h3>
<pre><code class="language-python">async def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool, str]
</code></pre>
<p>Called when receiving an event from a client or via federation. The module can return
either a <code>bool</code> to indicate whether the event must be rejected because of spam, or a <code>str</code>
to indicate the event must be rejected because of spam and to give a rejection reason to
forward to clients.</p>
+<h3 id="user_may_invite"><a class="header" href="#user_may_invite"><code>user_may_invite</code></a></h3>
<pre><code class="language-python">async def user_may_invite(inviter: str, invitee: str, room_id: str) -> bool
</code></pre>
<p>Called when processing an invitation. The module must return a <code>bool</code> indicating whether
the inviter can invite the invitee to the given room. Both inviter and invitee are
represented by their Matrix user ID (e.g. <code>@alice:example.com</code>).</p>
+<h3 id="user_may_create_room"><a class="header" href="#user_may_create_room"><code>user_may_create_room</code></a></h3>
<pre><code class="language-python">async def user_may_create_room(user: str) -> bool
</code></pre>
<p>Called when processing a room creation request. The module must return a <code>bool</code> indicating
whether the given user (represented by their Matrix user ID) is allowed to create a room.</p>
+<h3 id="user_may_create_room_alias"><a class="header" href="#user_may_create_room_alias"><code>user_may_create_room_alias</code></a></h3>
<pre><code class="language-python">async def user_may_create_room_alias(user: str, room_alias: "synapse.types.RoomAlias") -> bool
</code></pre>
<p>Called when trying to associate an alias with an existing room. The module must return a
<code>bool</code> indicating whether the given user (represented by their Matrix user ID) is allowed
to set the given alias.</p>
+<h3 id="user_may_publish_room"><a class="header" href="#user_may_publish_room"><code>user_may_publish_room</code></a></h3>
<pre><code class="language-python">async def user_may_publish_room(user: str, room_id: str) -> bool
</code></pre>
<p>Called when trying to publish a room to the homeserver's public rooms directory. The
module must return a <code>bool</code> indicating whether the given user (represented by their
Matrix user ID) is allowed to publish the given room.</p>
+<h3 id="check_username_for_spam"><a class="header" href="#check_username_for_spam"><code>check_username_for_spam</code></a></h3>
<pre><code class="language-python">async def check_username_for_spam(user_profile: Dict[str, str]) -> bool
</code></pre>
<p>Called when computing search results in the user directory. The module must return a
@@ -7611,6 +7670,7 @@ is represented as a dictionary with the following keys:</p>
</ul>
<p>The module is given a copy of the original dictionary, so modifying it from within the
module cannot modify a user's profile when included in user directory search results.</p>
+<h3 id="check_registration_for_spam"><a class="header" href="#check_registration_for_spam"><code>check_registration_for_spam</code></a></h3>
<pre><code class="language-python">async def check_registration_for_spam(
email_threepid: Optional[dict],
username: Optional[str],
@@ -7631,6 +7691,7 @@ second item is an IP address. These user agents and IP addresses are the ones th
used during the registration process.</li>
<li><code>auth_provider_id</code>: The identifier of the SSO authentication provider, if any.</li>
</ul>
+<h3 id="check_media_file_for_spam"><a class="header" href="#check_media_file_for_spam"><code>check_media_file_for_spam</code></a></h3>
<pre><code class="language-python">async def check_media_file_for_spam(
file_wrapper: "synapse.rest.media.v1.media_storage.ReadableFileWrapper",
file_info: "synapse.rest.media.v1._base.FileInfo",
@@ -7638,31 +7699,56 @@ used during the registration process.</li>
</code></pre>
<p>Called when storing a local or remote file. The module must return a boolean indicating
whether the given file can be stored in the homeserver's media store.</p>
-<h4 id="account-validity-callbacks"><a class="header" href="#account-validity-callbacks">Account validity callbacks</a></h4>
-<p>Account validity callbacks allow module developers to add extra steps to verify the
-validity on an account, i.e. see if a user can be granted access to their account on the
-Synapse instance. Account validity callbacks can be registered using the module API's
-<code>register_account_validity_callbacks</code> method.</p>
-<p>The available account validity callbacks are:</p>
-<pre><code class="language-python">async def is_user_expired(user: str) -> Optional[bool]
-</code></pre>
-<p>Called when processing any authenticated request (except for logout requests). The module
-can return a <code>bool</code> to indicate whether the user has expired and should be locked out of
-their account, or <code>None</code> if the module wasn't able to figure it out. The user is
-represented by their Matrix user ID (e.g. <code>@alice:example.com</code>).</p>
-<p>If the module returns <code>True</code>, the current request will be denied with the error code
-<code>ORG_MATRIX_EXPIRED_ACCOUNT</code> and the HTTP status code 403. Note that this doesn't
-invalidate the user's access token.</p>
-<pre><code class="language-python">async def on_user_registration(user: str) -> None
+<h2 id="example"><a class="header" href="#example">Example</a></h2>
+<p>The example below is a module that implements the spam checker callback
+<code>check_event_for_spam</code> to deny any message sent by users whose Matrix user IDs are
+mentioned in a configured list, and registers a web resource to the path
+<code>/_synapse/client/list_spam_checker/is_evil</code> that returns a JSON object indicating
+whether the provided user appears in that list.</p>
+<pre><code class="language-python">import json
+from typing import Union
+
+from twisted.web.resource import Resource
+from twisted.web.server import Request
+
+from synapse.module_api import ModuleApi
+
+
+class IsUserEvilResource(Resource):
+ def __init__(self, config):
+ super(IsUserEvilResource, self).__init__()
+ self.evil_users = config.get("evil_users") or []
+
+ def render_GET(self, request: Request):
+ user = request.args.get(b"user")[0]
+ request.setHeader(b"Content-Type", b"application/json")
+ return json.dumps({"evil": user in self.evil_users})
+
+
+class ListSpamChecker:
+ def __init__(self, config: dict, api: ModuleApi):
+ self.api = api
+ self.evil_users = config.get("evil_users") or []
+
+ self.api.register_spam_checker_callbacks(
+ check_event_for_spam=self.check_event_for_spam,
+ )
+
+ self.api.register_web_resource(
+ path="/_synapse/client/list_spam_checker/is_evil",
+ resource=IsUserEvilResource(config),
+ )
+
+ async def check_event_for_spam(self, event: "synapse.events.EventBase") -> Union[bool, str]:
+ return event.sender not in self.evil_users
</code></pre>
-<p>Called after successfully registering a user, in case the module needs to perform extra
-operations to keep track of them. (e.g. add them to a database table). The user is
-represented by their Matrix user ID.</p>
-<h4 id="third-party-rules-callbacks"><a class="header" href="#third-party-rules-callbacks">Third party rules callbacks</a></h4>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="third-party-rules-callbacks"><a class="header" href="#third-party-rules-callbacks">Third party rules callbacks</a></h1>
<p>Third party rules callbacks allow module developers to add extra checks to verify the
validity of incoming events. Third party event rules callbacks can be registered using
the module API's <code>register_third_party_rules_callbacks</code> method.</p>
+<h2 id="callbacks-1"><a class="header" href="#callbacks-1">Callbacks</a></h2>
<p>The available third party rules callbacks are:</p>
+<h3 id="check_event_allowed"><a class="header" href="#check_event_allowed"><code>check_event_allowed</code></a></h3>
<pre><code class="language-python">async def check_event_allowed(
event: "synapse.events.EventBase",
state_events: "synapse.types.StateMap",
@@ -7690,6 +7776,7 @@ that, it is recommended the module calls <code>event.get_dict()</code> to get th
dictionary, and modify the returned dictionary accordingly.</p>
<p>Note that replacing the event only works for events sent by local users, not for events
received over federation.</p>
+<h3 id="on_create_room"><a class="header" href="#on_create_room"><code>on_create_room</code></a></h3>
<pre><code class="language-python">async def on_create_room(
requester: "synapse.types.Requester",
request_content: dict,
@@ -7703,15 +7790,65 @@ for a list of possible parameters), and a boolean indicating whether the user pe
the request is a server admin.</p>
<p>Modules can modify the <code>request_content</code> (by e.g. adding events to its <code>initial_state</code>),
or deny the room's creation by raising a <code>module_api.errors.SynapseError</code>.</p>
-<h4 id="presence-router-callbacks"><a class="header" href="#presence-router-callbacks">Presence router callbacks</a></h4>
+<h3 id="check_threepid_can_be_invited"><a class="header" href="#check_threepid_can_be_invited"><code>check_threepid_can_be_invited</code></a></h3>
+<pre><code class="language-python">async def check_threepid_can_be_invited(
+ medium: str,
+ address: str,
+ state_events: "synapse.types.StateMap",
+) -> bool:
+</code></pre>
+<p>Called when processing an invite via a third-party identifier (i.e. email or phone number).
+The module must return a boolean indicating whether the invite can go through.</p>
+<h3 id="check_visibility_can_be_modified"><a class="header" href="#check_visibility_can_be_modified"><code>check_visibility_can_be_modified</code></a></h3>
+<pre><code class="language-python">async def check_visibility_can_be_modified(
+ room_id: str,
+ state_events: "synapse.types.StateMap",
+ new_visibility: str,
+) -> bool:
+</code></pre>
+<p>Called when changing the visibility of a room in the local public room directory. The
+visibility is a string that's either "public" or "private". The module must return a
+boolean indicating whether the change can go through.</p>
+<h2 id="example-1"><a class="header" href="#example-1">Example</a></h2>
+<p>The example below is a module that implements the third-party rules callback
+<code>check_event_allowed</code> to censor incoming messages as dictated by a third-party service.</p>
+<pre><code class="language-python">from typing import Optional, Tuple
+
+from synapse.module_api import ModuleApi
+
+_DEFAULT_CENSOR_ENDPOINT = "https://my-internal-service.local/censor-event"
+
+class EventCensorer:
+ def __init__(self, config: dict, api: ModuleApi):
+ self.api = api
+ self._endpoint = config.get("endpoint", _DEFAULT_CENSOR_ENDPOINT)
+
+ self.api.register_third_party_rules_callbacks(
+ check_event_allowed=self.check_event_allowed,
+ )
+
+ async def check_event_allowed(
+ self,
+ event: "synapse.events.EventBase",
+ state_events: "synapse.types.StateMap",
+ ) -> Tuple[bool, Optional[dict]]:
+ event_dict = event.get_dict()
+ new_event_content = await self.api.http_client.post_json_get_json(
+ uri=self._endpoint, post_json=event_dict,
+ )
+ event_dict["content"] = new_event_content
+ return event_dict
+</code></pre>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="presence-router-callbacks"><a class="header" href="#presence-router-callbacks">Presence router callbacks</a></h1>
<p>Presence router callbacks allow module developers to specify additional users (local or remote)
to receive certain presence updates from local users. Presence router callbacks can be
registered using the module API's <code>register_presence_router_callbacks</code> method.</p>
+<h2 id="callbacks-2"><a class="header" href="#callbacks-2">Callbacks</a></h2>
<p>The available presence router callbacks are:</p>
+<h3 id="get_users_for_states"><a class="header" href="#get_users_for_states"><code>get_users_for_states</code></a></h3>
<pre><code class="language-python">async def get_users_for_states(
- self,
state_updates: Iterable["synapse.api.UserPresenceState"],
-) -> Dict[str, Set["synapse.api.UserPresenceState"]]:
+) -> Dict[str, Set["synapse.api.UserPresenceState"]]
</code></pre>
<p><strong>Requires</strong> <code>get_interested_users</code> to also be registered</p>
<p>Called when processing updates to the presence state of one or more users. This callback can
@@ -7719,9 +7856,9 @@ be used to instruct the server to forward that presence state to specific users.
must return a dictionary that maps from Matrix user IDs (which can be local or remote) to the
<code>UserPresenceState</code> changes that they should be forwarded.</p>
<p>Synapse will then attempt to send the specified presence updates to each user when possible.</p>
+<h3 id="get_interested_users"><a class="header" href="#get_interested_users"><code>get_interested_users</code></a></h3>
<pre><code class="language-python">async def get_interested_users(
- self,
- user_id: str
+ user_id: str
) -> Union[Set[str], "synapse.module_api.PRESENCE_ALL_USERS"]
</code></pre>
<p><strong>Requires</strong> <code>get_users_for_states</code> to also be registered</p>
@@ -7733,383 +7870,84 @@ should return the Matrix user IDs of the users whose presence state they are all
query. The returned users can be local or remote. </p>
<p>Alternatively the callback can return <code>synapse.module_api.PRESENCE_ALL_USERS</code>
to indicate that the user should receive updates from all known users.</p>
-<p>For example, if the user <code>@alice:example.org</code> is passed to this method, and the Set
-<code>{"@bob:example.com", "@charlie:somewhere.org"}</code> is returned, this signifies that Alice
-should receive presence updates sent by Bob and Charlie, regardless of whether these users
-share a room.</p>
-<h3 id="porting-an-existing-module-that-uses-the-old-interface"><a class="header" href="#porting-an-existing-module-that-uses-the-old-interface">Porting an existing module that uses the old interface</a></h3>
-<p>In order to port a module that uses Synapse's old module interface, its author needs to:</p>
-<ul>
-<li>ensure the module's callbacks are all asynchronous.</li>
-<li>register their callbacks using one or more of the <code>register_[...]_callbacks</code> methods
-from the <code>ModuleApi</code> class in the module's <code>__init__</code> method (see <a href="modules.html#registering-a-callback">this section</a>
-for more info).</li>
-</ul>
-<p>Additionally, if the module is packaged with an additional web resource, the module
-should register this resource in its <code>__init__</code> method using the <code>register_web_resource</code>
-method from the <code>ModuleApi</code> class (see <a href="modules.html#registering-a-web-resource">this section</a> for
-more info).</p>
-<p>The module's author should also update any example in the module's configuration to only
-use the new <code>modules</code> section in Synapse's configuration file (see <a href="modules.html#using-modules">this section</a>
-for more info).</p>
-<h3 id="example"><a class="header" href="#example">Example</a></h3>
-<p>The example below is a module that implements the spam checker callback
-<code>user_may_create_room</code> to deny room creation to user <code>@evilguy:example.com</code>, and registers
-a web resource to the path <code>/_synapse/client/demo/hello</code> that returns a JSON object.</p>
-<pre><code class="language-python">import json
-
-from twisted.web.resource import Resource
-from twisted.web.server import Request
+<h2 id="example-2"><a class="header" href="#example-2">Example</a></h2>
+<p>The example below is a module that implements both presence router callbacks, and ensures
+that <code>@alice:example.org</code> receives all presence updates from <code>@bob:example.com</code> and
+<code>@charlie:somewhere.org</code>, regardless of whether Alice shares a room with any of them.</p>
+<pre><code class="language-python">from typing import Dict, Iterable, Set, Union
from synapse.module_api import ModuleApi
-class DemoResource(Resource):
- def __init__(self, config):
- super(DemoResource, self).__init__()
- self.config = config
-
- def render_GET(self, request: Request):
- name = request.args.get(b"name")[0]
- request.setHeader(b"Content-Type", b"application/json")
- return json.dumps({"hello": name})
-
-
-class DemoModule:
+class CustomPresenceRouter:
def __init__(self, config: dict, api: ModuleApi):
- self.config = config
self.api = api
- self.api.register_web_resource(
- path="/_synapse/client/demo/hello",
- resource=DemoResource(self.config),
- )
-
- self.api.register_spam_checker_callbacks(
- user_may_create_room=self.user_may_create_room,
+ self.api.register_presence_router_callbacks(
+ get_users_for_states=self.get_users_for_states,
+ get_interested_users=self.get_interested_users,
)
- @staticmethod
- def parse_config(config):
- return config
-
- async def user_may_create_room(self, user: str) -> bool:
- if user == "@evilguy:example.com":
- return False
-
- return True
-</code></pre>
-<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 style="color:red">
-This page of the Synapse documentation is now deprecated. For up to date
-documentation on setting up or writing a spam checker module, please see
-<a href="modules.html">this page</a>.
-</h2>
-<h1 id="handling-spam-in-synapse"><a class="header" href="#handling-spam-in-synapse">Handling spam in Synapse</a></h1>
-<p>Synapse has support to customize spam checking behavior. It can plug into a
-variety of events and affect how they are presented to users on your homeserver.</p>
-<p>The spam checking behavior is implemented as a Python class, which must be
-able to be imported by the running Synapse.</p>
-<h2 id="python-spam-checker-class"><a class="header" href="#python-spam-checker-class">Python spam checker class</a></h2>
-<p>The Python class is instantiated with two objects:</p>
-<ul>
-<li>Any configuration (see below).</li>
-<li>An instance of <code>synapse.module_api.ModuleApi</code>.</li>
-</ul>
-<p>It then implements methods which return a boolean to alter behavior in Synapse.
-All the methods must be defined.</p>
-<p>There's a generic method for checking every event (<code>check_event_for_spam</code>), as
-well as some specific methods:</p>
-<ul>
-<li><code>user_may_invite</code></li>
-<li><code>user_may_create_room</code></li>
-<li><code>user_may_create_room_alias</code></li>
-<li><code>user_may_publish_room</code></li>
-<li><code>check_username_for_spam</code></li>
-<li><code>check_registration_for_spam</code></li>
-<li><code>check_media_file_for_spam</code></li>
-</ul>
-<p>The details of each of these methods (as well as their inputs and outputs)
-are documented in the <code>synapse.events.spamcheck.SpamChecker</code> class.</p>
-<p>The <code>ModuleApi</code> class provides a way for the custom spam checker class to
-call back into the homeserver internals.</p>
-<p>Additionally, a <code>parse_config</code> method is mandatory and receives the plugin config
-dictionary. After parsing, It must return an object which will be
-passed to <code>__init__</code> later.</p>
-<h3 id="example-1"><a class="header" href="#example-1">Example</a></h3>
-<pre><code class="language-python">from synapse.spam_checker_api import RegistrationBehaviour
-
-class ExampleSpamChecker:
- def __init__(self, config, api):
- self.config = config
- self.api = api
-
- @staticmethod
- def parse_config(config):
- return config
-
- async def check_event_for_spam(self, foo):
- return False # allow all events
-
- async def user_may_invite(self, inviter_userid, invitee_userid, room_id):
- return True # allow all invites
-
- async def user_may_create_room(self, userid):
- return True # allow all room creations
-
- async def user_may_create_room_alias(self, userid, room_alias):
- return True # allow all room aliases
-
- async def user_may_publish_room(self, userid, room_id):
- return True # allow publishing of all rooms
-
- async def check_username_for_spam(self, user_profile):
- return False # allow all usernames
-
- async def check_registration_for_spam(
- self,
- email_threepid,
- username,
- request_info,
- auth_provider_id,
- ):
- return RegistrationBehaviour.ALLOW # allow all registrations
-
- async def check_media_file_for_spam(self, file_wrapper, file_info):
- return False # allow all media
-</code></pre>
-<h2 id="configuration-2"><a class="header" href="#configuration-2">Configuration</a></h2>
-<p>Modify the <code>spam_checker</code> section of your <code>homeserver.yaml</code> in the following
-manner:</p>
-<p>Create a list entry with the keys <code>module</code> and <code>config</code>.</p>
-<ul>
-<li>
-<p><code>module</code> should point to the fully qualified Python class that implements your
-custom logic, e.g. <code>my_module.ExampleSpamChecker</code>.</p>
-</li>
-<li>
-<p><code>config</code> is a dictionary that gets passed to the spam checker class.</p>
-</li>
-</ul>
-<h3 id="example-2"><a class="header" href="#example-2">Example</a></h3>
-<p>This section might look like:</p>
-<pre><code class="language-yaml">spam_checker:
- - module: my_module.ExampleSpamChecker
- config:
- # Enable or disable a specific option in ExampleSpamChecker.
- my_custom_option: true
-</code></pre>
-<p>More spam checkers can be added in tandem by appending more items to the list. An
-action is blocked when at least one of the configured spam checkers flags it.</p>
-<h2 id="examples"><a class="header" href="#examples">Examples</a></h2>
-<p>The <a href="https://github.com/matrix-org/mjolnir">Mjolnir</a> project is a full fledged
-example using the Synapse spam checking API, including a bot for dynamic
-configuration.</p>
-<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h2 style="color:red">
-This page of the Synapse documentation is now deprecated. For up to date
-documentation on setting up or writing a presence router module, please see
-<a href="modules.html">this page</a>.
-</h2>
-<h1 id="presence-router-module"><a class="header" href="#presence-router-module">Presence Router Module</a></h1>
-<p>Synapse supports configuring a module that can specify additional users
-(local or remote) to should receive certain presence updates from local
-users.</p>
-<p>Note that routing presence via Application Service transactions is not
-currently supported.</p>
-<p>The presence routing module is implemented as a Python class, which will
-be imported by the running Synapse.</p>
-<h2 id="python-presence-router-class"><a class="header" href="#python-presence-router-class">Python Presence Router Class</a></h2>
-<p>The Python class is instantiated with two objects:</p>
-<ul>
-<li>A configuration object of some type (see below).</li>
-<li>An instance of <code>synapse.module_api.ModuleApi</code>.</li>
-</ul>
-<p>It then implements methods related to presence routing.</p>
-<p>Note that one method of <code>ModuleApi</code> that may be useful is:</p>
-<pre><code class="language-python">async def ModuleApi.send_local_online_presence_to(users: Iterable[str]) -> None
-</code></pre>
-<p>which can be given a list of local or remote MXIDs to broadcast known, online user
-presence to (for those users that the receiving user is considered interested in).
-It does not include state for users who are currently offline, and it can only be
-called on workers that support sending federation. Additionally, this method must
-only be called from the process that has been configured to write to the
-the <a href="workers.html#stream-writers">presence stream</a>.
-By default, this is the main process, but another worker can be configured to do
-so.</p>
-<h3 id="module-structure"><a class="header" href="#module-structure">Module structure</a></h3>
-<p>Below is a list of possible methods that can be implemented, and whether they are
-required.</p>
-<h4 id="parse_config"><a class="header" href="#parse_config"><code>parse_config</code></a></h4>
-<pre><code class="language-python">def parse_config(config_dict: dict) -> Any
-</code></pre>
-<p><strong>Required.</strong> A static method that is passed a dictionary of config options, and
-should return a validated config object. This method is described further in
-<a href="presence_router_module.html#configuration">Configuration</a>.</p>
-<h4 id="get_users_for_states"><a class="header" href="#get_users_for_states"><code>get_users_for_states</code></a></h4>
-<pre><code class="language-python">async def get_users_for_states(
- self,
- state_updates: Iterable[UserPresenceState],
-) -> Dict[str, Set[UserPresenceState]]:
-</code></pre>
-<p><strong>Required.</strong> An asynchronous method that is passed an iterable of user presence
-state. This method can determine whether a given presence update should be sent to certain
-users. It does this by returning a dictionary with keys representing local or remote
-Matrix User IDs, and values being a python set
-of <code>synapse.handlers.presence.UserPresenceState</code> instances.</p>
-<p>Synapse will then attempt to send the specified presence updates to each user when
-possible.</p>
-<h4 id="get_interested_users"><a class="header" href="#get_interested_users"><code>get_interested_users</code></a></h4>
-<pre><code class="language-python">async def get_interested_users(self, user_id: str) -> Union[Set[str], str]
-</code></pre>
-<p><strong>Required.</strong> An asynchronous method that is passed a single Matrix User ID. This
-method is expected to return the users that the passed in user may be interested in the
-presence of. Returned users may be local or remote. The presence routed as a result of
-what this method returns is sent in addition to the updates already sent between users
-that share a room together. Presence updates are deduplicated.</p>
-<p>This method should return a python set of Matrix User IDs, or the object
-<code>synapse.events.presence_router.PresenceRouter.ALL_USERS</code> to indicate that the passed
-user should receive presence information for <em>all</em> known users.</p>
-<p>For clarity, if the user <code>@alice:example.org</code> is passed to this method, and the Set
-<code>{"@bob:example.com", "@charlie:somewhere.org"}</code> is returned, this signifies that Alice
-should receive presence updates sent by Bob and Charlie, regardless of whether these
-users share a room.</p>
-<h3 id="example-3"><a class="header" href="#example-3">Example</a></h3>
-<p>Below is an example implementation of a presence router class.</p>
-<pre><code class="language-python">from typing import Dict, Iterable, Set, Union
-from synapse.events.presence_router import PresenceRouter
-from synapse.handlers.presence import UserPresenceState
-from synapse.module_api import ModuleApi
-
-class PresenceRouterConfig:
- def __init__(self):
- # Config options with their defaults
- # A list of users to always send all user presence updates to
- self.always_send_to_users = [] # type: List[str]
-
- # A list of users to ignore presence updates for. Does not affect
- # shared-room presence relationships
- self.blacklisted_users = [] # type: List[str]
-
-class ExamplePresenceRouter:
- """An example implementation of synapse.presence_router.PresenceRouter.
- Supports routing all presence to a configured set of users, or a subset
- of presence from certain users to members of certain rooms.
-
- Args:
- config: A configuration object.
- module_api: An instance of Synapse's ModuleApi.
- """
- def __init__(self, config: PresenceRouterConfig, module_api: ModuleApi):
- self._config = config
- self._module_api = module_api
-
- @staticmethod
- def parse_config(config_dict: dict) -> PresenceRouterConfig:
- """Parse a configuration dictionary from the homeserver config, do
- some validation and return a typed PresenceRouterConfig.
-
- Args:
- config_dict: The configuration dictionary.
-
- Returns:
- A validated config object.
- """
- # Initialise a typed config object
- config = PresenceRouterConfig()
- always_send_to_users = config_dict.get("always_send_to_users")
- blacklisted_users = config_dict.get("blacklisted_users")
-
- # Do some validation of config options... otherwise raise a
- # synapse.config.ConfigError.
- config.always_send_to_users = always_send_to_users
- config.blacklisted_users = blacklisted_users
-
- return config
-
async def get_users_for_states(
self,
- state_updates: Iterable[UserPresenceState],
- ) -> Dict[str, Set[UserPresenceState]]:
- """Given an iterable of user presence updates, determine where each one
- needs to go. Returned results will not affect presence updates that are
- sent between users who share a room.
-
- Args:
- state_updates: An iterable of user presence state updates.
-
- Returns:
- A dictionary of user_id -> set of UserPresenceState that the user should
- receive.
- """
- destination_users = {} # type: Dict[str, Set[UserPresenceState]
-
- # Ignore any updates for blacklisted users
- desired_updates = set()
+ state_updates: Iterable["synapse.api.UserPresenceState"],
+ ) -> Dict[str, Set["synapse.api.UserPresenceState"]]:
+ res = {}
for update in state_updates:
- if update.state_key not in self._config.blacklisted_users:
- desired_updates.add(update)
+ if (
+ update.user_id == "@bob:example.com"
+ or update.user_id == "@charlie:somewhere.org"
+ ):
+ res.setdefault("@alice:example.com", set()).add(update)
- # Send all presence updates to specific users
- for user_id in self._config.always_send_to_users:
- destination_users[user_id] = desired_updates
-
- return destination_users
+ return res
async def get_interested_users(
self,
user_id: str,
- ) -> Union[Set[str], PresenceRouter.ALL_USERS]:
- """
- Retrieve a list of users that `user_id` is interested in receiving the
- presence of. This will be in addition to those they share a room with.
- Optionally, the object PresenceRouter.ALL_USERS can be returned to indicate
- that this user should receive all incoming local and remote presence updates.
-
- Note that this method will only be called for local users.
-
- Args:
- user_id: A user requesting presence updates.
-
- Returns:
- A set of user IDs to return additional presence updates for, or
- PresenceRouter.ALL_USERS to return presence updates for all other users.
- """
- if user_id in self._config.always_send_to_users:
- return PresenceRouter.ALL_USERS
+ ) -> Union[Set[str], "synapse.module_api.PRESENCE_ALL_USERS"]:
+ if user_id == "@alice:example.com":
+ return {"@bob:example.com", "@charlie:somewhere.org"}
return set()
</code></pre>
-<h4 id="a-note-on-get_users_for_states-and-get_interested_users"><a class="header" href="#a-note-on-get_users_for_states-and-get_interested_users">A note on <code>get_users_for_states</code> and <code>get_interested_users</code></a></h4>
-<p>Both of these methods are effectively two different sides of the same coin. The logic
-regarding which users should receive updates for other users should be the same
-between them.</p>
-<p><code>get_users_for_states</code> is called when presence updates come in from either federation
-or local users, and is used to either direct local presence to remote users, or to
-wake up the sync streams of local users to collect remote presence.</p>
-<p>In contrast, <code>get_interested_users</code> is used to determine the users that presence should
-be fetched for when a local user is syncing. This presence is then retrieved, before
-being fed through <code>get_users_for_states</code> once again, with only the syncing user's
-routing information pulled from the resulting dictionary.</p>
-<p>Their routing logic should thus line up, else you may run into unintended behaviour.</p>
-<h2 id="configuration-3"><a class="header" href="#configuration-3">Configuration</a></h2>
-<p>Once you've crafted your module and installed it into the same Python environment as
-Synapse, amend your homeserver config file with the following.</p>
-<pre><code class="language-yaml">presence:
- enabled: true
-
- presence_router:
- module: my_module.ExamplePresenceRouter
- config:
- # Any configuration options for your module. The below is an example.
- # of setting options for ExamplePresenceRouter.
- always_send_to_users: ["@presence_gobbler:example.org"]
- blacklisted_users:
- - "@alice:example.com"
- - "@bob:example.com"
- ...
-</code></pre>
-<p>The contents of <code>config</code> will be passed as a Python dictionary to the static
-<code>parse_config</code> method of your class. The object returned by this method will
-then be passed to the <code>__init__</code> method of your module as <code>config</code>.</p>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="account-validity-callbacks"><a class="header" href="#account-validity-callbacks">Account validity callbacks</a></h1>
+<p>Account validity callbacks allow module developers to add extra steps to verify the
+validity on an account, i.e. see if a user can be granted access to their account on the
+Synapse instance. Account validity callbacks can be registered using the module API's
+<code>register_account_validity_callbacks</code> method.</p>
+<p>The available account validity callbacks are:</p>
+<h3 id="is_user_expired"><a class="header" href="#is_user_expired"><code>is_user_expired</code></a></h3>
+<pre><code class="language-python">async def is_user_expired(user: str) -> Optional[bool]
+</code></pre>
+<p>Called when processing any authenticated request (except for logout requests). The module
+can return a <code>bool</code> to indicate whether the user has expired and should be locked out of
+their account, or <code>None</code> if the module wasn't able to figure it out. The user is
+represented by their Matrix user ID (e.g. <code>@alice:example.com</code>).</p>
+<p>If the module returns <code>True</code>, the current request will be denied with the error code
+<code>ORG_MATRIX_EXPIRED_ACCOUNT</code> and the HTTP status code 403. Note that this doesn't
+invalidate the user's access token.</p>
+<h3 id="on_user_registration"><a class="header" href="#on_user_registration"><code>on_user_registration</code></a></h3>
+<pre><code class="language-python">async def on_user_registration(user: str) -> None
+</code></pre>
+<p>Called after successfully registering a user, in case the module needs to perform extra
+operations to keep track of them. (e.g. add them to a database table). The user is
+represented by their Matrix user ID.</p>
+<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="porting-an-existing-module-that-uses-the-old-interface"><a class="header" href="#porting-an-existing-module-that-uses-the-old-interface">Porting an existing module that uses the old interface</a></h1>
+<p>In order to port a module that uses Synapse's old module interface, its author needs to:</p>
+<ul>
+<li>ensure the module's callbacks are all asynchronous.</li>
+<li>register their callbacks using one or more of the <code>register_[...]_callbacks</code> methods
+from the <code>ModuleApi</code> class in the module's <code>__init__</code> method (see <a href="modules/writing_a_module.html#registering-a-callback">this section</a>
+for more info).</li>
+</ul>
+<p>Additionally, if the module is packaged with an additional web resource, the module
+should register this resource in its <code>__init__</code> method using the <code>register_web_resource</code>
+method from the <code>ModuleApi</code> class (see <a href="modules/writing_a_module.html#registering-a-web-resource">this section</a> for
+more info).</p>
+<p>The module's author should also update any example in the module's configuration to only
+use the new <code>modules</code> section in Synapse's configuration file (see <a href="modules/index.html#using-modules">this section</a>
+for more info).</p>
<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="scaling-synapse-via-workers"><a class="header" href="#scaling-synapse-via-workers">Scaling synapse via workers</a></h1>
<p>For small instances it recommended to run Synapse in the default monolith mode.
For larger instances where performance is a concern it can be helpful to split
@@ -8274,6 +8112,8 @@ expressions:</p>
^/_matrix/federation/v1/user/devices/
^/_matrix/federation/v1/get_groups_publicised$
^/_matrix/key/v2/query
+^/_matrix/federation/unstable/org.matrix.msc2946/spaces/
+^/_matrix/federation/unstable/org.matrix.msc2946/hierarchy/
# Inbound federation transaction request
^/_matrix/federation/v1/send/
@@ -8285,6 +8125,9 @@ expressions:</p>
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
+^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/spaces$
+^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/hierarchy$
+^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
^/_matrix/client/(api/v1|r0|unstable)/devices$
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
@@ -9908,34 +9751,51 @@ clients in a confused state.</li>
the old room to the new.</li>
<li><code>new_room_id</code> - A string representing the room ID of the new room.</li>
</ul>
-<h2 id="undoing-room-shutdowns"><a class="header" href="#undoing-room-shutdowns">Undoing room shutdowns</a></h2>
-<p><em>Note</em>: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
+<h2 id="undoing-room-deletions"><a class="header" href="#undoing-room-deletions">Undoing room deletions</a></h2>
+<p><em>Note</em>: This guide may be outdated by the time you read it. By nature of room deletions being performed at the database level,
the structure can and does change without notice.</p>
-<p>First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
+<p>First, it's important to understand that a room deletion is very destructive. Undoing a deletion is not as simple as pretending it
never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
to recover at all:</p>
<ul>
<li>If the room was invite-only, your users will need to be re-invited.</li>
<li>If the room no longer has any members at all, it'll be impossible to rejoin.</li>
-<li>The first user to rejoin will have to do so via an alias on a different server.</li>
+<li>The first user to rejoin will have to do so via an alias on a different
+server (or receive an invite from a user on a different server).</li>
</ul>
<p>With all that being said, if you still want to try and recover the room:</p>
<ol>
+<li>
+<p>If the room was <code>block</code>ed, you must unblock it on your server. This can be
+accomplished as follows:</p>
+<ol>
<li>For safety reasons, shut down Synapse.</li>
<li>In the database, run <code>DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';</code>
<ul>
<li>For caution: it's recommended to run this in a transaction: <code>BEGIN; DELETE ...;</code>, verify you got 1 result, then <code>COMMIT;</code>.</li>
-<li>The room ID is the same one supplied to the shutdown room API, not the Content Violation room.</li>
+<li>The room ID is the same one supplied to the delete room API, not the Content Violation room.</li>
</ul>
</li>
<li>Restart Synapse.</li>
</ol>
-<p>You will have to manually handle, if you so choose, the following:</p>
-<ul>
-<li>Aliases that would have been redirected to the Content Violation room.</li>
-<li>Users that would have been booted from the room (and will have been force-joined to the Content Violation room).</li>
-<li>Removal of the Content Violation room if desired.</li>
-</ul>
+<p>This step is unnecessary if <code>block</code> was not set.</p>
+</li>
+<li>
+<p>Any room aliases on your server that pointed to the deleted room may have
+been deleted, or redirected to the Content Violation room. These will need
+to be restored manually.</p>
+</li>
+<li>
+<p>Users on your server that were in the deleted room will have been kicked
+from the room. Consider whether you want to update their membership
+(possibly via the <a href="admin_api/room_membership.html">Edit Room Membership API</a>) or let
+them handle rejoining themselves.</p>
+</li>
+<li>
+<p>If <code>new_room_user_id</code> was given, a 'Content Violation' will have been
+created. Consider whether you want to delete that roomm.</p>
+</li>
+</ol>
<h2 id="deprecated-endpoint"><a class="header" href="#deprecated-endpoint">Deprecated endpoint</a></h2>
<p>The previous deprecated API will be removed in a future release, it was:</p>
<pre><code>POST /_synapse/admin/v1/rooms/<room_id>/delete
@@ -9953,7 +9813,7 @@ optionally be specified, e.g.:</p>
</code></pre>
<h1 id="forward-extremities-admin-api"><a class="header" href="#forward-extremities-admin-api">Forward Extremities Admin API</a></h1>
<p>Enables querying and deleting forward extremities from rooms. When a lot of forward
-extremities accumulate in a room, performance can become degraded. For details, see
+extremities accumulate in a room, performance can become degraded. For details, see
<a href="https://github.com/matrix-org/synapse/issues/1760">#1760</a>.</p>
<h2 id="check-for-forward-extremities"><a class="header" href="#check-for-forward-extremities">Check for forward extremities</a></h2>
<p>To check the status of forward extremities for a room:</p>
@@ -9973,7 +9833,7 @@ extremities accumulate in a room, performance can become degraded. For details,
}
</code></pre>
<h2 id="deleting-forward-extremities"><a class="header" href="#deleting-forward-extremities">Deleting forward extremities</a></h2>
-<p><strong>WARNING</strong>: Please ensure you know what you're doing and have read
+<p><strong>WARNING</strong>: Please ensure you know what you're doing and have read
the related issue <a href="https://github.com/matrix-org/synapse/issues/1760">#1760</a>.
Under no situations should this API be executed as an automated maintenance task!</p>
<p>If a room has lots of forward extremities, the extra can be
@@ -11142,7 +11002,7 @@ debugging.</p>
<p>Note that this will give administrative access to synapse to <strong>all users</strong> with
shell access to the server. It should therefore <strong>not</strong> be enabled in
environments where untrusted users have shell access.</p>
-<hr />
+<h2 id="configuring-the-manhole"><a class="header" href="#configuring-the-manhole">Configuring the manhole</a></h2>
<p>To enable it, first uncomment the <code>manhole</code> listener configuration in
<code>homeserver.yaml</code>. The configuration is slightly different if you're using docker.</p>
<h4 id="docker-config"><a class="header" href="#docker-config">Docker config</a></h4>
@@ -11170,12 +11030,28 @@ The <code>bind_addresses</code> in the example below is important: it ensures th
bind_addresses: ['::1', '127.0.0.1']
type: manhole
</code></pre>
-<h4 id="accessing-synapse-manhole"><a class="header" href="#accessing-synapse-manhole">Accessing synapse manhole</a></h4>
+<h3 id="security-settings"><a class="header" href="#security-settings">Security settings</a></h3>
+<p>The following config options are available:</p>
+<ul>
+<li><code>username</code> - The username for the manhole (defaults to <code>matrix</code>)</li>
+<li><code>password</code> - The password for the manhole (defaults to <code>rabbithole</code>)</li>
+<li><code>ssh_priv_key</code> - The path to a private SSH key (defaults to a hardcoded value)</li>
+<li><code>ssh_pub_key</code> - The path to a public SSH key (defaults to a hardcoded value)</li>
+</ul>
+<p>For example:</p>
+<pre><code class="language-yaml">manhole_settings:
+ username: manhole
+ password: mypassword
+ ssh_priv_key: "/home/synapse/manhole_keys/id_rsa"
+ ssh_pub_key: "/home/synapse/manhole_keys/id_rsa.pub"
+</code></pre>
+<h2 id="accessing-synapse-manhole"><a class="header" href="#accessing-synapse-manhole">Accessing synapse manhole</a></h2>
<p>Then restart synapse, and point an ssh client at port 9000 on localhost, using
-the username <code>matrix</code>:</p>
+the username and password configured in <code>homeserver.yaml</code> - with the default
+configuration, this would be:</p>
<pre><code class="language-bash">ssh -p9000 matrix@localhost
</code></pre>
-<p>The password is <code>rabbithole</code>.</p>
+<p>Then enter the password when prompted (the default is <code>rabbithole</code>).</p>
<p>This gives a Python REPL in which <code>hs</code> gives access to the
<code>synapse.server.HomeServer</code> object - which in turn gives access to many other
parts of the process.</p>
@@ -12275,7 +12151,7 @@ and add it to each log record.</p>
<p>Logcontexts are also used for CPU and database accounting, so that we
can track which requests were responsible for high CPU use or database
activity.</p>
-<p>The <code>synapse.logging.context</code> module provides a facilities for managing
+<p>The <code>synapse.logging.context</code> module provides facilities for managing
the current log context (as well as providing the <code>LoggingContextFilter</code>
class).</p>
<p>Asynchronous functions make the whole thing complicated, so this document describes
@@ -12552,7 +12428,7 @@ and the awaitable chain is now orphaned, and will be garbage-collected at
some point. Note that <code>await_something_interesting</code> is a coroutine,
which Python implements as a generator function. When Python
garbage-collects generator functions, it gives them a chance to
-clean up by making the <code>async</code> (or <code>yield</code>) raise a <code>GeneratorExit</code>
+clean up by making the <code>await</code> (or <code>yield</code>) raise a <code>GeneratorExit</code>
exception. In our case, that means that the <code>__exit__</code> handler of
<code>PreserveLoggingContext</code> will carefully restore the request context, but
there is now nothing waiting for its return, so the request context is
@@ -12693,7 +12569,7 @@ connection errors.</p>
received for each stream so that on reconneciton it can start streaming
from the correct place. Note: not all RDATA have valid tokens due to
batching. See <code>RdataCommand</code> for more details.</p>
-<h3 id="example-4"><a class="header" href="#example-4">Example</a></h3>
+<h3 id="example-3"><a class="header" href="#example-3">Example</a></h3>
<p>An example iteraction is shown below. Each line is prefixed with '>'
or '<' to indicate which side is sending, these are <em>not</em> included on
the wire:</p>
@@ -12989,7 +12865,7 @@ graph), and one where we remove redundant links (the transitive reduction of the
links graph) e.g. if we have chains <code>C3 -> C2 -> C1</code> then the link <code>C3 -> C1</code>
would not be stored. Synapse uses the former implementations so that it doesn't
need to recurse to test reachability between chains.</p>
-<h3 id="example-5"><a class="header" href="#example-5">Example</a></h3>
+<h3 id="example-4"><a class="header" href="#example-4">Example</a></h3>
<p>An example auth graph would look like the following, where chains have been
formed based on type/state_key and are denoted by colour and are labelled with
<code>(chain ID, sequence number)</code>. Links are denoted by the arrows (links in grey
@@ -13057,7 +12933,7 @@ sqlite3 database indexed by <code>media_id</code>.</p>
remote content is assigned a local <code>"filesystem_id"</code> to ensure that the
directory structure <code>"remote_content/server_name/aa/bb/ccccccccdddddddddddd"</code>
is appropriate. Thumbnails for remote content are stored under
-<code>"remote_thumbnails/server_name/..."</code></p>
+<code>"remote_thumbnail/server_name/..."</code></p>
<div id="chapter_begin" style="break-before: page; page-break-before: always;"></div><h1 id="room-and-user-statistics"><a class="header" href="#room-and-user-statistics">Room and User Statistics</a></h1>
<p>Synapse maintains room and user statistics in various tables. These can be used
for administrative purposes but are also used when generating the public room
|
