From 7c5dd6ffb85f208632f7a2018a922b5ef2083c18 Mon Sep 17 00:00:00 2001
From: anoadragon453
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db
) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
+synctl stop
cp homeserver.db homeserver.db.snapshot
-./synctl start
+synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
@@ -820,10 +820,10 @@ run:
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml
:
-./synctl stop
+synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
-./synctl start
+synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
@@ -1374,8 +1374,9 @@ turn_allow_guests: True
After updating the homeserver configuration, you must restart synapse:
- If you use synctl:
-
cd /where/you/run/synapse
-./synctl restart
+# Depending on how Synapse is installed, synctl may already be on
+# your PATH. If not, you may need to activate a virtual environment.
+synctl restart
- If you use systemd:
@@ -1603,7 +1604,7 @@ pip install --upgrade .
-
Restart Synapse:
-./synctl restart
+synctl restart
@@ -1638,6 +1639,37 @@ dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
+Upgrading to v1.56.0
+Groups/communities feature has been deprecated
+The non-standard groups/communities feature in Synapse has been deprecated and will
+be disabled by default in Synapse v1.58.0.
+You can test disabling it by adding the following to your homeserver configuration:
+experimental_features:
+ groups_enabled: false
+
+Upgrading to v1.55.0
+synctl
script has been moved
+The synctl
script
+has been made an
+entry point
+and no longer exists at the root of Synapse's source tree. If you wish to use
+synctl
to manage your homeserver, you should invoke synctl
directly, e.g.
+synctl start
instead of ./synctl start
or /path/to/synctl start
.
+You will need to ensure synctl
is on your PATH
.
+
+- This is automatically the case when using
+Debian packages or
+docker images
+provided by Matrix.org.
+- When installing from a wheel, sdist, or PyPI, a
synctl
executable is added
+to your Python installation's bin
. This should be on your PATH
+automatically, though you might need to activate a virtual environment
+depending on how you installed Synapse.
+
+Compatibility dropped for Mjolnir 1.3.1 and earlier
+Synapse v1.55.0 drops support for Mjolnir 1.3.1 and earlier.
+If you use the Mjolnir module to moderate your homeserver,
+please upgrade Mjolnir to version 1.3.2 or later before upgrading Synapse.
Upgrading to v1.54.0
Legacy structured logging configuration removal
This release removes support for the structured: true
logging configuration
@@ -2904,7 +2936,8 @@ release of Synapse.
Running a demo federation of Synapses
If you want to get up and running quickly with a trio of homeservers in a
private federation, there is a script in the demo
directory. This is mainly
-useful just for development purposes. See demo/README.
+useful just for development purposes. See
+demo scripts.
Configuration
This section contains information on tweaking Synapse via the various options in the configuration file. A configuration
file should have been generated when you installed Synapse.
@@ -4865,8 +4898,14 @@ saml2_config:
#
# localpart_template: Jinja2 template for the localpart of the MXID.
# If this is not set, the user will be prompted to choose their
-# own username (see 'sso_auth_account_details.html' in the 'sso'
-# section of this file).
+# own username (see the documentation for the
+# 'sso_auth_account_details.html' template). This template can
+# use the 'localpart_from_email' filter.
+#
+# confirm_localpart: Whether to prompt the user to validate (or
+# change) the generated localpart (see the documentation for the
+# 'sso_auth_account_details.html' template), instead of
+# registering the account right away.
#
# display_name_template: Jinja2 template for the display name to set
# on first login. If unset, no displayname will be set.
@@ -5647,6 +5686,38 @@ redis:
# Optional password if configured on the Redis instance
#
#password: <secret_password>
+
+
+## Background Updates ##
+
+# Background updates are database updates that are run in the background in batches.
+# The duration, minimum batch size, default batch size, whether to sleep between batches and if so, how long to
+# sleep can all be configured. This is helpful to speed up or slow down the updates.
+#
+background_updates:
+ # How long in milliseconds to run a batch of background updates for. Defaults to 100. Uncomment and set
+ # a time to change the default.
+ #
+ #background_update_duration_ms: 500
+
+ # Whether to sleep between updates. Defaults to True. Uncomment to change the default.
+ #
+ #sleep_enabled: false
+
+ # If sleeping between updates, how long in milliseconds to sleep for. Defaults to 1000. Uncomment
+ # and set a duration to change the default.
+ #
+ #sleep_duration_ms: 300
+
+ # Minimum size a batch of background updates can be. Must be greater than 0. Defaults to 1. Uncomment and
+ # set a size to change the default.
+ #
+ #min_batch_size: 10
+
+ # The batch size to use for the first iteration of a new background update. The default is 100.
+ # Uncomment and set a size to change the default.
+ #
+ #default_batch_size: 50
Below is a sample logging configuration file. This file can be tweaked to control how your @@ -5902,6 +5973,10 @@ already available as part of Jinja 2:
Turns a mxc://
URL for media content into an HTTP(S) one using the homeserver's
public_baseurl
configuration setting as the URL's base.
Example: message.sender_avatar_url|mxc_to_http(32,32)
localpart_from_email(address: str) -> str
+
+Returns the local part of an email address (e.g. alice
in alice@example.com
).
Example: user.email_address|localpart_from_email
Below are the templates Synapse will look for when generating the content of an email:
user_attributes
: an object containing details about the user that
we received from the IdP. May have the following attributes:
display_name
: the user's display nameemails
: a list of email addresseslocalpart
: the local part of the Matrix user ID to register,
+if localpart_template
is set in the mapping provider configuration (empty
+string if not)
The template should render a form which submits the following fields:check_event_for_spam
instead.
If multiple modules implement this callback, Synapse runs them all in order.
+check_can_shutdown_room
First introduced in Synapse v1.55.0
+async def check_can_shutdown_room(
+ user_id: str, room_id: str,
+) -> bool:
+
+Called when an admin user requests the shutdown of a room. The module must return a
+boolean indicating whether the shutdown can go through. If the callback returns False
,
+the shutdown will not proceed and the caller will see a M_FORBIDDEN
error.
If multiple modules implement this callback, they will be considered in order. If a
+callback returns True
, Synapse falls through to the next one. The value of the first
+callback that does not return True
will be used. If this happens, Synapse will not call
+any of the subsequent implementations of this callback.
check_can_deactivate_user
First introduced in Synapse v1.55.0
+async def check_can_deactivate_user(
+ user_id: str, by_admin: bool,
+) -> bool:
+
+Called when the deactivation of a user is requested. User deactivation can be
+performed by an admin or the user themselves, so developers are encouraged to check the
+requester when implementing this callback. The module must return a
+boolean indicating whether the deactivation can go through. If the callback returns False
,
+the deactivation will not proceed and the caller will see a M_FORBIDDEN
error.
The module is passed two parameters, user_id
which is the ID of the user being deactivated, and by_admin
which is True
if the request is made by a serve admin, and False
otherwise.
If multiple modules implement this callback, they will be considered in order. If a
+callback returns True
, Synapse falls through to the next one. The value of the first
+callback that does not return True
will be used. If this happens, Synapse will not call
+any of the subsequent implementations of this callback.
on_profile_update
First introduced in Synapse v1.54.0
async def on_profile_update(
@@ -9043,8 +9150,10 @@ streams (such as events) off of the main process to a particular worker. (This
is only supported with Redis-based replication.)
To enable this, the worker must have a HTTP replication listener configured,
have a worker_name
and be listed in the instance_map
config. The same worker
-can handle multiple streams. For example, to move event persistence off to a
-dedicated worker, the shared configuration would include:
+can handle multiple streams, but unless otherwise documented, each stream can only
+have a single writer.
+For example, to move event persistence off to a dedicated worker, the shared
+configuration would include:
instance_map:
event_persister1:
host: localhost
@@ -9057,8 +9166,8 @@ stream_writers:
be routed to the workers handling that stream. See below for the currently supported
streams and the endpoints associated with them:
The events
stream
-The events
stream also experimentally supports having multiple writers, where
-work is sharded between them by room ID. Note that you must restart all worker
+
The events
stream experimentally supports having multiple writers, where work
+is sharded between them by room ID. Note that you must restart all worker
instances when adding or removing event persisters. An example stream_writers
configuration with multiple writers:
stream_writers:
@@ -9067,30 +9176,30 @@ configuration with multiple writers:
- event_persister2
The typing
stream
-The following endpoints should be routed directly to the workers configured as
-stream writers for the typing
stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the typing
stream:
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing
The to_device
stream
-The following endpoints should be routed directly to the workers configured as
-stream writers for the to_device
stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the to_device
stream:
^/_matrix/client/(api/v1|r0|v3|unstable)/sendToDevice/
The account_data
stream
-The following endpoints should be routed directly to the workers configured as
-stream writers for the account_data
stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the account_data
stream:
^/_matrix/client/(api/v1|r0|v3|unstable)/.*/tags
^/_matrix/client/(api/v1|r0|v3|unstable)/.*/account_data
The receipts
stream
-The following endpoints should be routed directly to the workers configured as
-stream writers for the receipts
stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the receipts
stream:
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/receipt
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/read_markers
The presence
stream
-The following endpoints should be routed directly to the workers configured as
-stream writers for the presence
stream:
+The following endpoints should be routed directly to the worker configured as
+the stream writer for the presence
stream:
^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
Background tasks
@@ -9369,7 +9478,7 @@ server admin. (Note that a server admin is distinct from a room admin.)
UPDATE users SET admin = 1 WHERE name = '@foo:bar.com';
A new server admin user can also be created using the register_new_matrix_user
-command. This is a script that is located in the scripts/
directory, or possibly
+command. This is a script that is distributed as part of synapse. It is possibly
already on your $PATH
depending on how Synapse was installed.
Finding your user's access_token
is client-dependent, but will usually be shown in the client's settings.
Making an Admin API request
@@ -13251,6 +13360,15 @@ accept anonymous contributions at this time.
Git allows you to add this signoff automatically when using the -s
flag to git commit
, which uses the name and email set in your
user.name
and user.email
git configs.
+Private Sign off
+If you would like to provide your legal name privately to the Matrix.org
+Foundation (instead of in a public commit or comment), you can do so
+by emailing your legal name and a link to the pull request to
+dco@matrix.org.
+It helps to include "sign off" or similar in the subject line. You will then
+be instructed further.
+Once private sign off is complete, doing so for future contributions will not
+be required.
10. Turn feedback into better code.
Once the Pull Request is opened, you will see a few things:
@@ -13461,7 +13579,7 @@ frobber:
#distance: 100
Note that the sample configuration is generated from the synapse code
-and is maintained by a script, scripts-dev/generate_sample_config
.
+and is maintained by a script, scripts-dev/generate_sample_config.sh
.
Making sure that the output from this script matches the desired format
is left as an exercise for the reader!
Synapse Release Cycle
@@ -13615,6 +13733,40 @@ most intuitive name. ^
[3]: Very, very occasionally (I think this has happened once in
the history of Synapse), we've had two releases in flight at once. Obviously,
release-v1.2
is more-stable than release-v1.3
. ^
+Synapse demo setup
+DO NOT USE THESE DEMO SERVERS IN PRODUCTION
+Requires you to have a Synapse development environment setup.
+The demo setup allows running three federation Synapse servers, with server
+names localhost:8080
, localhost:8081
, and localhost:8082
.
+You can access them via any Matrix client over HTTP at localhost:8080
,
+localhost:8081
, and localhost:8082
or over HTTPS at localhost:8480
,
+localhost:8481
, and localhost:8482
.
+To enable the servers to communicate, self-signed SSL certificates are generated
+and the servers are configured in a highly insecure way, including:
+
+- Not checking certificates over federation.
+- Not verifying keys.
+
+The servers are configured to store their data under demo/8080
, demo/8081
, and
+demo/8082
. This includes configuration, logs, SQLite databases, and media.
+Note that when joining a public room on a different HS via "#foo:bar.net", then
+you are (in the current impl) joining a room with room_id "foo". This means that
+it won't work if your HS already has a room with that name.
+Using the demo scripts
+There's three main scripts with straightforward purposes:
+
+start.sh
will start the Synapse servers, generating any missing configuration.
+
+- This accepts a single parameter
--no-rate-limit
to "disable" rate limits
+(they actually still exist, but are very high).
+
+
+stop.sh
will stop the Synapse servers.
+clean.sh
will delete the configuration, databases, log files, etc.
+
+To start a completely new set of servers, run:
+./demo/stop.sh; ./demo/clean.sh && ./demo/start.sh
+
OpenTracing
Background
OpenTracing is a semi-standard being adopted by a number of distributed
@@ -13821,9 +13973,9 @@ same as integers.
-
Any new boolean column must be added to the BOOLEAN_COLUMNS
list in
-scripts/synapse_port_db
. This tells the port script to cast the integer
-value from SQLite to a boolean before writing the value to the postgres
-database.
+synapse/_scripts/synapse_port_db.py
. This tells the port script to cast
+the integer value from SQLite to a boolean before writing the value to the
+postgres database.
-
Before SQLite 3.23, TRUE
and FALSE
were not recognised as constants by
@@ -14534,24 +14686,62 @@ incrementing integer, but backfilled events start with stream_ordering=-1<
rather than skipping any that arrived late; whereas if you're looking at a
historical section of timeline (i.e. /messages
), you want to see the best
representation of the state of the room as others were seeing it at the time.
+Outliers
+We mark an event as an outlier
when we haven't figured out the state for the
+room at that point in the DAG yet. They are "floating" events that we haven't
+yet correlated to the DAG.
+Outliers typically arise when we fetch the auth chain or state for a given
+event. When that happens, we just grab the events in the state/auth chain,
+without calculating the state at those events, or backfilling their
+prev_events
.
+So, typically, we won't have the prev_events
of an outlier
in the database,
+(though it's entirely possible that we might have them for some other
+reason). Other things that make outliers different from regular events:
+
+-
+
We don't have state for them, so there should be no entry in
+event_to_state_groups
for an outlier. (In practice this isn't always
+the case, though I'm not sure why: see https://github.com/matrix-org/synapse/issues/12201).
+
+-
+
We don't record entries for them in the event_edges
,
+event_forward_extremeties
or event_backward_extremities
tables.
+
+
+Since outliers are not tied into the DAG, they do not normally form part of the
+timeline sent down to clients via /sync
or /messages
; however there is an
+exception:
+Out-of-band membership events
+A special case of outlier events are some membership events for federated rooms
+that we aren't full members of. For example:
+
+- invites received over federation, before we join the room
+- rejections for said invites
+- knock events for rooms that we would like to join but have not yet joined.
+
+In all the above cases, we don't have the state for the room, which is why they
+are treated as outliers. They are a bit special though, in that they are
+proactively sent to clients via /sync
.
Forward extremity
-Most-recent-in-time events in the DAG which are not referenced by any other events' prev_events
yet.
-The forward extremities of a room are used as the prev_events
when the next event is sent.
+Most-recent-in-time events in the DAG which are not referenced by any other
+events' prev_events
yet. (In this definition, outliers, rejected events, and
+soft-failed events don't count.)
+The forward extremities of a room (or at least, a subset of them, if there are
+more than ten) are used as the prev_events
when the next event is sent.
+The "current state" of a room (ie: the state which would be used if we
+generated a new event) is, therefore, the resolution of the room states
+at each of the forward extremities.
Backward extremity
The current marker of where we have backfilled up to and will generally be the
prev_events
of the oldest-in-time events we have in the DAG. This gives a starting point when
backfilling history.
-When we persist a non-outlier event, we clear it as a backward extremity and set
-all of its prev_events
as the new backward extremities if they aren't already
-persisted in the events
table.
-Outliers
-We mark an event as an outlier
when we haven't figured out the state for the
-room at that point in the DAG yet.
-We won't necessarily have the prev_events
of an outlier
in the database,
-but it's entirely possible that we might.
-For example, when we fetch the event auth chain or state for a given event, we
-mark all of those claimed auth events as outliers because we haven't done the
-state calculation ourself.
+Note that, unlike forward extremities, we typically don't have any backward
+extremity events themselves in the database - or, if we do, they will be "outliers" (see
+above). Either way, we don't expect to have the room state at a backward extremity.
+When we persist a non-outlier event, if it was previously a backward extremity,
+we clear it as a backward extremity and set all of its prev_events
as the new
+backward extremities if they aren't already persisted as non-outliers. This
+therefore keeps the backward extremities up-to-date.
State groups
For every non-outlier event we need to know the state at that event. Instead of
storing the full state for each event in the DB (i.e. a event_id -> state
@@ -14768,28 +14958,29 @@ to constantly update their platform dependencies to the latest versions.
Anything that requires modifying the device list #7721 will take a while to propagate, again taking the client "Offline" until it's complete. This includes signing in and out, editing the public name and verifying e2ee. The main mitigation I recommend is to keep long-running sessions open e.g. by using Firefox SSB "Use this site in App mode" or Chromium PWA "Install Element".
Recommended configuration
Put the below in a new file at /etc/matrix-synapse/conf.d/sbc.yaml to override the defaults in homeserver.yaml.
-# Set to false to disable presence tracking on this homeserver.
+# Disable presence tracking, which is currently fairly resource intensive
+# More info: https://github.com/matrix-org/synapse/issues/9478
use_presence: false
-# When this is enabled, the room "complexity" will be checked before a user
-# joins a new remote room. If it is above the complexity limit, the server will
-# disallow joining, or will instantly leave.
+# Set a small complexity limit, preventing users from joining large rooms
+# which may be resource-intensive to remain a part of.
+#
+# Note that this will not prevent users from joining smaller rooms that
+# eventually become complex.
limit_remote_rooms:
- # Uncomment to enable room complexity checking.
- #enabled: true
+ enabled: true
complexity: 3.0
# Database configuration
database:
+ # Use postgres for the best performance
name: psycopg2
args:
user: matrix-synapse
- # Generate a long, secure one with a password manager
+ # Generate a long, secure password using a password manager
password: hunter2
database: matrix-synapse
host: localhost
- cp_min: 5
- cp_max: 10
Currently the complexity is measured by current_state_events / 500. You can find join times and your most complex rooms like this:
admin@homeserver:~$ zgrep '/client/r0/join/' /var/log/matrix-synapse/homeserver.log* | awk '{print $18, $25}' | sort --human-numeric-sort
--
cgit 1.5.1