From 6d67e2f8c53f5391ea20a39135ee5e2aa2b0ed62 Mon Sep 17 00:00:00 2001 From: babolivier Date: Tue, 19 Jul 2022 14:14:03 +0000 Subject: deploy: 6fccd72f42ac76b185cc20ed258792ff94ee81fb --- latest/print.html | 411 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 303 insertions(+), 108 deletions(-) (limited to 'latest/print.html') diff --git a/latest/print.html b/latest/print.html index a37d107fc5..914e5e569f 100644 --- a/latest/print.html +++ b/latest/print.html @@ -77,7 +77,7 @@ @@ -386,7 +386,9 @@ python -m synapse.app.homeserver \ --generate-config \ --report-stats=[yes|no] -

... substituting an appropriate value for --server-name.

+

... substituting an appropriate value for --server-name and choosing whether +or not to report usage statistics (hostname, Synapse version, uptime, total +users, etc.) to the developers via the --report-stats argument.

This command will generate you a config file that you can then customise, but it will also generate a set of keys for you. These keys will allow your homeserver to identify itself to other homeserver, so don't lose or delete them. It would be @@ -501,11 +503,11 @@ over HTTPS.

Alternatively, you can configure Synapse to expose an HTTPS port. To do so, you will need to edit homeserver.yaml, as follows:

-
  - port: 8448
+
listeners:
+  - port: 8448
     type: http
     tls: true
     resources:
@@ -513,9 +515,13 @@ each line). The relevant lines are like this:
 

+

databases

+

The databases option allows specifying a mapping between certain database tables and +database host details, spreading the load of a single Synapse instance across multiple +database backends. This is often referred to as "database sharding". This option is only +supported for PostgreSQL database backends.

+

Important note: This is a supported option, but is not currently used in production by the +Matrix.org Foundation. Proceed with caution and always make backups.

+

databases is a dictionary of arbitrarily-named database entries. Each entry is equivalent +to the value of the database homeserver config option (see above), with the addition of +a data_stores key. data_stores is an array of strings that specifies the data store(s) +(a defined label for a set of tables) that should be stored on the associated database +backend entry.

+

The currently defined values for data_stores are:

+ +

All databases will end up with additional tables used for tracking database schema migrations +and any pending background updates. Synapse will create these automatically on startup when checking for +and/or performing database schema migrations.

+

To migrate an existing database configuration (e.g. all tables on a single database) to a different +configuration (e.g. the "main" data store on one database, and "state" on another), do the following:

+
    +
  1. +

    Take a backup of your existing database. Things can and do go wrong and database corruption is no joke!

    +
  2. +
  3. +

    Ensure all pending database migrations have been applied and background updates have run. The simplest +way to do this is to use the update_synapse_database script supplied with your Synapse installation.

    +
    update_synapse_database --database-config homeserver.yaml --run-background-updates
    +
    +
  4. +
  5. +

    Copy over the necessary tables and sequences from one database to the other. Tables relating to database +migrations, schemas, schema versions and background updates should not be copied.

    +

    As an example, say that you'd like to split out the "state" data store from an existing database which +currently contains all data stores.

    +

    Simply copy the tables and sequences defined above for the "state" datastore from the existing database +to the secondary database. As noted above, additional tables will be created in the secondary database +when Synapse is started.

    +
  6. +
  7. +

    Modify/create the databases option in your homeserver.yaml to match the desired database configuration.

    +
  8. +
  9. +

    Start Synapse. Check that it starts up successfully and that things generally seem to be working.

    +
  10. +
  11. +

    Drop the old tables that were copied in step 3.

    +
  12. +
+

Only one of the options database or databases may be specified in your config, but not both.

+

Example configuration:

+
databases:
+  basement_box:
+    name: psycopg2
+    txn_limit: 10000
+    data_stores: ["main"]
+    args:
+      user: synapse_user
+      password: secretpassword
+      database: synapse_main
+      host: localhost
+      port: 5432
+      cp_min: 5
+      cp_max: 10
+
+  my_other_database:
+    name: psycopg2
+    txn_limit: 10000
+    data_stores: ["state"]
+    args:
+      user: synapse_user
+      password: secretpassword
+      database: synapse_state
+      host: localhost
+      port: 5432
+      cp_min: 5
+      cp_max: 10
+
+

Logging

Config options related to logging.


@@ -4652,7 +4760,7 @@ Defaults to https://www.recaptcha.net/recaptcha/api/siteverify.

turn_shared_secret: "YOUR_SHARED_SECRET"
 

-

Config options: turn_username and turn_password

+

turn_username and turn_password

The Username and password if the TURN server needs them and does not use a token.

Example configuration:

turn_username: "TURNSERVER_USERNAME"
@@ -4994,15 +5102,19 @@ performance problems on large homeservers.


report_stats

-

Whether or not to report anonymized homeserver usage statistics. This is originally +

Whether or not to report homeserver usage statistics. This is originally set when generating the config. Set this option to true or false to change the current -behavior.

+behavior. See +Reporting Homeserver Usage Statistics +for information on what data is reported.

+

Statistics will be reported 5 minutes after Synapse starts, and then every 3 hours +after that.

Example configuration:

report_stats: true
 

report_stats_endpoint

-

The endpoint to report the anonymized homeserver usage statistics to. +

The endpoint to report homeserver usage statistics to. Defaults to https://matrix.org/report-usage-stats/push

Example configuration:

report_stats_endpoint: https://example.com/report-usage-stats/push
@@ -5647,7 +5759,7 @@ to log in and reauthenticate, whilst preventing new users from setting passwords
 
  • localdb_enabled: Set to false to disable authentication against the local password database. This is ignored if enabled is false, and is only useful if you have other password_providers. Defaults to true.
  • -
  • pepper: Set the value here to a secret random string for extra security. # Uncomment and change to a secret random string for extra security. +
  • pepper: Set the value here to a secret random string for extra security. DO NOT CHANGE THIS AFTER INITIAL SETUP!
  • policy: Define and enforce a password policy, such as minimum lengths for passwords, etc. Each parameter is optional. This is an implementation of MSC2000. Parameters are as follows: @@ -6020,7 +6132,7 @@ can create aliases.

    action: deny

  • -

    Config options: room_list_publication_rules

    +

    room_list_publication_rules

    The room_list_publication_rules option controls who can publish and which rooms can be published in the public room list.

    The format of this option is the same as that for @@ -6212,6 +6324,8 @@ generally required to apply any changes made to this file.

    a real homeserver.yaml. Instead, if you are starting from scratch, please generate a fresh config using Synapse by following the instructions in Installation.

    +

    Documentation for all configuration options can be found in the +Configuration Manual.

    # This file is maintained as an up-to-date snapshot of the default
     # homeserver.yaml configuration generated by Synapse. You can find a
     # complete accounting of possible configuration options at
    @@ -6749,8 +6863,8 @@ maintainer.

    To enable the OpenID integration, you should then add a section to the oidc_providers -setting in your configuration file (or uncomment one of the existing examples). -See sample_config.yaml for some sample settings, as well as +setting in your configuration file. +See the configuration manual for some sample settings, as well as the text below for example configurations for specific providers.

    Sample configs

    Here are a few configs for providers that should work with Synapse.

    @@ -7239,8 +7353,8 @@ file for more details.

    Synapse supports authenticating users via the Central Authentication Service protocol (CAS) natively.

    -

    Please see the cas_config and sso sections of the Synapse configuration -file for more details.

    +

    Please see the cas_config and sso +sections of the configuration manual for more details.

    SSO Mapping Providers

    A mapping provider is a Python class (loaded via a Python module) that works out how to map attributes of a SSO response to Matrix-specific @@ -7650,9 +7764,8 @@ provided by matrix.org so no further action is needed.

    maintainer.

    -

    To enable the JSON web token integration, you should then add a jwt_config section -to your configuration file (or uncomment the enabled: true line in the -existing section). See sample_config.yaml for some +

    To enable the JSON web token integration, you should then add a jwt_config option +to your configuration file. See the configuration manual for some sample settings.

    How to test JWT as a developer

    Although JSON Web Tokens are typically generated from an external server, the @@ -8268,9 +8381,9 @@ though it will always hide it from clients.

    delete the last message in a room. It will, however, hide it from clients.

    Server configuration

    -

    Support for this feature can be enabled and configured in the -retention section of the Synapse configuration file (see the -sample file).

    +

    Support for this feature can be enabled and configured by adding a the +retention in the Synapse configuration file (see +configuration manual).

    To enable support for message retention policies, set the setting enabled in this section to true.

    Default policy

    @@ -8279,8 +8392,8 @@ configuration that is used by Synapse for every room that doesn't have a message retention policy configured in its state. This allows server admins to ensure that messages are never kept indefinitely in a server's database.

    -

    A default policy can be defined as such, in the retention section of -the configuration file:

    +

    A default policy can be defined as such, by adding the retention option in +the configuration file and adding these sub-options:

    default_policy:
       min_lifetime: 1d
       max_lifetime: 1y
    @@ -8294,8 +8407,8 @@ duration (using the units s (seconds), m (minutes), sample configuration file.

    +Synapse will use a default configuration, which is described here in the +configuration manual.

    Some server admins might want a finer control on when events are removed depending on an event's room's policy. This can be done by setting the purge_jobs sub-section in the retention section of the configuration @@ -8336,8 +8449,8 @@ local users between its expiry date and the moment it gets purged from the server's database.

    Lifetime limits

    Server admins can set limits on the values of max_lifetime to use when -purging old events in a room. These limits can be defined as such in the -retention section of the configuration file:

    +purging old events in a room. These limits can be defined under the +retention option in the configuration file:

    allowed_lifetime_min: 1d
     allowed_lifetime_max: 1y
     
    @@ -12119,9 +12232,8 @@ belonging to a user.
  • external_ids - array, optional. Allow setting the identifier of the external identity -provider for SSO (Single sign-on). Details in -Sample Configuration File -section sso and oidc_providers. +provider for SSO (Single sign-on). Details in the configuration manual under the +sections sso and oidc_providers.
    • auth_provider - string. ID of the external identity provider. Value of idp_id in the homeserver configuration. Note that no error is raised if the provided @@ -13183,8 +13295,10 @@ debugging.

      shell access to the server. It should therefore not be enabled in environments where untrusted users have shell access.

      Configuring the manhole

      -

      To enable it, first uncomment the manhole listener configuration in -homeserver.yaml. The configuration is slightly different if you're using docker.

      +

      To enable it, first add the manhole listener configuration in your +homeserver.yaml. You can find information on how to do that +in the configuration manual. +The configuration is slightly different if you're using docker.

      Docker config

      If you are using Docker, set bind_addresses to ['0.0.0.0'] as shown:

      listeners:
      @@ -13481,6 +13595,80 @@ renamed.

      python_twisted_reactor_pending_callsreactor_pending_calls python_twisted_reactor_tick_timereactor_tick_time +

      Reporting Homeserver Usage Statistics

      +

      When generating your Synapse configuration file, you are asked whether you +would like to report usage statistics to Matrix.org. These statistics +provide the foundation a glimpse into the number of Synapse homeservers +participating in the network, as well as statistics such as the number of +rooms being created and messages being sent. This feature is sometimes +affectionately called "phone home" stats. Reporting +is optional +and the reporting endpoint +can be configured, +in case you would like to instead report statistics from a set of homeservers +to your own infrastructure.

      +

      This documentation aims to define the statistics available and the +homeserver configuration options that exist to tweak it.

      +

      Available Statistics

      +

      The following statistics are sent to the configured reporting endpoint:

      + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
      Statistic NameTypeDescription
      homeserverstringThe homeserver's server name.
      memory_rssintThe memory usage of the process (in kilobytes on Unix-based systems, bytes on MacOS).
      cpu_averageintCPU time in % of a single core (not % of all cores).
      server_contextstringAn arbitrary string used to group statistics from a set of homeservers.
      timestampintThe current time, represented as the number of seconds since the epoch.
      uptime_secondsintThe number of seconds since the homeserver was last started.
      python_versionstringThe Python version number in use (e.g "3.7.1"). Taken from sys.version_info.
      total_usersintThe number of registered users on the homeserver.
      total_nonbridged_usersintThe number of users, excluding those created by an Application Service.
      daily_user_type_nativeintThe number of native users created in the last 24 hours.
      daily_user_type_guestintThe number of guest users created in the last 24 hours.
      daily_user_type_bridgedintThe number of users created by Application Services in the last 24 hours.
      total_room_countintThe total number of rooms present on the homeserver.
      daily_active_usersintThe number of unique users1 that have used the homeserver in the last 24 hours.
      monthly_active_usersintThe number of unique users1 that have used the homeserver in the last 30 days.
      daily_active_roomsintThe number of rooms that have had a (state) event with the type m.room.message sent in them in the last 24 hours.
      daily_active_e2ee_roomsintThe number of rooms that have had a (state) event with the type m.room.encrypted sent in them in the last 24 hours.
      daily_messagesintThe number of (state) events with the type m.room.message seen in the last 24 hours.
      daily_e2ee_messagesintThe number of (state) events with the type m.room.encrypted seen in the last 24 hours.
      daily_sent_messagesintThe number of (state) events sent by a local user with the type m.room.message seen in the last 24 hours.
      daily_sent_e2ee_messagesintThe number of (state) events sent by a local user with the type m.room.encrypted seen in the last 24 hours.
      r30_users_allintThe number of 30 day retained users, defined as users who have created their accounts more than 30 days ago, where they were last seen at most 30 days ago and where those two timestamps are over 30 days apart. Includes clients that do not fit into the below r30 client types.
      r30_users_androidintThe number of 30 day retained users, as defined above. Filtered only to clients with "Android" in the user agent string.
      r30_users_iosintThe number of 30 day retained users, as defined above. Filtered only to clients with "iOS" in the user agent string.
      r30_users_electronintThe number of 30 day retained users, as defined above. Filtered only to clients with "Electron" in the user agent string.
      r30_users_webintThe number of 30 day retained users, as defined above. Filtered only to clients with "Mozilla" or "Gecko" in the user agent string.
      r30v2_users_allintThe number of 30 day retained users, with a revised algorithm. Defined as users that appear more than once in the past 60 days, and have more than 30 days between the most and least recent appearances in the past 60 days. Includes clients that do not fit into the below r30 client types.
      r30v2_users_androidintThe number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "android" (case-insensitive) in the user agent string.
      r30v2_users_iosintThe number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "ios" (case-insensitive) in the user agent string.
      r30v2_users_electronintThe number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "electron" (case-insensitive) in the user agent string.
      r30v2_users_webintThe number of 30 day retained users, as defined above. Filtered only to clients with "mozilla" or "gecko" (case-insensitive) in the user agent string.
      cache_factorintThe configured global factor value for caching.
      event_cache_sizeintThe configured event_cache_size value for caching.
      database_enginestringThe database engine that is in use. Either "psycopg2" meaning PostgreSQL is in use, or "sqlite3" for SQLite3.
      database_server_versionstringThe version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system.
      log_levelstringThe log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc.
      +
      1 +

      Native matrix users and guests are always counted. If the +track_puppeted_user_ips +option is set to true, "puppeted" users (users that an Application Service have performed +an action on behalf of) +will also be counted. Note that an Application Service can "puppet" any user in their +user namespace, +not only users that the Application Service has created. If this happens, the Application Service +will additionally be counted as a user (irrespective of track_puppeted_user_ips).

      +
      +

      Using a Custom Statistics Collection Server

      +

      If statistics reporting is enabled, the endpoint that Synapse sends metrics to is configured by the +report_stats_endpoint config +option. By default, statistics are sent to Matrix.org.

      +

      If you would like to set up your own statistics collection server and send metrics there, you may +consider using one of the following known implementations:

      +

      Understanding Synapse through Grafana graphs

      It is possible to monitor much of the internal state of Synapse using Prometheus metrics and Grafana. @@ -14024,6 +14212,9 @@ Here is how to run your local Synapse checkout against your local Complement che

    • Passing POSTGRES=1 as an environment variable to use the Postgres database instead.
    • Passing WORKERS=1 as an environment variable to use a workerised setup instead. This option implies the use of Postgres.
    +

    To increase the log level for the tests, set SYNAPSE_TEST_LOG_LEVEL, e.g:

    +
    SYNAPSE_TEST_LOG_LEVEL=DEBUG COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages
    +

    Prettier formatting with gotestfmt

    If you want to format the output of the tests the same way as it looks in CI, install gotestfmt.

    @@ -14050,7 +14241,7 @@ install gotestfmt.<
  • git push your commit to your fork of Synapse;
  • on GitHub, create the Pull Request;
  • add a changelog entry and push it to your Pull Request;
  • -
  • for most contributors, that's all - however, if you are a member of the organization matrix-org, on GitHub, please request a review from matrix.org / Synapse Core.
  • +
  • that's it for now, a non-draft pull request will automatically request review from the team;
  • if you need to update your PR, please avoid rebasing and just add new commits to your branch.
  • Changelog

    @@ -14199,7 +14390,11 @@ be required.

  • If there is any error, fix the error.
  • -
  • If a developer has requested changes, make these changes and let us know if it is ready for a developer to review again.
  • +
  • If a developer has requested changes, make these changes and let us know if it is ready for a developer to review again. +
      +
    • A pull request is a conversation, if you disagree with the suggestions, please respond and discuss it.
    • +
    +
  • Create a new commit with the changes.
    • Please do NOT overwrite the history. New commits make the reviewer's life easier.
    • @@ -14207,6 +14402,8 @@ be required.

  • Back to 1.
  • +
  • Once the pull request is ready for review again please re-request review from whichever developer did your initial +review (or leave a comment in the pull request that you believe all required changes have been done).
  • Once both the CI and the developers are happy, the patch will be merged into Synapse and released shortly!

    11. Find a new issue.

    @@ -14286,99 +14483,97 @@ relative imports (from .types import UserID).

    -

    Configuration file format

    -

    The sample configuration file acts as a +

    Configuration code and documentation format

    +

    When adding a configuration option to the code, if several settings are grouped into a single dict, ensure that your code +correctly handles the top-level option being set to None (as it will be if no sub-options are enabled).

    +

    The configuration manual acts as a reference to Synapse's configuration options for server administrators. Remember that many readers will be unfamiliar with YAML and server -administration in general, so that it is important that the file be as -easy to understand as possible, which includes following a consistent -format.

    +administration in general, so it is important that when you add +a configuration option the documentation be as easy to understand as possible, which +includes following a consistent format.

    Some guidelines follow:

    • -

      Sections should be separated with a heading consisting of a single -line prefixed and suffixed with ##. There should be two blank -lines before the section header, and one after.

      -
    • -
    • -

      Each option should be listed in the file with the following format:

      +

      Each option should be listed in the config manual with the following format:

      • -

        A comment describing the setting. Each line of this comment -should be prefixed with a hash (#) and a space.

        -

        The comment should describe the default behaviour (ie, what -happens if the setting is omitted), as well as what the effect -will be if the setting is changed.

        -

        Often, the comment end with something like "uncomment the -following to ".

        +

        The name of the option, prefixed by ###.

      • -

        A line consisting of only #.

        +

        A comment which describes the default behaviour (i.e. what +happens if the setting is omitted), as well as what the effect +will be if the setting is changed.

      • -

        A commented-out example setting, prefixed with only #.

        +

        An example setting, using backticks to define the code block

        For boolean (on/off) options, convention is that this example -should be the opposite to the default (so the comment will end -with "Uncomment the following to enable [or disable] -." For other options, the example should give some -non-default value which is likely to be useful to the reader.

        +should be the opposite to the default. For other options, the example should give +some non-default value which is likely to be useful to the reader.

    • -

      There should be a blank line between each option.

      -
    • -
    • -

      Where several settings are grouped into a single dict, avoid the -convention where the whole block is commented out, resulting in -comment lines starting # #, as this is hard to read and confusing -to edit. Instead, leave the top-level config option uncommented, and -follow the conventions above for sub-options. Ensure that your code -correctly handles the top-level option being set to None (as it -will be if no sub-options are enabled).

      -
    • -
    • -

      Lines should be wrapped at 80 characters.

      -
    • -
    • -

      Use two-space indents.

      +

      There should be a horizontal rule between each option, which can be achieved by adding --- before and +after the option.

    • true and false are spelt thus (as opposed to True, etc.)

    • -
    • -

      Use single quotes (') rather than double-quotes (") or backticks -(`) to refer to configuration options.

      -

    Example:

    -
    ## Frobnication ##
    -
    -# The frobnicator will ensure that all requests are fully frobnicated.
    -# To enable it, uncomment the following.
    -#
    -#frobnicator_enabled: true
    -
    -# By default, the frobnicator will frobnicate with the default frobber.
    -# The following will make it use an alternative frobber.
    -#
    -#frobincator_frobber: special_frobber
    -
    -# Settings for the frobber
    -#
    -frobber:
    -  # frobbing speed. Defaults to 1.
    -  #
    -  #speed: 10
    -
    -  # frobbing distance. Defaults to 1000.
    -  #
    -  #distance: 100
    +
    +

    modules

    +

    Use the module sub-option to add a module under modules to extend functionality. +The module setting then has a sub-option, config, which can be used to define some configuration +for the module.

    +

    Defaults to none.

    +

    Example configuration:

    +
    modules:
    +  - module: my_super_module.MySuperClass
    +    config:
    +      do_thing: true
    +  - module: my_other_super_module.SomeClass
    +    config: {}
     
    +

    Note that the sample configuration is generated from the synapse code and is maintained by a script, scripts-dev/generate_sample_config.sh. Making sure that the output from this script matches the desired format is left as an exercise for the reader!

    +

    Some notes on how we do reviews

    +

    The Synapse team works off a shared review queue -- any new pull requests for +Synapse (or related projects) has a review requested from the entire team. Team +members should process this queue using the following rules:

    +
      +
    • Any high urgency pull requests (e.g. fixes for broken continuous integration +or fixes for release blockers);
    • +
    • Follow-up reviews for pull requests which have previously received reviews;
    • +
    • Any remaining pull requests.
    • +
    +

    For the latter two categories above, older pull requests should be prioritised.

    +

    It is explicit that there is no priority given to pull requests from the team +(vs from the community). If a pull request requires a quick turn around, please +explicitly communicate this via #synapse-dev:matrix.org +or as a comment on the pull request.

    +

    Once an initial review has been completed and the author has made additional changes, +follow-up reviews should go back to the same reviewer. This helps build a shared +context and conversation between author and reviewer.

    +

    As a team we aim to keep the number of inflight pull requests to a minimum to ensure +that ongoing work is finished before starting new work.

    +

    Performing a review

    +

    To communicate to the rest of the team the status of each pull request, team +members should do the following:

    +
      +
    • Assign themselves to the pull request (they should be left assigned to the +pull request until it is merged, closed, or are no longer the reviewer);
    • +
    • Review the pull request by leaving comments, questions, and suggestions;
    • +
    • Mark the pull request appropriately (as needing changes or accepted).
    • +
    +

    If you are unsure about a particular part of the pull request (or are not confident +in your understanding of part of the code) then ask questions or request review +from the team again. When requesting review from the team be sure to leave a comment +with the rationale on why you're putting it back in the queue.

    Synapse Release Cycle

    Releases of Synapse follow a two week release cycle with new releases usually occurring on Tuesdays:

    -- cgit 1.5.1