From 6d4ab8fb90ae2eaa3393dcbd37c83c486202b347 Mon Sep 17 00:00:00 2001 From: DMRobertson Date: Tue, 27 Sep 2022 12:50:14 +0000 Subject: deploy: 3853011d01ad3f5034f53a9dfb7a06e36cf70ae9 --- latest/print.html | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 239 insertions(+), 16 deletions(-) (limited to 'latest/print.html') diff --git a/latest/print.html b/latest/print.html index b0a7e87299..26cbe1e8d2 100644 --- a/latest/print.html +++ b/latest/print.html @@ -360,6 +360,9 @@ and mounting it to /var/synapse should be taken into consideration.
  • Python 3.7 or later, up to Python 3.10.
  • At least 1GB of free RAM if you want to join large public rooms like #matrix:matrix.org
    • +

    If building on an uncommon architecture for which pre-built wheels are +unavailable, you will need to have a recent Rust compiler installed. The easiest +way of installing the latest version is to use rustup.

    To install the Synapse homeserver run:

    mkdir -p ~/synapse
 virtualenv -p python3 ~/synapse/env
@@ -430,8 +433,9 @@ sudo dnf groupinstall "Development Tools"
 
    You may need to install the latest Xcode developer tools:
    
 
    xcode-select --install
 
    
-
    On ARM-based Macs you may need to explicitly install libjpeg which is a pillow dependency. You can use Homebrew (https://brew.sh):
    
-
     brew install jpeg
+
    On ARM-based Macs you may need to install libjpeg and libpq. 
+You can use Homebrew (https://brew.sh):
    
+
     brew install jpeg libpq
 
    
 
    On macOS Catalina (10.15) you may need to explicitly install OpenSSL
 via brew and inform pip about it so that psycopg2 builds:
    
@@ -886,6 +890,10 @@ consider setting bind_addresses: ['127.0.0.1'] so that the server o
 listens to traffic on localhost. (Do not change bind_addresses to 127.0.0.1 
 when using a containerized Synapse, as that will prevent it from responding
 to proxied traffic.)
    
+
    Optionally, you can also set
+request_id_header
+so that the server extracts and re-uses the same request ID format that the
+reverse proxy is using.
    
 
    Reverse-proxy configuration examples
    
 
    NOTE: You only need one of these.
    
 
    nginx
    
@@ -1547,9 +1555,8 @@ which are no longer maintained.
    
 summaries.
    
 
 
  • 
-
    If Synapse was installed using prebuilt
-packages, you will need to follow the
-normal process for upgrading those packages.
    
+
    If Synapse was installed using prebuilt packages,
+you will need to follow the normal process for upgrading those packages.
    
 
    • 
 
  • 
 
    If Synapse was installed using pip then upgrade to the latest
@@ -1617,6 +1624,17 @@ dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
    • 
 
 
+
    Upgrading to v1.68.0
    
+
    As announced in the upgrade notes for v1.67.0, Synapse now requires a SQLite
+version of 3.27.0 or higher if SQLite is in use and source checkouts of Synapse
+now require a recent Rust compiler.
    
+
    Installations using 
    
+
      
+
    • Docker images from matrixdotorg,
      • 
+
    • Debian packages from Matrix.org, or
      • 
+
    • PyPI wheels via pip install matrix-synapse (on supported platforms and architectures)
      • 
+
    
+
    will not be affected.
    
 
    Upgrading to v1.67.0
    
 
    Direct TCP replication is no longer supported: migrate to Redis
    
 
    Redis support was added in v1.13.0 with it becoming the recommended method in
@@ -1642,7 +1660,7 @@ will require a recent Rust compiler. Those using packages or
 
    From the next major release (v1.68.0) Synapse will require SQLite 3.27.0 or 
 higher. Synapse v1.67.0 will be the last major release supporting SQLite
 versions 3.22 to 3.26.
    
-
    Those using docker images or Debian packages from Matrix.org will not be
+
    Those using Docker images or Debian packages from Matrix.org will not be
 affected. If you have installed from source, you should check the version of 
 SQLite used by Python with:
    
 
    python -c "import sqlite3; print(sqlite3.sqlite_version)"
@@ -3539,7 +3557,16 @@ configuration.
    
 
 
  • 
 
    x_forwarded: Only valid for an 'http' listener. Set to true to use the X-Forwarded-For header as the client IP. Useful when Synapse is
-behind a reverse-proxy.
    
+behind a reverse-proxy.
    
+
    • 
+
  • 
+
    request_id_header: The header extracted from each incoming request that is
+used as the basis for the request ID. The request ID is used in
+logs and tracing to
+correlate and match up requests. When unset, Synapse will automatically
+generate sequential request IDs. This option is useful when Synapse is behind
+a reverse-proxy.
    
+
    Added in Synapse 1.68.0.
    
 
    • 
 
  • 
 
    resources: Only valid for an 'http' listener. A list of resources to host
@@ -4059,8 +4086,9 @@ on this homeserver.
    
 
    Options related to caching.
    
 
    
 
    event_cache_size
    
-
    The number of events to cache in memory. Not affected by
-caches.global_factor and is not part of the caches section. Defaults to 10K.
    
+
    The number of events to cache in memory. Defaults to 10K. Like other caches,
+this is affected by caches.global_factor (see below).
    
+
    Note that this option is not part of the caches section.
    
 
    Example configuration:
    
 
    event_cache_size: 15K
 
    
@@ -4365,7 +4393,7 @@ client is attempting to log into. Defaults to per_second: 0.17,
 burst_count: 3.
    
 
    • 
 
  • 
-
    failted_attempts ratelimits login requests based on the account the
+
    failed_attempts ratelimits login requests based on the account the
 client is attempting to log into, based on the amount of failed login
 attempts for this account. Defaults to per_second: 0.17, burst_count: 3.
    
 
    • 
@@ -11583,6 +11611,133 @@ If the room does not define a type, the value will be null.
   ]
 }
 
    
+
    Room Messages API
    
+
    The Room Messages admin API allows server admins to get all messages
+sent to a room in a given timeframe. There are various parameters available
+that allow for filtering and ordering the returned list. This API supports pagination.
    
+
    To use it, you will need to authenticate by providing an access_token
+for a server admin: see Admin API.
    
+
    This endpoint mirrors the Matrix Spec defined Messages API.
    
+
    The API is:
    
+
    GET /_synapse/admin/v1/rooms/<room_id>/messages
+
    
+
    Parameters
    
+
    The following path parameters are required:
    
+
      
+
    • room_id - The ID of the room you wish you fetch messages from.
      • 
+
    
+
    The following query parameters are available:
    
+
      
+
    • from (required) - The token to start returning events from. This token can be obtained from a prev_batch
+or next_batch token returned by the /sync endpoint, or from an end token returned by a previous request to this endpoint.
      • 
+
    • to - The token to spot returning events at.
      • 
+
    • limit - The maximum number of events to return. Defaults to 10.
      • 
+
    • filter - A JSON RoomEventFilter to filter returned events with.
      • 
+
    • dir - The direction to return events from. Either f for forwards or b for backwards. Setting
+this value to b will reverse the above sort order. Defaults to f.
      • 
+
    
+
    Response
    
+
    The following fields are possible in the JSON response body:
    
+
      
+
    • chunk - A list of room events. The order depends on the dir parameter.
+Note that an empty chunk does not necessarily imply that no more events are available. Clients should continue to paginate until no end property is returned.
      • 
+
    • end - A token corresponding to the end of chunk. This token can be passed back to this endpoint to request further events.
+If no further events are available, this property is omitted from the response.
      • 
+
    • start - A token corresponding to the start of chunk.
      • 
+
    • state - A list of state events relevant to showing the chunk.
      • 
+
    
+
    Example
    
+
    For more details on each chunk, read the Matrix specification.
    
+
    {
+  "chunk": [
+    {
+      "content": {
+        "body": "This is an example text message",
+        "format": "org.matrix.custom.html",
+        "formatted_body": "<b>This is an example text message</b>",
+        "msgtype": "m.text"
+      },
+      "event_id": "$143273582443PhrSn:example.org",
+      "origin_server_ts": 1432735824653,
+      "room_id": "!636q39766251:example.com",
+      "sender": "@example:example.org",
+      "type": "m.room.message",
+      "unsigned": {
+        "age": 1234
+      }
+    },
+    {
+      "content": {
+        "name": "The room name"
+      },
+      "event_id": "$143273582443PhrSn:example.org",
+      "origin_server_ts": 1432735824653,
+      "room_id": "!636q39766251:example.com",
+      "sender": "@example:example.org",
+      "state_key": "",
+      "type": "m.room.name",
+      "unsigned": {
+        "age": 1234
+      }
+    },
+    {
+      "content": {
+        "body": "Gangnam Style",
+        "info": {
+          "duration": 2140786,
+          "h": 320,
+          "mimetype": "video/mp4",
+          "size": 1563685,
+          "thumbnail_info": {
+            "h": 300,
+            "mimetype": "image/jpeg",
+            "size": 46144,
+            "w": 300
+          },
+          "thumbnail_url": "mxc://example.org/FHyPlCeYUSFFxlgbQYZmoEoe",
+          "w": 480
+        },
+        "msgtype": "m.video",
+        "url": "mxc://example.org/a526eYUSFFxlgbQYZmo442"
+      },
+      "event_id": "$143273582443PhrSn:example.org",
+      "origin_server_ts": 1432735824653,
+      "room_id": "!636q39766251:example.com",
+      "sender": "@example:example.org",
+      "type": "m.room.message",
+      "unsigned": {
+        "age": 1234
+      }
+    }
+  ],
+  "end": "t47409-4357353_219380_26003_2265",
+  "start": "t47429-4392820_219380_26003_2265"
+}
+
    
+
    Room Timestamp to Event API
    
+
    The Room Timestamp to Event API endpoint fetches the event_id of the closest event to the given
+timestamp (ts query parameter) in the given direction (dir query parameter).
    
+
    Useful for cases like jump to date so you can start paginating messages from
+a given date in the archive.
    
+
    The API is:
    
+
      GET /_synapse/admin/v1/rooms/<room_id>/timestamp_to_event
+
    
+
    Parameters
    
+
    The following path parameters are required:
    
+
      
+
    • room_id - The ID of the room you wish to check.
      • 
+
    
+
    The following query parameters are available:
    
+
      
+
    • ts - a timestamp in milliseconds where we will find the closest event in
+the given direction.
      • 
+
    • dir - can be f or b to indicate forwards and backwards in time from the
+given timestamp. Defaults to f.
      • 
+
    
+
    Response
    
+
      
+
    • event_id - converted from timestamp
      • 
+
    
 
    Block Room API
    
 
    The Block Room admin API allows server admins to block and unblock rooms,
 and query to see if a given room is blocked.
@@ -12193,6 +12348,7 @@ for a server admin: see Ad
     "appservice_id": null,
     "consent_server_notice_sent": null,
     "consent_version": null,
+    "consent_ts": null,
     "external_ids": [
         {
             "auth_provider": "<provider1>",
@@ -12500,6 +12656,7 @@ is set to true:
    
 
  • Remove the user's creation (registration) timestamp
    • 
 
  • Remove rate limit overrides
    • 
 
  • Remove from monthly active users
    • 
+
  • Remove user's consent information (consent version and timestamp)
    • 
 
 
    Reset password
    
 
    Changes the password of another user. This will automatically log the user out of all their devices.
    
@@ -13148,6 +13305,30 @@ for more information.

    The request and response format is the same as the /_matrix/client/r0/register/available API.

    +

    Find a user based on their ID in an auth provider

    +

    The API is:

    +
    GET /_synapse/admin/v1/auth_providers/$provider/users/$external_id
+
    +

    When a user matched the given ID for the given provider, an HTTP code 200 with a response body like the following is returned:

    +
    {
+    "user_id": "@hello:example.org"
+}
+
    +

    Parameters

    +

    The following parameters should be set in the URL:

    + +

    The external_id may have characters that are not URL-safe (typically /, : or @), so it is advised to URL-encode those parameters.

    +

    Errors

    +

    Returns a 404 HTTP status code if no user was found, with a response body like this:

    +
    {
+    "errcode":"M_NOT_FOUND",
+    "error":"User not found"
+}
+
    +

    Added in Synapse 1.68.0.

    Version API

    This API returns the running Synapse version and the Python version on which Synapse is being run. This is useful when a Synapse instance @@ -14037,14 +14218,14 @@ WHERE room_stats_state.room_id = event_json.room_id" | psql -d synapse -h l -AAAAAAAAAAAAAAAAAAAAA- -BBBBBBBBBBBBBBBBBBBBBB- -C- -DD- -EEEEEE- -FFFFFFFFF- -GG- -HHHHHHHHHHHHHHHHHHHHHHH- -IIIIII- -JJJJJJJ- -KKKKKK-, -LLLLLL- -MMMMMMM- -NNNNNN- O -P- -QQ- -RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- -SSSSSSSSSSSS- -TTTTTT- - + - + @@ -14208,6 +14389,8 @@ on Windows is not officially supported.

    Synapse can connect to PostgreSQL via the psycopg2 Python library. Building this library from source requires access to PostgreSQL's C header files. On Debian or Ubuntu Linux, these can be installed with sudo apt install libpq-dev.

    The source code of Synapse is hosted on GitHub. You will also need a recent version of git.

    For some tests, you will need a recent version of Docker.

    +

    A recent version of the Rust compiler is needed to build the native modules. The +easiest way of installing the latest version is to use rustup.

    3. Get the source.

    The preferred and easiest way to contribute changes is to fork the relevant project on GitHub, and then create a pull request to ask us to pull your @@ -14264,6 +14447,10 @@ to check that your contributions render correctly. The docs are written in

    Some documentation also exists in Synapse's GitHub Wiki, although this is primarily contributed to by community authors.

    +

    When changes are made to any Rust code then you must call either poetry install +or maturin develop (if installed) to rebuild the Rust code. Using maturin +is quicker than poetry install, so is recommended when making frequent +changes to the Rust code.

    8. Test, test, test!

    While you're developing and before submitting a patch, you'll @@ -14312,7 +14499,7 @@ working directory. Typically, this ends up being _trial_temp/test.dbThe database file can then be inspected with:

    sqlite3 _trial_temp/test.db
    -

    Note that the database file is cleared at the beginning of each test run. Thus it +

    Note that the database file is cleared at the beginning of each test run. Thus it will always only contain the data generated by the last run test. Though generally when debugging, one is only running a single test anyway.

    Running tests under PostgreSQL

    @@ -15319,6 +15506,20 @@ context of poetry's venv, without having to run poetry shell before

    ...reset my venv to the locked environment?

    poetry install --extras all --remove-untracked
    +

    ...delete everything and start over from scratch?

    +
    # Stop the current virtualenv if active
+$ deactivate
+
+# Remove all of the files from the current environment.
+# Don't worry, even though it says "all", this will only
+# remove the Poetry virtualenvs for the current project.
+$ poetry env remove --all
+
+# Reactivate Poetry shell to create the virtualenv again
+$ poetry shell
+# Install everything again
+$ poetry install --extras all
+

    ...run a command in the poetry virtualenv?

    Use poetry run cmd args when you need the python virtualenv context. To avoid typing poetry run all the time, you can run poetry shell @@ -15407,6 +15608,11 @@ installed poetry with pipx, try pipx runpip poet

    Poetry caches a bunch of information about packages that isn't readily available from PyPI. (This is what makes poetry seem slow when doing the first poetry install.) Try poetry cache list and poetry cache clear --all <name of cache> to see if that fixes things.

    +

    Remove outdated egg-info

    +

    Delete the matrix_synapse.egg-info/ directory from the root of your Synapse +install.

    +

    This stores some cached information about dependencies and often conflicts with +letting Poetry do the right thing.

    Try --verbose or --dry-run arguments.

    Sometimes useful to see what poetry's internal logic is.

    Cancellation

    @@ -16674,9 +16880,9 @@ table. Each subject can have only one.

    Stats correspond to the present values. Current rows contain the most up-to-date statistics for a room. Each subject can only have one entry.

    Deprecation Policy for Platform Dependencies

    -

    Synapse has a number of platform dependencies, including Python and PostgreSQL. -This document outlines the policy towards which versions we support, and when we -drop support for versions in the future.

    +

    Synapse has a number of platform dependencies, including Python, Rust, +PostgreSQL and SQLite. This document outlines the policy towards which versions +we support, and when we drop support for versions in the future.

    Policy

    Synapse follows the upstream support life cycles for Python and PostgreSQL, i.e. when a version reaches End of Life Synapse will withdraw support for that @@ -16684,6 +16890,13 @@ version in future releases.

    Details on the upstream support life cycles for Python and PostgreSQL are documented at https://endoflife.date/python and https://endoflife.date/postgresql.

    +

    A Rust compiler is required to build Synapse from source. For any given release +the minimum required version may be bumped up to a recent Rust version, and so +people building from source should ensure they can fetch recent versions of Rust +(e.g. by using rustup).

    +

    The oldest supported version of SQLite is the version +provided by +Debian oldstable.

    Context

    It is important for system admins to have a clear understanding of the platform requirements of Synapse and its deprecation policies so that they can @@ -16694,6 +16907,16 @@ long process.

    By following the upstream support life cycles Synapse can ensure that its dependencies continue to get security patches, while not requiring system admins to constantly update their platform dependencies to the latest versions.

    +

    For Rust, the situation is a bit different given that a) the Rust foundation +does not generally support older Rust versions, and b) the library ecosystem +generally bump their minimum support Rust versions frequently. In general, the +Synapse team will try to avoid updating the dependency on Rust to the absolute +latest version, but introducing a formal policy is hard given the constraints of +the ecosystem.

    +

    On a similar note, SQLite does not generally have a concept of "supported +release"; bugfixes are published for the latest minor release only. We chose to +track Debian's oldstable as this is relatively conservative, predictably updated +and is consistent with the .deb packages released by Matrix.org.

    Summary of performance impact of running on resource constrained devices such as SBCs

    I've been running my homeserver on a cubietruck at home now for some time and am often replying to statements like "you need loads of ram to join large rooms" with "it works fine for me". I thought it might be useful to curate a summary of the issues you're likely to run into to help as a scaling-down guide, maybe highlight these for development work or end up as documentation. It seems that once you get up to about 4x1.5GHz arm64 4GiB these issues are no longer a problem.

      -- cgit 1.5.1
    PartExplanation
    AAAATimestamp request was logged (not recieved)
    AAAATimestamp request was logged (not received)
    BBBBLogger name (synapse.access.(http\|https).<tag>, where 'tag' is defined in the listeners config section, normally the port)
    CCCCLine number in code
    DDDDLog Level
    EEEERequest Identifier (This identifier is shared by related log lines)
    FFFFSource IP (Or X-Forwarded-For if enabled)
    GGGGServer Port
    HHHHFederated Server or Local User making request (blank if unauthenticated or not supplied)
    HHHHFederated Server or Local User making request (blank if unauthenticated or not supplied).
    If this is of the form `@aaa:example.com
    IIIITotal Time to process the request
    JJJJTime to send response over network once generated (this may be negative if the socket is closed before the response is generated)
    KKKKUserland CPU time