From e6245e6d48bcb0a1d426b73d010988e0f2d92b35 Mon Sep 17 00:00:00 2001 From: Aaron Raimist Date: Thu, 10 Jun 2021 05:40:24 -0500 Subject: Mention that you need to configure max upload size in reverse proxy as well (#10122) Signed-off-by: Aaron Raimist --- docs/sample_config.yaml | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'docs') diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index 7b97f73a29..f8925a5e24 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -954,6 +954,10 @@ media_store_path: "DATADIR/media_store" # The largest allowed upload size in bytes # +# If you are using a reverse proxy you may also need to set this value in +# your reverse proxy's config. Notably Nginx has a small max body size by default. +# See https://matrix-org.github.io/synapse/develop/reverse_proxy.html. +# #max_upload_size: 50M # Maximum number of pixels that will be thumbnailed -- cgit 1.5.1 From c1b9922498dea4b2882d26a4eaef3e0a37e727fd Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 11 Jun 2021 14:45:53 +0100 Subject: Support for database schema version ranges (#9933) This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different. --- changelog.d/9933.misc | 1 + docs/SUMMARY.md | 3 +- docs/development/database_schema.md | 95 ++++++++++++++++++ synapse/storage/prepare_database.py | 121 +++++++++++++++-------- synapse/storage/schema/README.md | 37 +------ synapse/storage/schema/__init__.py | 19 +++- synapse/storage/schema/common/schema_version.sql | 7 ++ 7 files changed, 206 insertions(+), 77 deletions(-) create mode 100644 changelog.d/9933.misc create mode 100644 docs/development/database_schema.md (limited to 'docs') diff --git a/changelog.d/9933.misc b/changelog.d/9933.misc new file mode 100644 index 0000000000..0860026670 --- /dev/null +++ b/changelog.d/9933.misc @@ -0,0 +1 @@ +Update the database schema versioning to support gradual migration away from legacy tables. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 8f39ae0270..af2c968c9a 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -69,6 +69,7 @@ - [Git Usage](dev/git.md) - [Testing]() - [OpenTracing](opentracing.md) + - [Database Schemas](development/database_schema.md) - [Synapse Architecture]() - [Log Contexts](log_contexts.md) - [Replication](replication.md) @@ -84,4 +85,4 @@ - [Scripts]() # Other - - [Dependency Deprecation Policy](deprecation_policy.md) \ No newline at end of file + - [Dependency Deprecation Policy](deprecation_policy.md) diff --git a/docs/development/database_schema.md b/docs/development/database_schema.md new file mode 100644 index 0000000000..7fe8ec63e1 --- /dev/null +++ b/docs/development/database_schema.md @@ -0,0 +1,95 @@ +# Synapse database schema files + +Synapse's database schema is stored in the `synapse.storage.schema` module. + +## Logical databases + +Synapse supports splitting its datastore across multiple physical databases (which can +be useful for large installations), and the schema files are therefore split according +to the logical database they apply to. + +At the time of writing, the following "logical" databases are supported: + +* `state` - used to store Matrix room state (more specifically, `state_groups`, + their relationships and contents). +* `main` - stores everything else. + +Additionally, the `common` directory contains schema files for tables which must be +present on *all* physical databases. + +## Synapse schema versions + +Synapse manages its database schema via "schema versions". These are mainly used to +help avoid confusion if the Synapse codebase is rolled back after the database is +updated. They work as follows: + + * The Synapse codebase defines a constant `synapse.storage.schema.SCHEMA_VERSION` + which represents the expectations made about the database by that version. For + example, as of Synapse v1.36, this is `59`. + + * The database stores a "compatibility version" in + `schema_compat_version.compat_version` which defines the `SCHEMA_VERSION` of the + oldest version of Synapse which will work with the database. On startup, if + `compat_version` is found to be newer than `SCHEMA_VERSION`, Synapse will refuse to + start. + + Synapse automatically updates this field from + `synapse.storage.schema.SCHEMA_COMPAT_VERSION`. + + * Whenever a backwards-incompatible change is made to the database format (normally + via a `delta` file), `synapse.storage.schema.SCHEMA_COMPAT_VERSION` is also updated + so that administrators can not accidentally roll back to a too-old version of Synapse. + +Generally, the goal is to maintain compatibility with at least one or two previous +releases of Synapse, so any substantial change tends to require multiple releases and a +bit of forward-planning to get right. + +As a worked example: we want to remove the `room_stats_historical` table. Here is how it +might pan out. + + 1. Replace any code that *reads* from `room_stats_historical` with alternative + implementations, but keep writing to it in case of rollback to an earlier version. + Also, increase `synapse.storage.schema.SCHEMA_VERSION`. In this + instance, there is no existing code which reads from `room_stats_historical`, so + our starting point is: + + v1.36.0: `SCHEMA_VERSION=59`, `SCHEMA_COMPAT_VERSION=59` + + 2. Next (say in Synapse v1.37.0): remove the code that *writes* to + `room_stats_historical`, but don’t yet remove the table in case of rollback to + v1.36.0. Again, we increase `synapse.storage.schema.SCHEMA_VERSION`, but + because we have not broken compatibility with v1.36, we do not yet update + `SCHEMA_COMPAT_VERSION`. We now have: + + v1.37.0: `SCHEMA_VERSION=60`, `SCHEMA_COMPAT_VERSION=59`. + + 3. Later (say in Synapse v1.38.0): we can remove the table altogether. This will + break compatibility with v1.36.0, so we must update `SCHEMA_COMPAT_VERSION` accordingly. + There is no need to update `synapse.storage.schema.SCHEMA_VERSION`, since there is no + change to the Synapse codebase here. So we end up with: + + v1.38.0: `SCHEMA_VERSION=60`, `SCHEMA_COMPAT_VERSION=60`. + +If in doubt about whether to update `SCHEMA_VERSION` or not, it is generally best to +lean towards doing so. + +## Full schema dumps + +In the `full_schemas` directories, only the most recently-numbered snapshot is used +(`54` at the time of writing). Older snapshots (eg, `16`) are present for historical +reference only. + +### Building full schema dumps + +If you want to recreate these schemas, they need to be made from a database that +has had all background updates run. + +To do so, use `scripts-dev/make_full_schema.sh`. This will produce new +`full.sql.postgres` and `full.sql.sqlite` files. + +Ensure postgres is installed, then run: + + ./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/ + +NB at the time of writing, this script predates the split into separate `state`/`main` +databases so will require updates to handle that correctly. diff --git a/synapse/storage/prepare_database.py b/synapse/storage/prepare_database.py index 3799d46734..683e5e3b90 100644 --- a/synapse/storage/prepare_database.py +++ b/synapse/storage/prepare_database.py @@ -1,5 +1,4 @@ -# Copyright 2014 - 2016 OpenMarket Ltd -# Copyright 2018 New Vector Ltd +# Copyright 2014 - 2021 The Matrix.org Foundation C.I.C. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -26,7 +25,7 @@ from synapse.config.homeserver import HomeServerConfig from synapse.storage.database import LoggingDatabaseConnection from synapse.storage.engines import BaseDatabaseEngine from synapse.storage.engines.postgres import PostgresEngine -from synapse.storage.schema import SCHEMA_VERSION +from synapse.storage.schema import SCHEMA_COMPAT_VERSION, SCHEMA_VERSION from synapse.storage.types import Cursor logger = logging.getLogger(__name__) @@ -59,6 +58,28 @@ UNAPPLIED_DELTA_ON_WORKER_ERROR = ( ) +@attr.s +class _SchemaState: + current_version: int = attr.ib() + """The current schema version of the database""" + + compat_version: Optional[int] = attr.ib() + """The SCHEMA_VERSION of the oldest version of Synapse for this database + + If this is None, we have an old version of the database without the necessary + table. + """ + + applied_deltas: Collection[str] = attr.ib(factory=tuple) + """Any delta files for `current_version` which have already been applied""" + + upgraded: bool = attr.ib(default=False) + """Whether the current state was reached by applying deltas. + + If False, we have run the full schema for `current_version`, and have applied no + deltas since. If True, we have run some deltas since the original creation.""" + + def prepare_database( db_conn: LoggingDatabaseConnection, database_engine: BaseDatabaseEngine, @@ -96,12 +117,11 @@ def prepare_database( version_info = _get_or_create_schema_state(cur, database_engine) if version_info: - user_version, delta_files, upgraded = version_info logger.info( "%r: Existing schema is %i (+%i deltas)", databases, - user_version, - len(delta_files), + version_info.current_version, + len(version_info.applied_deltas), ) # config should only be None when we are preparing an in-memory SQLite db, @@ -113,16 +133,18 @@ def prepare_database( # if it's a worker app, refuse to upgrade the database, to avoid multiple # workers doing it at once. - if config.worker_app is not None and user_version != SCHEMA_VERSION: + if ( + config.worker_app is not None + and version_info.current_version != SCHEMA_VERSION + ): raise UpgradeDatabaseException( - OUTDATED_SCHEMA_ON_WORKER_ERROR % (SCHEMA_VERSION, user_version) + OUTDATED_SCHEMA_ON_WORKER_ERROR + % (SCHEMA_VERSION, version_info.current_version) ) _upgrade_existing_database( cur, - user_version, - delta_files, - upgraded, + version_info, database_engine, config, databases=databases, @@ -261,9 +283,7 @@ def _setup_new_database( _upgrade_existing_database( cur, - current_version=max_current_ver, - applied_delta_files=[], - upgraded=False, + _SchemaState(current_version=max_current_ver, compat_version=None), database_engine=database_engine, config=None, databases=databases, @@ -273,9 +293,7 @@ def _setup_new_database( def _upgrade_existing_database( cur: Cursor, - current_version: int, - applied_delta_files: List[str], - upgraded: bool, + current_schema_state: _SchemaState, database_engine: BaseDatabaseEngine, config: Optional[HomeServerConfig], databases: Collection[str], @@ -321,12 +339,8 @@ def _upgrade_existing_database( Args: cur - current_version: The current version of the schema. - applied_delta_files: A list of deltas that have already been applied. - upgraded: Whether the current version was generated by having - applied deltas or from full schema file. If `True` the function - will never apply delta files for the given `current_version`, since - the current_version wasn't generated by applying those delta files. + current_schema_state: The current version of the schema, as + returned by _get_or_create_schema_state database_engine config: None if we are initialising a blank database, otherwise the application @@ -337,13 +351,16 @@ def _upgrade_existing_database( upgrade portions of the delta scripts. """ if is_empty: - assert not applied_delta_files + assert not current_schema_state.applied_deltas else: assert config is_worker = config and config.worker_app is not None - if current_version > SCHEMA_VERSION: + if ( + current_schema_state.compat_version is not None + and current_schema_state.compat_version > SCHEMA_VERSION + ): raise ValueError( "Cannot use this database as it is too " + "new for the server to understand" @@ -357,14 +374,26 @@ def _upgrade_existing_database( assert config is not None check_database_before_upgrade(cur, database_engine, config) - start_ver = current_version + # update schema_compat_version before we run any upgrades, so that if synapse + # gets downgraded again, it won't try to run against the upgraded database. + if ( + current_schema_state.compat_version is None + or current_schema_state.compat_version < SCHEMA_COMPAT_VERSION + ): + cur.execute("DELETE FROM schema_compat_version") + cur.execute( + "INSERT INTO schema_compat_version(compat_version) VALUES (?)", + (SCHEMA_COMPAT_VERSION,), + ) + + start_ver = current_schema_state.current_version # if we got to this schema version by running a full_schema rather than a series # of deltas, we should not run the deltas for this version. - if not upgraded: + if not current_schema_state.upgraded: start_ver += 1 - logger.debug("applied_delta_files: %s", applied_delta_files) + logger.debug("applied_delta_files: %s", current_schema_state.applied_deltas) if isinstance(database_engine, PostgresEngine): specific_engine_extension = ".postgres" @@ -440,7 +469,7 @@ def _upgrade_existing_database( absolute_path = entry.absolute_path logger.debug("Found file: %s (%s)", relative_path, absolute_path) - if relative_path in applied_delta_files: + if relative_path in current_schema_state.applied_deltas: continue root_name, ext = os.path.splitext(file_name) @@ -621,7 +650,7 @@ def execute_statements_from_stream(cur: Cursor, f: TextIO) -> None: def _get_or_create_schema_state( txn: Cursor, database_engine: BaseDatabaseEngine -) -> Optional[Tuple[int, List[str], bool]]: +) -> Optional[_SchemaState]: # Bluntly try creating the schema_version tables. sql_path = os.path.join(schema_path, "common", "schema_version.sql") executescript(txn, sql_path) @@ -629,17 +658,31 @@ def _get_or_create_schema_state( txn.execute("SELECT version, upgraded FROM schema_version") row = txn.fetchone() + if row is None: + # new database + return None + + current_version = int(row[0]) + upgraded = bool(row[1]) + + compat_version: Optional[int] = None + txn.execute("SELECT compat_version FROM schema_compat_version") + row = txn.fetchone() if row is not None: - current_version = int(row[0]) - txn.execute( - "SELECT file FROM applied_schema_deltas WHERE version >= ?", - (current_version,), - ) - applied_deltas = [d for d, in txn] - upgraded = bool(row[1]) - return current_version, applied_deltas, upgraded + compat_version = int(row[0]) + + txn.execute( + "SELECT file FROM applied_schema_deltas WHERE version >= ?", + (current_version,), + ) + applied_deltas = tuple(d for d, in txn) - return None + return _SchemaState( + current_version=current_version, + compat_version=compat_version, + applied_deltas=applied_deltas, + upgraded=upgraded, + ) @attr.s(slots=True) diff --git a/synapse/storage/schema/README.md b/synapse/storage/schema/README.md index 030153db64..729f44ea6c 100644 --- a/synapse/storage/schema/README.md +++ b/synapse/storage/schema/README.md @@ -1,37 +1,4 @@ # Synapse Database Schemas -This directory contains the schema files used to build Synapse databases. - -Synapse supports splitting its datastore across multiple physical databases (which can -be useful for large installations), and the schema files are therefore split according -to the logical database they are apply to. - -At the time of writing, the following "logical" databases are supported: - -* `state` - used to store Matrix room state (more specifically, `state_groups`, - their relationships and contents.) -* `main` - stores everything else. - -Addionally, the `common` directory contains schema files for tables which must be -present on *all* physical databases. - -## Full schema dumps - -In the `full_schemas` directories, only the most recently-numbered snapshot is useful -(`54` at the time of writing). Older snapshots (eg, `16`) are present for historical -reference only. - -## Building full schema dumps - -If you want to recreate these schemas, they need to be made from a database that -has had all background updates run. - -To do so, use `scripts-dev/make_full_schema.sh`. This will produce new -`full.sql.postgres` and `full.sql.sqlite` files. - -Ensure postgres is installed, then run: - - ./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/ - -NB at the time of writing, this script predates the split into separate `state`/`main` -databases so will require updates to handle that correctly. +This directory contains the schema files used to build Synapse databases. For more +information, see /docs/development/database_schema.md. diff --git a/synapse/storage/schema/__init__.py b/synapse/storage/schema/__init__.py index f0d9f23167..d36ba1d773 100644 --- a/synapse/storage/schema/__init__.py +++ b/synapse/storage/schema/__init__.py @@ -12,6 +12,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Remember to update this number every time a change is made to database -# schema files, so the users will be informed on server restarts. SCHEMA_VERSION = 59 +"""Represents the expectations made by the codebase about the database schema + +This should be incremented whenever the codebase changes its requirements on the +shape of the database schema (even if those requirements are backwards-compatible with +older versions of Synapse). + +See `README.md `_ for more information on how this +works. +""" + + +SCHEMA_COMPAT_VERSION = 59 +"""Limit on how far the synapse codebase can be rolled back without breaking db compat + +This value is stored in the database, and checked on startup. If the value in the +database is greater than SCHEMA_VERSION, then Synapse will refuse to start. +""" diff --git a/synapse/storage/schema/common/schema_version.sql b/synapse/storage/schema/common/schema_version.sql index 42e5cb6df5..f41fde5d2d 100644 --- a/synapse/storage/schema/common/schema_version.sql +++ b/synapse/storage/schema/common/schema_version.sql @@ -20,6 +20,13 @@ CREATE TABLE IF NOT EXISTS schema_version( CHECK (Lock='X') ); +CREATE TABLE IF NOT EXISTS schema_compat_version( + Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row. + -- The SCHEMA_VERSION of the oldest synapse this database can be used with + compat_version INTEGER NOT NULL, + CHECK (Lock='X') +); + CREATE TABLE IF NOT EXISTS applied_schema_deltas( version INTEGER NOT NULL, file TEXT NOT NULL, -- cgit 1.5.1 From 13577aa55ebe6087e8b813c0643bbb53148e9510 Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 11 Jun 2021 17:13:56 +0100 Subject: Notes on boolean columns in database schemas (#10164) --- changelog.d/10164.misc | 1 + docs/development/database_schema.md | 42 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 changelog.d/10164.misc (limited to 'docs') diff --git a/changelog.d/10164.misc b/changelog.d/10164.misc new file mode 100644 index 0000000000..a98f1e7c7a --- /dev/null +++ b/changelog.d/10164.misc @@ -0,0 +1 @@ +Add some developer documentation about boolean columns in database schemas. diff --git a/docs/development/database_schema.md b/docs/development/database_schema.md index 7fe8ec63e1..20740cf5ac 100644 --- a/docs/development/database_schema.md +++ b/docs/development/database_schema.md @@ -93,3 +93,45 @@ Ensure postgres is installed, then run: NB at the time of writing, this script predates the split into separate `state`/`main` databases so will require updates to handle that correctly. + +## Boolean columns + +Boolean columns require special treatment, since SQLite treats booleans the +same as integers. + +There are three separate aspects to this: + + * Any new boolean column must be added to the `BOOLEAN_COLUMNS` list in + `scripts/synapse_port_db`. This tells the port script to cast the integer + value from SQLite to a boolean before writing the value to the postgres + database. + + * Before SQLite 3.23, `TRUE` and `FALSE` were not recognised as constants by + SQLite, and the `IS [NOT] TRUE`/`IS [NOT] FALSE` operators were not + supported. This makes it necessary to avoid using `TRUE` and `FALSE` + constants in SQL commands. + + For example, to insert a `TRUE` value into the database, write: + + ```python + txn.execute("INSERT INTO tbl(col) VALUES (?)", (True, )) + ``` + + * Default values for new boolean columns present a particular + difficulty. Generally it is best to create separate schema files for + Postgres and SQLite. For example: + + ```sql + # in 00delta.sql.postgres: + ALTER TABLE tbl ADD COLUMN col BOOLEAN DEFAULT FALSE; + ``` + + ```sql + # in 00delta.sql.sqlite: + ALTER TABLE tbl ADD COLUMN col BOOLEAN DEFAULT 0; + ``` + + Note that there is a particularly insidious failure mode here: the Postgres + flavour will be accepted by SQLite 3.22, but will give a column whose + default value is the **string** `"FALSE"` - which, when cast back to a boolean + in Python, evaluates to `True`. -- cgit 1.5.1 From 0adc2882c1a67419207a500f00c41a94be51857a Mon Sep 17 00:00:00 2001 From: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Date: Wed, 16 Jun 2021 14:15:52 +0200 Subject: Fix broken links in documentation (#10180) * Fix broken links in documentation * newsfile --- changelog.d/10180.doc | 1 + docs/admin_api/README.rst | 4 ++-- docs/admin_api/delete_group.md | 2 +- docs/admin_api/event_reports.md | 4 ++-- docs/admin_api/media_admin_api.md | 4 ++-- docs/admin_api/purge_history_api.md | 2 +- docs/admin_api/room_membership.md | 2 +- docs/admin_api/rooms.md | 2 +- docs/admin_api/statistics.md | 2 +- docs/admin_api/user_admin_api.md | 40 ++++++++++++++++++------------------- docs/consent_tracking.md | 4 ++-- docs/federate.md | 6 +++--- docs/message_retention_policies.md | 4 ++-- docs/metrics-howto.md | 3 +-- docs/presence_router_module.md | 2 +- docs/reverse_proxy.md | 2 +- docs/sso_mapping_providers.md | 4 ++-- docs/systemd-with-workers/README.md | 14 +++++++------ docs/workers.md | 2 +- 19 files changed, 53 insertions(+), 51 deletions(-) create mode 100644 changelog.d/10180.doc (limited to 'docs') diff --git a/changelog.d/10180.doc b/changelog.d/10180.doc new file mode 100644 index 0000000000..1568450198 --- /dev/null +++ b/changelog.d/10180.doc @@ -0,0 +1 @@ +Fix broken links in documentation. \ No newline at end of file diff --git a/docs/admin_api/README.rst b/docs/admin_api/README.rst index 37cee87d32..8d6e76580a 100644 --- a/docs/admin_api/README.rst +++ b/docs/admin_api/README.rst @@ -2,7 +2,7 @@ Admin APIs ========== **Note**: The latest documentation can be viewed `here `_. -See `docs/README.md <../docs/README.md>`_ for more information. +See `docs/README.md <../README.md>`_ for more information. **Please update links to point to the website instead.** Existing files in this directory are preserved to maintain historical links, but may be moved in the future. @@ -10,5 +10,5 @@ are preserved to maintain historical links, but may be moved in the future. This directory includes documentation for the various synapse specific admin APIs available. Updates to the existing Admin API documentation should still be made to these files, but any new documentation files should instead be placed under -`docs/usage/administration/admin_api <../docs/usage/administration/admin_api>`_. +`docs/usage/administration/admin_api <../usage/administration/admin_api>`_. diff --git a/docs/admin_api/delete_group.md b/docs/admin_api/delete_group.md index 9c335ff759..2e0a1d2474 100644 --- a/docs/admin_api/delete_group.md +++ b/docs/admin_api/delete_group.md @@ -11,4 +11,4 @@ POST /_synapse/admin/v1/delete_group/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). diff --git a/docs/admin_api/event_reports.md b/docs/admin_api/event_reports.md index 186139185e..3abb06099c 100644 --- a/docs/admin_api/event_reports.md +++ b/docs/admin_api/event_reports.md @@ -7,7 +7,7 @@ The api is: GET /_synapse/admin/v1/event_reports?from=0&limit=10 ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). It returns a JSON body like the following: @@ -95,7 +95,7 @@ The api is: GET /_synapse/admin/v1/event_reports/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). It returns a JSON body like the following: diff --git a/docs/admin_api/media_admin_api.md b/docs/admin_api/media_admin_api.md index 9ab5269881..b033fc03ef 100644 --- a/docs/admin_api/media_admin_api.md +++ b/docs/admin_api/media_admin_api.md @@ -28,7 +28,7 @@ The API is: GET /_synapse/admin/v1/room//media ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). The API returns a JSON body like the following: ```json @@ -311,7 +311,7 @@ The following fields are returned in the JSON response body: * `deleted`: integer - The number of media items successfully deleted To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). If the user re-requests purged remote media, synapse will re-request the media from the originating server. diff --git a/docs/admin_api/purge_history_api.md b/docs/admin_api/purge_history_api.md index 25decc3e61..13b991eacf 100644 --- a/docs/admin_api/purge_history_api.md +++ b/docs/admin_api/purge_history_api.md @@ -17,7 +17,7 @@ POST /_synapse/admin/v1/purge_history/[/] ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) By default, events sent by local users are not deleted, as they may represent the only copies of this content in existence. (Events sent by remote users are diff --git a/docs/admin_api/room_membership.md b/docs/admin_api/room_membership.md index ed40366099..8a5ce191df 100644 --- a/docs/admin_api/room_membership.md +++ b/docs/admin_api/room_membership.md @@ -24,7 +24,7 @@ POST /_synapse/admin/v1/join/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). Response: diff --git a/docs/admin_api/rooms.md b/docs/admin_api/rooms.md index dc007fa00e..bb7828a525 100644 --- a/docs/admin_api/rooms.md +++ b/docs/admin_api/rooms.md @@ -443,7 +443,7 @@ with a body of: ``` To use it, you will need to authenticate by providing an ``access_token`` for a -server admin: see [Admin API](../../usage/administration/admin_api). +server admin: see [Admin API](../usage/administration/admin_api). A response body like the following is returned: diff --git a/docs/admin_api/statistics.md b/docs/admin_api/statistics.md index d93d52a3ac..1901f1eea0 100644 --- a/docs/admin_api/statistics.md +++ b/docs/admin_api/statistics.md @@ -10,7 +10,7 @@ GET /_synapse/admin/v1/statistics/users/media ``` To use it, you will need to authenticate by providing an `access_token` -for a server admin: see [Admin API](../../usage/administration/admin_api). +for a server admin: see [Admin API](../usage/administration/admin_api). A response body like the following is returned: diff --git a/docs/admin_api/user_admin_api.md b/docs/admin_api/user_admin_api.md index c835e4a0cd..ef1e735e33 100644 --- a/docs/admin_api/user_admin_api.md +++ b/docs/admin_api/user_admin_api.md @@ -11,7 +11,7 @@ GET /_synapse/admin/v2/users/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) It returns a JSON body like the following: @@ -78,7 +78,7 @@ with a body of: ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) URL parameters: @@ -119,7 +119,7 @@ GET /_synapse/admin/v2/users?from=0&limit=10&guests=false ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -237,7 +237,7 @@ See also: [Client Server API Whois](https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid). To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) It returns a JSON body like the following: @@ -294,7 +294,7 @@ with a body of: ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) The erase parameter is optional and defaults to `false`. An empty body may be passed for backwards compatibility. @@ -339,7 +339,7 @@ with a body of: ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) The parameter `new_password` is required. The parameter `logout_devices` is optional and defaults to `true`. @@ -354,7 +354,7 @@ GET /_synapse/admin/v1/users//admin ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -384,7 +384,7 @@ with a body of: ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) ## List room memberships of a user @@ -398,7 +398,7 @@ GET /_synapse/admin/v1/users//joined_rooms ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -443,7 +443,7 @@ GET /_synapse/admin/v1/users//media ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -591,7 +591,7 @@ GET /_synapse/admin/v2/users//devices ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -659,7 +659,7 @@ POST /_synapse/admin/v2/users//delete_devices ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) An empty JSON dict is returned. @@ -683,7 +683,7 @@ GET /_synapse/admin/v2/users//devices/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -731,7 +731,7 @@ PUT /_synapse/admin/v2/users//devices/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) An empty JSON dict is returned. @@ -760,7 +760,7 @@ DELETE /_synapse/admin/v2/users//devices/ ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) An empty JSON dict is returned. @@ -781,7 +781,7 @@ GET /_synapse/admin/v1/users//pushers ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -872,7 +872,7 @@ POST /_synapse/admin/v1/users//shadow_ban ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) An empty JSON dict is returned. @@ -897,7 +897,7 @@ GET /_synapse/admin/v1/users//override_ratelimit ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -939,7 +939,7 @@ POST /_synapse/admin/v1/users//override_ratelimit ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) A response body like the following is returned: @@ -984,7 +984,7 @@ DELETE /_synapse/admin/v1/users//override_ratelimit ``` To use it, you will need to authenticate by providing an `access_token` for a -server admin: [Admin API](../../usage/administration/admin_api) +server admin: [Admin API](../usage/administration/admin_api) An empty JSON dict is returned. diff --git a/docs/consent_tracking.md b/docs/consent_tracking.md index c586b5f0b6..3f997e5903 100644 --- a/docs/consent_tracking.md +++ b/docs/consent_tracking.md @@ -24,8 +24,8 @@ To enable this, first create templates for the policy and success pages. These should be stored on the local filesystem. These templates use the [Jinja2](http://jinja.pocoo.org) templating language, -and [docs/privacy_policy_templates](privacy_policy_templates) gives -examples of the sort of thing that can be done. +and [docs/privacy_policy_templates](https://github.com/matrix-org/synapse/tree/develop/docs/privacy_policy_templates/) +gives examples of the sort of thing that can be done. Note that the templates must be stored under a name giving the language of the template - currently this must always be `en` (for "English"); diff --git a/docs/federate.md b/docs/federate.md index b15cd724d1..89c2b19638 100644 --- a/docs/federate.md +++ b/docs/federate.md @@ -14,7 +14,7 @@ you set the `server_name` to match your machine's public DNS hostname. For this default configuration to work, you will need to listen for TLS connections on port 8448. The preferred way to do that is by using a -reverse proxy: see [reverse_proxy.md]() for instructions +reverse proxy: see [reverse_proxy.md](reverse_proxy.md) for instructions on how to correctly set one up. In some cases you might not want to run Synapse on the machine that has @@ -44,7 +44,7 @@ a complicated dance which requires connections in both directions). Another common problem is that people on other servers can't join rooms that you invite them to. This can be caused by an incorrectly-configured reverse -proxy: see [reverse_proxy.md]() for instructions on how to correctly +proxy: see [reverse_proxy.md](reverse_proxy.md) for instructions on how to correctly configure a reverse proxy. ### Known issues @@ -63,4 +63,4 @@ release of Synapse. If you want to get up and running quickly with a trio of homeservers in a private federation, there is a script in the `demo` directory. This is mainly -useful just for development purposes. See [demo/README](<../demo/README>). +useful just for development purposes. See [demo/README](https://github.com/matrix-org/synapse/tree/develop/demo/). diff --git a/docs/message_retention_policies.md b/docs/message_retention_policies.md index 75d2028e17..ea3d46cc10 100644 --- a/docs/message_retention_policies.md +++ b/docs/message_retention_policies.md @@ -51,7 +51,7 @@ clients. Support for this feature can be enabled and configured in the `retention` section of the Synapse configuration file (see the -[sample file](https://github.com/matrix-org/synapse/blob/v1.7.3/docs/sample_config.yaml#L332-L393)). +[sample file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518)). To enable support for message retention policies, set the setting `enabled` in this section to `true`. @@ -87,7 +87,7 @@ expired events from the database. They are only run if support for message retention policies is enabled in the server's configuration. If no configuration for purge jobs is configured by the server admin, Synapse will use a default configuration, which is described in the -[sample configuration file](https://github.com/matrix-org/synapse/blob/master/docs/sample_config.yaml#L332-L393). +[sample configuration file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518). Some server admins might want a finer control on when events are removed depending on an event's room's policy. This can be done by setting the diff --git a/docs/metrics-howto.md b/docs/metrics-howto.md index 6b84153274..4a77d5604c 100644 --- a/docs/metrics-howto.md +++ b/docs/metrics-howto.md @@ -72,8 +72,7 @@ ## Monitoring workers -To monitor a Synapse installation using -[workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md), +To monitor a Synapse installation using [workers](workers.md), every worker needs to be monitored independently, in addition to the main homeserver process. This is because workers don't send their metrics to the main homeserver process, but expose them diff --git a/docs/presence_router_module.md b/docs/presence_router_module.md index d2844915df..bf859e4254 100644 --- a/docs/presence_router_module.md +++ b/docs/presence_router_module.md @@ -30,7 +30,7 @@ presence to (for those users that the receiving user is considered interested in It does not include state for users who are currently offline, and it can only be called on workers that support sending federation. Additionally, this method must only be called from the process that has been configured to write to the -the [presence stream](https://github.com/matrix-org/synapse/blob/master/docs/workers.md#stream-writers). +the [presence stream](workers.md#stream-writers). By default, this is the main process, but another worker can be configured to do so. diff --git a/docs/reverse_proxy.md b/docs/reverse_proxy.md index cf1b835b9d..01db466f96 100644 --- a/docs/reverse_proxy.md +++ b/docs/reverse_proxy.md @@ -21,7 +21,7 @@ port 8448. Where these are different, we refer to the 'client port' and the 'federation port'. See [the Matrix specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names) for more details of the algorithm used for federation connections, and -[delegate.md]() for instructions on setting up delegation. +[delegate.md](delegate.md) for instructions on setting up delegation. **NOTE**: Your reverse proxy must not `canonicalise` or `normalise` the requested URI in any way (for example, by decoding `%xx` escapes). diff --git a/docs/sso_mapping_providers.md b/docs/sso_mapping_providers.md index 6db2dc8be5..7a407012e0 100644 --- a/docs/sso_mapping_providers.md +++ b/docs/sso_mapping_providers.md @@ -108,7 +108,7 @@ A custom mapping provider must specify the following methods: Synapse has a built-in OpenID mapping provider if a custom provider isn't specified in the config. It is located at -[`synapse.handlers.oidc.JinjaOidcMappingProvider`](../synapse/handlers/oidc.py). +[`synapse.handlers.oidc.JinjaOidcMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/oidc.py). ## SAML Mapping Providers @@ -194,4 +194,4 @@ A custom mapping provider must specify the following methods: Synapse has a built-in SAML mapping provider if a custom provider isn't specified in the config. It is located at -[`synapse.handlers.saml.DefaultSamlMappingProvider`](../synapse/handlers/saml.py). +[`synapse.handlers.saml.DefaultSamlMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/saml.py). diff --git a/docs/systemd-with-workers/README.md b/docs/systemd-with-workers/README.md index a1135e9ed5..a7de2de88a 100644 --- a/docs/systemd-with-workers/README.md +++ b/docs/systemd-with-workers/README.md @@ -6,16 +6,18 @@ well as a `matrix-synapse-worker@` service template for any workers you require. Additionally, to group the required services, it sets up a `matrix-synapse.target`. -See the folder [system](system) for the systemd unit files. +See the folder [system](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/system/) +for the systemd unit files. -The folder [workers](workers) contains an example configuration for the -`federation_reader` worker. +The folder [workers](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/workers/) +contains an example configuration for the `federation_reader` worker. ## Synapse configuration files See [workers.md](../workers.md) for information on how to set up the configuration files and reverse-proxy correctly. You can find an example worker -config in the [workers](workers) folder. +config in the [workers](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/workers/) +folder. Systemd manages daemonization itself, so ensure that none of the configuration files set either `daemonize` or `worker_daemonize`. @@ -29,8 +31,8 @@ There is no need for a separate configuration file for the master process. ## Set up 1. Adjust synapse configuration files as above. -1. Copy the `*.service` and `*.target` files in [system](system) to -`/etc/systemd/system`. +1. Copy the `*.service` and `*.target` files in [system](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/system/) +to `/etc/systemd/system`. 1. Run `systemctl daemon-reload` to tell systemd to load the new unit files. 1. Run `systemctl enable matrix-synapse.service`. This will configure the synapse master process to be started as part of the `matrix-synapse.target` diff --git a/docs/workers.md b/docs/workers.md index 46b5e4b737..797758ee84 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -16,7 +16,7 @@ workers only work with PostgreSQL-based Synapse deployments. SQLite should only be used for demo purposes and any admin considering workers should already be running PostgreSQL. -See also https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability +See also [Matrix.org blog post](https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability) for a higher level overview. ## Main process/worker communication -- cgit 1.5.1 From b8b282aa32063d712e276373b6bc90c39cecc353 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Wed, 16 Jun 2021 13:31:55 +0100 Subject: A guide to the request log lines format. (#8436) This doc is short but a useful guide to what the request log lines mean. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Erik Johnston Co-authored-by: Daniele Sluijters --- changelog.d/8436.doc | 1 + docs/SUMMARY.md | 1 + docs/usage/administration/request_log.md | 44 ++++++++++++++++++++++++++++++++ 3 files changed, 46 insertions(+) create mode 100644 changelog.d/8436.doc create mode 100644 docs/usage/administration/request_log.md (limited to 'docs') diff --git a/changelog.d/8436.doc b/changelog.d/8436.doc new file mode 100644 index 0000000000..77fc098200 --- /dev/null +++ b/changelog.d/8436.doc @@ -0,0 +1 @@ +Add a new guide to decoding request logs. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index af2c968c9a..01ef4ff600 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -61,6 +61,7 @@ - [Server Version](admin_api/version_api.md) - [Manhole](manhole.md) - [Monitoring](metrics-howto.md) + - [Request log format](usage/administration/request_log.md) - [Scripts]() # Development diff --git a/docs/usage/administration/request_log.md b/docs/usage/administration/request_log.md new file mode 100644 index 0000000000..316304c734 --- /dev/null +++ b/docs/usage/administration/request_log.md @@ -0,0 +1,44 @@ +# Request log format + +HTTP request logs are written by synapse (see [`site.py`](../synapse/http/site.py) for details). + +See the following for how to decode the dense data available from the default logging configuration. + +``` +2020-10-01 12:00:00,000 - synapse.access.http.8008 - 311 - INFO - PUT-1000- 192.168.0.1 - 8008 - {another-matrix-server.com} Processed request: 0.100sec/-0.000sec (0.000sec, 0.000sec) (0.001sec/0.090sec/3) 11B !200 "PUT /_matrix/federation/v1/send/1600000000000 HTTP/1.1" "Synapse/1.20.1" [0 dbevts] +-AAAAAAAAAAAAAAAAAAAAA- -BBBBBBBBBBBBBBBBBBBBBB- -C- -DD- -EEEEEE- -FFFFFFFFF- -GG- -HHHHHHHHHHHHHHHHHHHHHHH- -IIIIII- -JJJJJJJ- -KKKKKK-, -LLLLLL- -MMMMMMM- -NNNNNN- O -P- -QQ- -RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- -SSSSSSSSSSSS- -TTTTTT- +``` + + +| Part | Explanation | +| ----- | ------------ | +| AAAA | Timestamp request was logged (not recieved) | +| BBBB | Logger name (`synapse.access.(http\|https).`, where 'tag' is defined in the `listeners` config section, normally the port) | +| CCCC | Line number in code | +| DDDD | Log Level | +| EEEE | Request Identifier (This identifier is shared by related log lines)| +| FFFF | Source IP (Or X-Forwarded-For if enabled) | +| GGGG | Server Port | +| HHHH | Federated Server or Local User making request (blank if unauthenticated or not supplied) | +| IIII | Total Time to process the request | +| JJJJ | Time to send response over network once generated (this may be negative if the socket is closed before the response is generated)| +| KKKK | Userland CPU time | +| LLLL | System CPU time | +| MMMM | Total time waiting for a free DB connection from the pool across all parallel DB work from this request | +| NNNN | Total time waiting for response to DB queries across all parallel DB work from this request | +| OOOO | Count of DB transactions performed | +| PPPP | Response body size | +| QQQQ | Response status code (prefixed with ! if the socket was closed before the response was generated) | +| RRRR | Request | +| SSSS | User-agent | +| TTTT | Events fetched from DB to service this request (note that this does not include events fetched from the cache) | + + +MMMM / NNNN can be greater than IIII if there are multiple slow database queries +running in parallel. + +Some actions can result in multiple identical http requests, which will return +the same data, but only the first request will report time/transactions in +`KKKK`/`LLLL`/`MMMM`/`NNNN`/`OOOO` - the others will be awaiting the first query to return a +response and will simultaneously return with the first request, but with very +small processing times. -- cgit 1.5.1