diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 8f39ae0270..01ef4ff600 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -61,6 +61,7 @@
- [Server Version](admin_api/version_api.md)
- [Manhole](manhole.md)
- [Monitoring](metrics-howto.md)
+ - [Request log format](usage/administration/request_log.md)
- [Scripts]()
# Development
@@ -69,6 +70,7 @@
- [Git Usage](dev/git.md)
- [Testing]()
- [OpenTracing](opentracing.md)
+ - [Database Schemas](development/database_schema.md)
- [Synapse Architecture]()
- [Log Contexts](log_contexts.md)
- [Replication](replication.md)
@@ -84,4 +86,4 @@
- [Scripts]()
# Other
- - [Dependency Deprecation Policy](deprecation_policy.md)
\ No newline at end of file
+ - [Dependency Deprecation Policy](deprecation_policy.md)
diff --git a/docs/admin_api/README.rst b/docs/admin_api/README.rst
index 37cee87d32..8d6e76580a 100644
--- a/docs/admin_api/README.rst
+++ b/docs/admin_api/README.rst
@@ -2,7 +2,7 @@ Admin APIs
==========
**Note**: The latest documentation can be viewed `here <https://matrix-org.github.io/synapse>`_.
-See `docs/README.md <../docs/README.md>`_ for more information.
+See `docs/README.md <../README.md>`_ for more information.
**Please update links to point to the website instead.** Existing files in this directory
are preserved to maintain historical links, but may be moved in the future.
@@ -10,5 +10,5 @@ are preserved to maintain historical links, but may be moved in the future.
This directory includes documentation for the various synapse specific admin
APIs available. Updates to the existing Admin API documentation should still
be made to these files, but any new documentation files should instead be placed under
-`docs/usage/administration/admin_api <../docs/usage/administration/admin_api>`_.
+`docs/usage/administration/admin_api <../usage/administration/admin_api>`_.
diff --git a/docs/admin_api/delete_group.md b/docs/admin_api/delete_group.md
index 9c335ff759..2e0a1d2474 100644
--- a/docs/admin_api/delete_group.md
+++ b/docs/admin_api/delete_group.md
@@ -11,4 +11,4 @@ POST /_synapse/admin/v1/delete_group/<group_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
diff --git a/docs/admin_api/event_reports.md b/docs/admin_api/event_reports.md
index 186139185e..3abb06099c 100644
--- a/docs/admin_api/event_reports.md
+++ b/docs/admin_api/event_reports.md
@@ -7,7 +7,7 @@ The api is:
GET /_synapse/admin/v1/event_reports?from=0&limit=10
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
It returns a JSON body like the following:
@@ -95,7 +95,7 @@ The api is:
GET /_synapse/admin/v1/event_reports/<report_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
It returns a JSON body like the following:
diff --git a/docs/admin_api/media_admin_api.md b/docs/admin_api/media_admin_api.md
index 9ab5269881..b033fc03ef 100644
--- a/docs/admin_api/media_admin_api.md
+++ b/docs/admin_api/media_admin_api.md
@@ -28,7 +28,7 @@ The API is:
GET /_synapse/admin/v1/room/<room_id>/media
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
The API returns a JSON body like the following:
```json
@@ -311,7 +311,7 @@ The following fields are returned in the JSON response body:
* `deleted`: integer - The number of media items successfully deleted
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
If the user re-requests purged remote media, synapse will re-request the media
from the originating server.
diff --git a/docs/admin_api/purge_history_api.md b/docs/admin_api/purge_history_api.md
index 25decc3e61..13b991eacf 100644
--- a/docs/admin_api/purge_history_api.md
+++ b/docs/admin_api/purge_history_api.md
@@ -17,7 +17,7 @@ POST /_synapse/admin/v1/purge_history/<room_id>[/<event_id>]
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
By default, events sent by local users are not deleted, as they may represent
the only copies of this content in existence. (Events sent by remote users are
diff --git a/docs/admin_api/room_membership.md b/docs/admin_api/room_membership.md
index ed40366099..8a5ce191df 100644
--- a/docs/admin_api/room_membership.md
+++ b/docs/admin_api/room_membership.md
@@ -24,7 +24,7 @@ POST /_synapse/admin/v1/join/<room_id_or_alias>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
Response:
diff --git a/docs/admin_api/rooms.md b/docs/admin_api/rooms.md
index dc007fa00e..bb7828a525 100644
--- a/docs/admin_api/rooms.md
+++ b/docs/admin_api/rooms.md
@@ -443,7 +443,7 @@ with a body of:
```
To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see [Admin API](../../usage/administration/admin_api).
+server admin: see [Admin API](../usage/administration/admin_api).
A response body like the following is returned:
diff --git a/docs/admin_api/statistics.md b/docs/admin_api/statistics.md
index d93d52a3ac..1901f1eea0 100644
--- a/docs/admin_api/statistics.md
+++ b/docs/admin_api/statistics.md
@@ -10,7 +10,7 @@ GET /_synapse/admin/v1/statistics/users/media
```
To use it, you will need to authenticate by providing an `access_token`
-for a server admin: see [Admin API](../../usage/administration/admin_api).
+for a server admin: see [Admin API](../usage/administration/admin_api).
A response body like the following is returned:
diff --git a/docs/admin_api/user_admin_api.md b/docs/admin_api/user_admin_api.md
index c835e4a0cd..ef1e735e33 100644
--- a/docs/admin_api/user_admin_api.md
+++ b/docs/admin_api/user_admin_api.md
@@ -11,7 +11,7 @@ GET /_synapse/admin/v2/users/<user_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
It returns a JSON body like the following:
@@ -78,7 +78,7 @@ with a body of:
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
URL parameters:
@@ -119,7 +119,7 @@ GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -237,7 +237,7 @@ See also: [Client Server
API Whois](https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-admin-whois-userid).
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
It returns a JSON body like the following:
@@ -294,7 +294,7 @@ with a body of:
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
The erase parameter is optional and defaults to `false`.
An empty body may be passed for backwards compatibility.
@@ -339,7 +339,7 @@ with a body of:
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
The parameter `new_password` is required.
The parameter `logout_devices` is optional and defaults to `true`.
@@ -354,7 +354,7 @@ GET /_synapse/admin/v1/users/<user_id>/admin
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -384,7 +384,7 @@ with a body of:
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
## List room memberships of a user
@@ -398,7 +398,7 @@ GET /_synapse/admin/v1/users/<user_id>/joined_rooms
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -443,7 +443,7 @@ GET /_synapse/admin/v1/users/<user_id>/media
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -591,7 +591,7 @@ GET /_synapse/admin/v2/users/<user_id>/devices
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -659,7 +659,7 @@ POST /_synapse/admin/v2/users/<user_id>/delete_devices
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
@@ -683,7 +683,7 @@ GET /_synapse/admin/v2/users/<user_id>/devices/<device_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -731,7 +731,7 @@ PUT /_synapse/admin/v2/users/<user_id>/devices/<device_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
@@ -760,7 +760,7 @@ DELETE /_synapse/admin/v2/users/<user_id>/devices/<device_id>
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
@@ -781,7 +781,7 @@ GET /_synapse/admin/v1/users/<user_id>/pushers
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -872,7 +872,7 @@ POST /_synapse/admin/v1/users/<user_id>/shadow_ban
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
@@ -897,7 +897,7 @@ GET /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -939,7 +939,7 @@ POST /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
A response body like the following is returned:
@@ -984,7 +984,7 @@ DELETE /_synapse/admin/v1/users/<user_id>/override_ratelimit
```
To use it, you will need to authenticate by providing an `access_token` for a
-server admin: [Admin API](../../usage/administration/admin_api)
+server admin: [Admin API](../usage/administration/admin_api)
An empty JSON dict is returned.
diff --git a/docs/consent_tracking.md b/docs/consent_tracking.md
index c586b5f0b6..3f997e5903 100644
--- a/docs/consent_tracking.md
+++ b/docs/consent_tracking.md
@@ -24,8 +24,8 @@ To enable this, first create templates for the policy and success pages.
These should be stored on the local filesystem.
These templates use the [Jinja2](http://jinja.pocoo.org) templating language,
-and [docs/privacy_policy_templates](privacy_policy_templates) gives
-examples of the sort of thing that can be done.
+and [docs/privacy_policy_templates](https://github.com/matrix-org/synapse/tree/develop/docs/privacy_policy_templates/)
+gives examples of the sort of thing that can be done.
Note that the templates must be stored under a name giving the language of the
template - currently this must always be `en` (for "English");
diff --git a/docs/development/database_schema.md b/docs/development/database_schema.md
new file mode 100644
index 0000000000..20740cf5ac
--- /dev/null
+++ b/docs/development/database_schema.md
@@ -0,0 +1,137 @@
+# Synapse database schema files
+
+Synapse's database schema is stored in the `synapse.storage.schema` module.
+
+## Logical databases
+
+Synapse supports splitting its datastore across multiple physical databases (which can
+be useful for large installations), and the schema files are therefore split according
+to the logical database they apply to.
+
+At the time of writing, the following "logical" databases are supported:
+
+* `state` - used to store Matrix room state (more specifically, `state_groups`,
+ their relationships and contents).
+* `main` - stores everything else.
+
+Additionally, the `common` directory contains schema files for tables which must be
+present on *all* physical databases.
+
+## Synapse schema versions
+
+Synapse manages its database schema via "schema versions". These are mainly used to
+help avoid confusion if the Synapse codebase is rolled back after the database is
+updated. They work as follows:
+
+ * The Synapse codebase defines a constant `synapse.storage.schema.SCHEMA_VERSION`
+ which represents the expectations made about the database by that version. For
+ example, as of Synapse v1.36, this is `59`.
+
+ * The database stores a "compatibility version" in
+ `schema_compat_version.compat_version` which defines the `SCHEMA_VERSION` of the
+ oldest version of Synapse which will work with the database. On startup, if
+ `compat_version` is found to be newer than `SCHEMA_VERSION`, Synapse will refuse to
+ start.
+
+ Synapse automatically updates this field from
+ `synapse.storage.schema.SCHEMA_COMPAT_VERSION`.
+
+ * Whenever a backwards-incompatible change is made to the database format (normally
+ via a `delta` file), `synapse.storage.schema.SCHEMA_COMPAT_VERSION` is also updated
+ so that administrators can not accidentally roll back to a too-old version of Synapse.
+
+Generally, the goal is to maintain compatibility with at least one or two previous
+releases of Synapse, so any substantial change tends to require multiple releases and a
+bit of forward-planning to get right.
+
+As a worked example: we want to remove the `room_stats_historical` table. Here is how it
+might pan out.
+
+ 1. Replace any code that *reads* from `room_stats_historical` with alternative
+ implementations, but keep writing to it in case of rollback to an earlier version.
+ Also, increase `synapse.storage.schema.SCHEMA_VERSION`. In this
+ instance, there is no existing code which reads from `room_stats_historical`, so
+ our starting point is:
+
+ v1.36.0: `SCHEMA_VERSION=59`, `SCHEMA_COMPAT_VERSION=59`
+
+ 2. Next (say in Synapse v1.37.0): remove the code that *writes* to
+ `room_stats_historical`, but don’t yet remove the table in case of rollback to
+ v1.36.0. Again, we increase `synapse.storage.schema.SCHEMA_VERSION`, but
+ because we have not broken compatibility with v1.36, we do not yet update
+ `SCHEMA_COMPAT_VERSION`. We now have:
+
+ v1.37.0: `SCHEMA_VERSION=60`, `SCHEMA_COMPAT_VERSION=59`.
+
+ 3. Later (say in Synapse v1.38.0): we can remove the table altogether. This will
+ break compatibility with v1.36.0, so we must update `SCHEMA_COMPAT_VERSION` accordingly.
+ There is no need to update `synapse.storage.schema.SCHEMA_VERSION`, since there is no
+ change to the Synapse codebase here. So we end up with:
+
+ v1.38.0: `SCHEMA_VERSION=60`, `SCHEMA_COMPAT_VERSION=60`.
+
+If in doubt about whether to update `SCHEMA_VERSION` or not, it is generally best to
+lean towards doing so.
+
+## Full schema dumps
+
+In the `full_schemas` directories, only the most recently-numbered snapshot is used
+(`54` at the time of writing). Older snapshots (eg, `16`) are present for historical
+reference only.
+
+### Building full schema dumps
+
+If you want to recreate these schemas, they need to be made from a database that
+has had all background updates run.
+
+To do so, use `scripts-dev/make_full_schema.sh`. This will produce new
+`full.sql.postgres` and `full.sql.sqlite` files.
+
+Ensure postgres is installed, then run:
+
+ ./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/
+
+NB at the time of writing, this script predates the split into separate `state`/`main`
+databases so will require updates to handle that correctly.
+
+## Boolean columns
+
+Boolean columns require special treatment, since SQLite treats booleans the
+same as integers.
+
+There are three separate aspects to this:
+
+ * Any new boolean column must be added to the `BOOLEAN_COLUMNS` list in
+ `scripts/synapse_port_db`. This tells the port script to cast the integer
+ value from SQLite to a boolean before writing the value to the postgres
+ database.
+
+ * Before SQLite 3.23, `TRUE` and `FALSE` were not recognised as constants by
+ SQLite, and the `IS [NOT] TRUE`/`IS [NOT] FALSE` operators were not
+ supported. This makes it necessary to avoid using `TRUE` and `FALSE`
+ constants in SQL commands.
+
+ For example, to insert a `TRUE` value into the database, write:
+
+ ```python
+ txn.execute("INSERT INTO tbl(col) VALUES (?)", (True, ))
+ ```
+
+ * Default values for new boolean columns present a particular
+ difficulty. Generally it is best to create separate schema files for
+ Postgres and SQLite. For example:
+
+ ```sql
+ # in 00delta.sql.postgres:
+ ALTER TABLE tbl ADD COLUMN col BOOLEAN DEFAULT FALSE;
+ ```
+
+ ```sql
+ # in 00delta.sql.sqlite:
+ ALTER TABLE tbl ADD COLUMN col BOOLEAN DEFAULT 0;
+ ```
+
+ Note that there is a particularly insidious failure mode here: the Postgres
+ flavour will be accepted by SQLite 3.22, but will give a column whose
+ default value is the **string** `"FALSE"` - which, when cast back to a boolean
+ in Python, evaluates to `True`.
diff --git a/docs/federate.md b/docs/federate.md
index b15cd724d1..89c2b19638 100644
--- a/docs/federate.md
+++ b/docs/federate.md
@@ -14,7 +14,7 @@ you set the `server_name` to match your machine's public DNS hostname.
For this default configuration to work, you will need to listen for TLS
connections on port 8448. The preferred way to do that is by using a
-reverse proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions
+reverse proxy: see [reverse_proxy.md](reverse_proxy.md) for instructions
on how to correctly set one up.
In some cases you might not want to run Synapse on the machine that has
@@ -44,7 +44,7 @@ a complicated dance which requires connections in both directions).
Another common problem is that people on other servers can't join rooms that
you invite them to. This can be caused by an incorrectly-configured reverse
-proxy: see [reverse_proxy.md](<reverse_proxy.md>) for instructions on how to correctly
+proxy: see [reverse_proxy.md](reverse_proxy.md) for instructions on how to correctly
configure a reverse proxy.
### Known issues
@@ -63,4 +63,4 @@ release of Synapse.
If you want to get up and running quickly with a trio of homeservers in a
private federation, there is a script in the `demo` directory. This is mainly
-useful just for development purposes. See [demo/README](<../demo/README>).
+useful just for development purposes. See [demo/README](https://github.com/matrix-org/synapse/tree/develop/demo/).
diff --git a/docs/message_retention_policies.md b/docs/message_retention_policies.md
index 75d2028e17..ea3d46cc10 100644
--- a/docs/message_retention_policies.md
+++ b/docs/message_retention_policies.md
@@ -51,7 +51,7 @@ clients.
Support for this feature can be enabled and configured in the
`retention` section of the Synapse configuration file (see the
-[sample file](https://github.com/matrix-org/synapse/blob/v1.7.3/docs/sample_config.yaml#L332-L393)).
+[sample file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518)).
To enable support for message retention policies, set the setting
`enabled` in this section to `true`.
@@ -87,7 +87,7 @@ expired events from the database. They are only run if support for
message retention policies is enabled in the server's configuration. If
no configuration for purge jobs is configured by the server admin,
Synapse will use a default configuration, which is described in the
-[sample configuration file](https://github.com/matrix-org/synapse/blob/master/docs/sample_config.yaml#L332-L393).
+[sample configuration file](https://github.com/matrix-org/synapse/blob/v1.36.0/docs/sample_config.yaml#L451-L518).
Some server admins might want a finer control on when events are removed
depending on an event's room's policy. This can be done by setting the
diff --git a/docs/metrics-howto.md b/docs/metrics-howto.md
index 6b84153274..4a77d5604c 100644
--- a/docs/metrics-howto.md
+++ b/docs/metrics-howto.md
@@ -72,8 +72,7 @@
## Monitoring workers
-To monitor a Synapse installation using
-[workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md),
+To monitor a Synapse installation using [workers](workers.md),
every worker needs to be monitored independently, in addition to
the main homeserver process. This is because workers don't send
their metrics to the main homeserver process, but expose them
diff --git a/docs/presence_router_module.md b/docs/presence_router_module.md
index d2844915df..bf859e4254 100644
--- a/docs/presence_router_module.md
+++ b/docs/presence_router_module.md
@@ -30,7 +30,7 @@ presence to (for those users that the receiving user is considered interested in
It does not include state for users who are currently offline, and it can only be
called on workers that support sending federation. Additionally, this method must
only be called from the process that has been configured to write to the
-the [presence stream](https://github.com/matrix-org/synapse/blob/master/docs/workers.md#stream-writers).
+the [presence stream](workers.md#stream-writers).
By default, this is the main process, but another worker can be configured to do
so.
diff --git a/docs/reverse_proxy.md b/docs/reverse_proxy.md
index cf1b835b9d..01db466f96 100644
--- a/docs/reverse_proxy.md
+++ b/docs/reverse_proxy.md
@@ -21,7 +21,7 @@ port 8448. Where these are different, we refer to the 'client port' and the
'federation port'. See [the Matrix
specification](https://matrix.org/docs/spec/server_server/latest#resolving-server-names)
for more details of the algorithm used for federation connections, and
-[delegate.md](<delegate.md>) for instructions on setting up delegation.
+[delegate.md](delegate.md) for instructions on setting up delegation.
**NOTE**: Your reverse proxy must not `canonicalise` or `normalise`
the requested URI in any way (for example, by decoding `%xx` escapes).
diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml
index 7b97f73a29..f8925a5e24 100644
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -954,6 +954,10 @@ media_store_path: "DATADIR/media_store"
# The largest allowed upload size in bytes
#
+# If you are using a reverse proxy you may also need to set this value in
+# your reverse proxy's config. Notably Nginx has a small max body size by default.
+# See https://matrix-org.github.io/synapse/develop/reverse_proxy.html.
+#
#max_upload_size: 50M
# Maximum number of pixels that will be thumbnailed
diff --git a/docs/sso_mapping_providers.md b/docs/sso_mapping_providers.md
index 6db2dc8be5..7a407012e0 100644
--- a/docs/sso_mapping_providers.md
+++ b/docs/sso_mapping_providers.md
@@ -108,7 +108,7 @@ A custom mapping provider must specify the following methods:
Synapse has a built-in OpenID mapping provider if a custom provider isn't
specified in the config. It is located at
-[`synapse.handlers.oidc.JinjaOidcMappingProvider`](../synapse/handlers/oidc.py).
+[`synapse.handlers.oidc.JinjaOidcMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/oidc.py).
## SAML Mapping Providers
@@ -194,4 +194,4 @@ A custom mapping provider must specify the following methods:
Synapse has a built-in SAML mapping provider if a custom provider isn't
specified in the config. It is located at
-[`synapse.handlers.saml.DefaultSamlMappingProvider`](../synapse/handlers/saml.py).
+[`synapse.handlers.saml.DefaultSamlMappingProvider`](https://github.com/matrix-org/synapse/blob/develop/synapse/handlers/saml.py).
diff --git a/docs/systemd-with-workers/README.md b/docs/systemd-with-workers/README.md
index a1135e9ed5..a7de2de88a 100644
--- a/docs/systemd-with-workers/README.md
+++ b/docs/systemd-with-workers/README.md
@@ -6,16 +6,18 @@ well as a `matrix-synapse-worker@` service template for any workers you
require. Additionally, to group the required services, it sets up a
`matrix-synapse.target`.
-See the folder [system](system) for the systemd unit files.
+See the folder [system](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/system/)
+for the systemd unit files.
-The folder [workers](workers) contains an example configuration for the
-`federation_reader` worker.
+The folder [workers](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/workers/)
+contains an example configuration for the `federation_reader` worker.
## Synapse configuration files
See [workers.md](../workers.md) for information on how to set up the
configuration files and reverse-proxy correctly. You can find an example worker
-config in the [workers](workers) folder.
+config in the [workers](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/workers/)
+folder.
Systemd manages daemonization itself, so ensure that none of the configuration
files set either `daemonize` or `worker_daemonize`.
@@ -29,8 +31,8 @@ There is no need for a separate configuration file for the master process.
## Set up
1. Adjust synapse configuration files as above.
-1. Copy the `*.service` and `*.target` files in [system](system) to
-`/etc/systemd/system`.
+1. Copy the `*.service` and `*.target` files in [system](https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/system/)
+to `/etc/systemd/system`.
1. Run `systemctl daemon-reload` to tell systemd to load the new unit files.
1. Run `systemctl enable matrix-synapse.service`. This will configure the
synapse master process to be started as part of the `matrix-synapse.target`
diff --git a/docs/usage/administration/request_log.md b/docs/usage/administration/request_log.md
new file mode 100644
index 0000000000..316304c734
--- /dev/null
+++ b/docs/usage/administration/request_log.md
@@ -0,0 +1,44 @@
+# Request log format
+
+HTTP request logs are written by synapse (see [`site.py`](../synapse/http/site.py) for details).
+
+See the following for how to decode the dense data available from the default logging configuration.
+
+```
+2020-10-01 12:00:00,000 - synapse.access.http.8008 - 311 - INFO - PUT-1000- 192.168.0.1 - 8008 - {another-matrix-server.com} Processed request: 0.100sec/-0.000sec (0.000sec, 0.000sec) (0.001sec/0.090sec/3) 11B !200 "PUT /_matrix/federation/v1/send/1600000000000 HTTP/1.1" "Synapse/1.20.1" [0 dbevts]
+-AAAAAAAAAAAAAAAAAAAAA- -BBBBBBBBBBBBBBBBBBBBBB- -C- -DD- -EEEEEE- -FFFFFFFFF- -GG- -HHHHHHHHHHHHHHHHHHHHHHH- -IIIIII- -JJJJJJJ- -KKKKKK-, -LLLLLL- -MMMMMMM- -NNNNNN- O -P- -QQ- -RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- -SSSSSSSSSSSS- -TTTTTT-
+```
+
+
+| Part | Explanation |
+| ----- | ------------ |
+| AAAA | Timestamp request was logged (not recieved) |
+| BBBB | Logger name (`synapse.access.(http\|https).<tag>`, where 'tag' is defined in the `listeners` config section, normally the port) |
+| CCCC | Line number in code |
+| DDDD | Log Level |
+| EEEE | Request Identifier (This identifier is shared by related log lines)|
+| FFFF | Source IP (Or X-Forwarded-For if enabled) |
+| GGGG | Server Port |
+| HHHH | Federated Server or Local User making request (blank if unauthenticated or not supplied) |
+| IIII | Total Time to process the request |
+| JJJJ | Time to send response over network once generated (this may be negative if the socket is closed before the response is generated)|
+| KKKK | Userland CPU time |
+| LLLL | System CPU time |
+| MMMM | Total time waiting for a free DB connection from the pool across all parallel DB work from this request |
+| NNNN | Total time waiting for response to DB queries across all parallel DB work from this request |
+| OOOO | Count of DB transactions performed |
+| PPPP | Response body size |
+| QQQQ | Response status code (prefixed with ! if the socket was closed before the response was generated) |
+| RRRR | Request |
+| SSSS | User-agent |
+| TTTT | Events fetched from DB to service this request (note that this does not include events fetched from the cache) |
+
+
+MMMM / NNNN can be greater than IIII if there are multiple slow database queries
+running in parallel.
+
+Some actions can result in multiple identical http requests, which will return
+the same data, but only the first request will report time/transactions in
+`KKKK`/`LLLL`/`MMMM`/`NNNN`/`OOOO` - the others will be awaiting the first query to return a
+response and will simultaneously return with the first request, but with very
+small processing times.
diff --git a/docs/workers.md b/docs/workers.md
index 46b5e4b737..797758ee84 100644
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -16,7 +16,7 @@ workers only work with PostgreSQL-based Synapse deployments. SQLite should only
be used for demo purposes and any admin considering workers should already be
running PostgreSQL.
-See also https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability
+See also [Matrix.org blog post](https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability)
for a higher level overview.
## Main process/worker communication
|
