diff --git a/docs/usage/administration/README.md b/docs/usage/administration/README.md
deleted file mode 100644
index e1e57546ab..0000000000
--- a/docs/usage/administration/README.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# Administration
-
-This section contains information on managing your Synapse homeserver. This includes:
-
-* Managing users, rooms and media via the Admin API.
-* Setting up metrics and monitoring to give you insight into your homeserver's health.
-* Configuring structured logging.
\ No newline at end of file
diff --git a/docs/usage/administration/admin_faq.md b/docs/usage/administration/admin_faq.md
deleted file mode 100644
index e349a4e34f..0000000000
--- a/docs/usage/administration/admin_faq.md
+++ /dev/null
@@ -1,185 +0,0 @@
-## Admin FAQ
-
-How do I become a server admin?
----
-If your server already has an admin account you should use the [User Admin API](../../admin_api/user_admin_api.md#Change-whether-a-user-is-a-server-administrator-or-not) to promote other accounts to become admins.
-
-If you don't have any admin accounts yet you won't be able to use the admin API, so you'll have to edit the database manually. Manually editing the database is generally not recommended so once you have an admin account: use the admin APIs to make further changes.
-
-```sql
-UPDATE users SET admin = 1 WHERE name = '@foo:bar.com';
-```
-What servers are my server talking to?
----
-Run this sql query on your db:
-```sql
-SELECT * FROM destinations;
-```
-
-What servers are currently participating in this room?
----
-Run this sql query on your db:
-```sql
-SELECT DISTINCT split_part(state_key, ':', 2)
- FROM current_state_events AS c
- INNER JOIN room_memberships AS m USING (room_id, event_id)
- WHERE room_id = '!cURbafjkfsMDVwdRDQ:matrix.org' AND membership = 'join';
-```
-
-What users are registered on my server?
----
-```sql
-SELECT NAME from users;
-```
-
-Manually resetting passwords
----
-Users can reset their password through their client. Alternatively, a server admin
-can reset a user's password using the [admin API](../../admin_api/user_admin_api.md#reset-password).
-
-
-I have a problem with my server. Can I just delete my database and start again?
----
-Deleting your database is unlikely to make anything better.
-
-It's easy to make the mistake of thinking that you can start again from a clean slate by dropping your database, but things don't work like that in a federated network: lots of other servers have information about your server.
-
-For example: other servers might think that you are in a room, your server will think that you are not, and you'll probably be unable to interact with that room in a sensible way ever again.
-
-In general, there are better solutions to any problem than dropping the database. Come and seek help in https://matrix.to/#/#synapse:matrix.org.
-
-There are two exceptions when it might be sensible to delete your database and start again:
-* You have *never* joined any rooms which are federated with other servers. For instance, a local deployment which the outside world can't talk to.
-* You are changing the `server_name` in the homeserver configuration. In effect this makes your server a completely new one from the point of view of the network, so in this case it makes sense to start with a clean database.
-(In both cases you probably also want to clear out the media_store.)
-
-I've stuffed up access to my room, how can I delete it to free up the alias?
----
-Using the following curl command:
-```
-curl -H 'Authorization: Bearer <access-token>' -X DELETE https://matrix.org/_matrix/client/r0/directory/room/<room-alias>
-```
-`<access-token>` - can be obtained in riot by looking in the riot settings, down the bottom is:
-Access Token:\<click to reveal\>
-
-`<room-alias>` - the room alias, eg. #my_room:matrix.org this possibly needs to be URL encoded also, for example %23my_room%3Amatrix.org
-
-How can I find the lines corresponding to a given HTTP request in my homeserver log?
----
-
-Synapse tags each log line according to the HTTP request it is processing. When it finishes processing each request, it logs a line containing the words `Processed request: `. For example:
-
-```
-2019-02-14 22:35:08,196 - synapse.access.http.8008 - 302 - INFO - GET-37 - ::1 - 8008 - {@richvdh:localhost} Processed request: 0.173sec/0.001sec (0.002sec, 0.000sec) (0.027sec/0.026sec/2) 687B 200 "GET /_matrix/client/r0/sync HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" [0 dbevts]"
-```
-
-Here we can see that the request has been tagged with `GET-37`. (The tag depends on the method of the HTTP request, so might start with `GET-`, `PUT-`, `POST-`, `OPTIONS-` or `DELETE-`.) So to find all lines corresponding to this request, we can do:
-
-```
-grep 'GET-37' homeserver.log
-```
-
-If you want to paste that output into a github issue or matrix room, please remember to surround it with triple-backticks (```) to make it legible (see https://help.github.com/en/articles/basic-writing-and-formatting-syntax#quoting-code).
-
-
-What do all those fields in the 'Processed' line mean?
----
-See [Request log format](request_log.md).
-
-
-What are the biggest rooms on my server?
----
-
-```sql
-SELECT s.canonical_alias, g.room_id, count(*) AS num_rows
-FROM
- state_groups_state AS g,
- room_stats_state AS s
-WHERE g.room_id = s.room_id
-GROUP BY s.canonical_alias, g.room_id
-ORDER BY num_rows desc
-LIMIT 10;
-```
-
-You can also use the [List Room API](../../admin_api/rooms.md#list-room-api)
-and `order_by` `state_events`.
-
-
-People can't accept room invitations from me
----
-
-The typical failure mode here is that you send an invitation to someone
-to join a room or direct chat, but when they go to accept it, they get an
-error (typically along the lines of "Invalid signature"). They might see
-something like the following in their logs:
-
- 2019-09-11 19:32:04,271 - synapse.federation.transport.server - 288 - WARNING - GET-11752 - authenticate_request failed: 401: Invalid signature for server <server> with key ed25519:a_EqML: Unable to verify signature for <server>
-
-This is normally caused by a misconfiguration in your reverse-proxy. See [the reverse proxy docs](docs/reverse_proxy.md) and double-check that your settings are correct.
-
-
-Help!! Synapse is slow and eats all my RAM/CPU!
------------------------------------------------
-
-First, ensure you are running the latest version of Synapse, using Python 3
-with a [PostgreSQL database](../../setup/postgres.md).
-
-Synapse's architecture is quite RAM hungry currently - we deliberately
-cache a lot of recent room data and metadata in RAM in order to speed up
-common requests. We'll improve this in the future, but for now the easiest
-way to either reduce the RAM usage (at the risk of slowing things down)
-is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
-variable. The default is 0.5, which can be decreased to reduce RAM usage
-in memory constrained environments, or increased if performance starts to
-degrade.
-
-However, degraded performance due to a low cache factor, common on
-machines with slow disks, often leads to explosions in memory use due
-backlogged requests. In this case, reducing the cache factor will make
-things worse. Instead, try increasing it drastically. 2.0 is a good
-starting value.
-
-Using [libjemalloc](https://jemalloc.net) can also yield a significant
-improvement in overall memory use, and especially in terms of giving back
-RAM to the OS. To use it, the library must simply be put in the
-LD_PRELOAD environment variable when launching Synapse. On Debian, this
-can be done by installing the `libjemalloc1` package and adding this
-line to `/etc/default/matrix-synapse`:
-
- LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
-
-This made a significant difference on Python 2.7 - it's unclear how
-much of an improvement it provides on Python 3.x.
-
-If you're encountering high CPU use by the Synapse process itself, you
-may be affected by a bug with presence tracking that leads to a
-massive excess of outgoing federation requests (see [discussion](https://github.com/matrix-org/synapse/issues/3971)). If metrics
-indicate that your server is also issuing far more outgoing federation
-requests than can be accounted for by your users' activity, this is a
-likely cause. The misbehavior can be worked around by disabling presence
-in the Synapse config file: [see here](../configuration/config_documentation.md#presence).
-
-
-Running out of File Handles
----------------------------
-
-If Synapse runs out of file handles, it typically fails badly - live-locking
-at 100% CPU, and/or failing to accept new TCP connections (blocking the
-connecting client). Matrix currently can legitimately use a lot of file handles,
-thanks to busy rooms like `#matrix:matrix.org` containing hundreds of participating
-servers. The first time a server talks in a room it will try to connect
-simultaneously to all participating servers, which could exhaust the available
-file descriptors between DNS queries & HTTPS sockets, especially if DNS is slow
-to respond. (We need to improve the routing algorithm used to be better than
-full mesh, but as of March 2019 this hasn't happened yet).
-
-If you hit this failure mode, we recommend increasing the maximum number of
-open file handles to be at least 4096 (assuming a default of 1024 or 256).
-This is typically done by editing ``/etc/security/limits.conf``
-
-Separately, Synapse may leak file handles if inbound HTTP requests get stuck
-during processing - e.g. blocked behind a lock or talking to a remote server etc.
-This is best diagnosed by matching up the 'Received request' and 'Processed request'
-log lines and looking for any 'Processed request' lines which take more than
-a few seconds to execute. Please let us know at [`#synapse:matrix.org`](https://matrix.to/#/#synapse-dev:matrix.org) if
-you see this failure mode so we can help debug it, however.
diff --git a/docs/usage/administration/database_maintenance_tools.md b/docs/usage/administration/database_maintenance_tools.md
deleted file mode 100644
index 92b805d413..0000000000
--- a/docs/usage/administration/database_maintenance_tools.md
+++ /dev/null
@@ -1,18 +0,0 @@
-This blog post by Victor Berger explains how to use many of the tools listed on this page: https://levans.fr/shrink-synapse-database.html
-
-# List of useful tools and scripts for maintenance Synapse database:
-
-## [Purge Remote Media API](../../admin_api/media_admin_api.md#purge-remote-media-api)
-The purge remote media API allows server admins to purge old cached remote media.
-
-## [Purge Local Media API](../../admin_api/media_admin_api.md#delete-local-media)
-This API deletes the *local* media from the disk of your own server.
-
-## [Purge History API](../../admin_api/purge_history_api.md)
-The purge history API allows server admins to purge historic events from their database, reclaiming disk space.
-
-## [synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state)
-Tool for compressing (deduplicating) `state_groups_state` table.
-
-## [SQL for analyzing Synapse PostgreSQL database stats](useful_sql_for_admins.md)
-Some easy SQL that reports useful stats about your Synapse database.
\ No newline at end of file
diff --git a/docs/usage/administration/monitoring/reporting_homeserver_usage_statistics.md b/docs/usage/administration/monitoring/reporting_homeserver_usage_statistics.md
deleted file mode 100644
index 4e53f9883a..0000000000
--- a/docs/usage/administration/monitoring/reporting_homeserver_usage_statistics.md
+++ /dev/null
@@ -1,81 +0,0 @@
-# Reporting Homeserver Usage Statistics
-
-When generating your Synapse configuration file, you are asked whether you
-would like to report usage statistics to Matrix.org. These statistics
-provide the foundation a glimpse into the number of Synapse homeservers
-participating in the network, as well as statistics such as the number of
-rooms being created and messages being sent. This feature is sometimes
-affectionately called "phone home" stats. Reporting
-[is optional](../../configuration/config_documentation.md#report_stats)
-and the reporting endpoint
-[can be configured](../../configuration/config_documentation.md#report_stats_endpoint),
-in case you would like to instead report statistics from a set of homeservers
-to your own infrastructure.
-
-This documentation aims to define the statistics available and the
-homeserver configuration options that exist to tweak it.
-
-## Available Statistics
-
-The following statistics are sent to the configured reporting endpoint:
-
-| Statistic Name | Type | Description |
-|----------------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `homeserver` | string | The homeserver's server name. |
-| `memory_rss` | int | The memory usage of the process (in kilobytes on Unix-based systems, bytes on MacOS). |
-| `cpu_average` | int | CPU time in % of a single core (not % of all cores). |
-| `server_context` | string | An arbitrary string used to group statistics from a set of homeservers. |
-| `timestamp` | int | The current time, represented as the number of seconds since the epoch. |
-| `uptime_seconds` | int | The number of seconds since the homeserver was last started. |
-| `python_version` | string | The Python version number in use (e.g "3.7.1"). Taken from `sys.version_info`. |
-| `total_users` | int | The number of registered users on the homeserver. |
-| `total_nonbridged_users` | int | The number of users, excluding those created by an Application Service. |
-| `daily_user_type_native` | int | The number of native users created in the last 24 hours. |
-| `daily_user_type_guest` | int | The number of guest users created in the last 24 hours. |
-| `daily_user_type_bridged` | int | The number of users created by Application Services in the last 24 hours. |
-| `total_room_count` | int | The total number of rooms present on the homeserver. |
-| `daily_active_users` | int | The number of unique users[^1] that have used the homeserver in the last 24 hours. |
-| `monthly_active_users` | int | The number of unique users[^1] that have used the homeserver in the last 30 days. |
-| `daily_active_rooms` | int | The number of rooms that have had a (state) event with the type `m.room.message` sent in them in the last 24 hours. |
-| `daily_active_e2ee_rooms` | int | The number of rooms that have had a (state) event with the type `m.room.encrypted` sent in them in the last 24 hours. |
-| `daily_messages` | int | The number of (state) events with the type `m.room.message` seen in the last 24 hours. |
-| `daily_e2ee_messages` | int | The number of (state) events with the type `m.room.encrypted` seen in the last 24 hours. |
-| `daily_sent_messages` | int | The number of (state) events sent by a local user with the type `m.room.message` seen in the last 24 hours. |
-| `daily_sent_e2ee_messages` | int | The number of (state) events sent by a local user with the type `m.room.encrypted` seen in the last 24 hours. |
-| `r30_users_all` | int | The number of 30 day retained users, defined as users who have created their accounts more than 30 days ago, where they were last seen at most 30 days ago and where those two timestamps are over 30 days apart. Includes clients that do not fit into the below r30 client types. |
-| `r30_users_android` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "Android" in the user agent string. |
-| `r30_users_ios` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "iOS" in the user agent string. |
-| `r30_users_electron` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "Electron" in the user agent string. |
-| `r30_users_web` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "Mozilla" or "Gecko" in the user agent string. |
-| `r30v2_users_all` | int | The number of 30 day retained users, with a revised algorithm. Defined as users that appear more than once in the past 60 days, and have more than 30 days between the most and least recent appearances in the past 60 days. Includes clients that do not fit into the below r30 client types. |
-| `r30v2_users_android` | int | The number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "android" (case-insensitive) in the user agent string. |
-| `r30v2_users_ios` | int | The number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "ios" (case-insensitive) in the user agent string. |
-| `r30v2_users_electron` | int | The number of 30 day retained users, as defined above. Filtered only to clients with ("riot" or "element") and "electron" (case-insensitive) in the user agent string. |
-| `r30v2_users_web` | int | The number of 30 day retained users, as defined above. Filtered only to clients with "mozilla" or "gecko" (case-insensitive) in the user agent string. |
-| `cache_factor` | int | The configured [`global factor`](../../configuration/config_documentation.md#caching) value for caching. |
-| `event_cache_size` | int | The configured [`event_cache_size`](../../configuration/config_documentation.md#caching) value for caching. |
-| `database_engine` | string | The database engine that is in use. Either "psycopg2" meaning PostgreSQL is in use, or "sqlite3" for SQLite3. |
-| `database_server_version` | string | The version of the database server. Examples being "10.10" for PostgreSQL server version 10.0, and "3.38.5" for SQLite 3.38.5 installed on the system. |
-| `log_level` | string | The log level in use. Examples are "INFO", "WARNING", "ERROR", "DEBUG", etc. |
-
-
-[^1]: Native matrix users and guests are always counted. If the
-[`track_puppeted_user_ips`](../../configuration/config_documentation.md#track_puppeted_user_ips)
-option is set to `true`, "puppeted" users (users that an Application Service have performed
-[an action on behalf of](https://spec.matrix.org/v1.3/application-service-api/#identity-assertion))
-will also be counted. Note that an Application Service can "puppet" any user in their
-[user namespace](https://spec.matrix.org/v1.3/application-service-api/#registration),
-not only users that the Application Service has created. If this happens, the Application Service
-will additionally be counted as a user (irrespective of `track_puppeted_user_ips`).
-
-## Using a Custom Statistics Collection Server
-
-If statistics reporting is enabled, the endpoint that Synapse sends metrics to is configured by the
-[`report_stats_endpoint`](../../configuration/config_documentation.md#report_stats_endpoint) config
-option. By default, statistics are sent to Matrix.org.
-
-If you would like to set up your own statistics collection server and send metrics there, you may
-consider using one of the following known implementations:
-
-* [Matrix.org's Panopticon](https://github.com/matrix-org/panopticon)
-* [Famedly's Barad-dûr](https://gitlab.com/famedly/company/devops/services/barad-dur)
diff --git a/docs/usage/administration/monthly_active_users.md b/docs/usage/administration/monthly_active_users.md
deleted file mode 100644
index d4e9037284..0000000000
--- a/docs/usage/administration/monthly_active_users.md
+++ /dev/null
@@ -1,84 +0,0 @@
-# Monthly Active Users
-
-Synapse can be configured to record the number of monthly active users (also referred to as MAU) on a given homeserver.
-For clarity's sake, MAU only tracks local users.
-
-Please note that the metrics recorded by the [Homeserver Usage Stats](../../usage/administration/monitoring/reporting_homeserver_usage_statistics.md)
-are calculated differently. The `monthly_active_users` from the usage stats does not take into account any
-of the rules below, and counts any users who have made a request to the homeserver in the last 30 days.
-
-See the [configuration manual](../../usage/configuration/config_documentation.md#limit_usage_by_mau) for details on how to configure MAU.
-
-## Calculating active users
-
-Individual user activity is measured in active days. If a user performs an action, the exact time of that action is then recorded. When
-calculating the MAU figure, any users with a recorded action in the last 30 days are considered part of the cohort. Days are measured
-as a rolling window from the current system time to 30 days ago.
-
-So for example, if Synapse were to calculate the active users on the 15th July at 13:25, it would include any activity from 15th June 13:25 onwards.
-
-A user is **never** considered active if they are either:
- - Part of the trial day cohort (described below)
- - Owned by an application service.
- - Note: This **only** covers users that are part of an application service `namespaces.users` registration. The namespace
- must also be marked as `exclusive`.
-
-Otherwise, any request to Synapse will mark the user as active. Please note that registration will not mark a user as active *unless*
-they register with a 3pid that is included in the config field `mau_limits_reserved_threepids`.
-
-The Prometheus metric for MAU is refreshed every 5 minutes.
-
-Once an hour, Synapse checks to see if any users are inactive (with only activity timestamps later than 30 days). These users
-are removed from the active users cohort. If they then become active, they are immediately restored to the cohort.
-
-It is important to note that **deactivated** users are not immediately removed from the pool of active users, but as these users won't
-perform actions they will eventually be removed from the cohort.
-
-### Trial days
-
-If the config option `mau_trial_days` is set, a user must have been active this many days **after** registration to be active. A user is in the
-trial period if their registration timestamp (also known as the `creation_ts`) is less than `mau_trial_days` old.
-
-As an example, if `mau_trial_days` is set to `3` and a user is active **after** 3 days (72 hours from registration time) then they will be counted as active.
-
-The `mau_appservice_trial_days` config further extends this rule by applying different durations depending on the `appservice_id` of the user.
-Users registered by an application service will be recorded with an `appservice_id` matching the `id` key in the registration file for that service.
-
-
-## Limiting usage of the homeserver when the maximum MAU is reached
-
-If both config options `limit_usage_by_mau` and `max_mau_value` is set, and the current MAU value exceeds the maximum value, the
-homeserver will begin to block some actions.
-
-Individual users matching **any** of the below criteria never have their actions blocked:
- - Considered part of the cohort of MAU users.
- - Considered part of the trial period.
- - Registered as a `support` user.
- - Application service users if `track_appservice_user_ips` is NOT set.
-
-Please not that server admins are **not** exempt from blocking.
-
-The following actions are blocked when the MAU limit is exceeded:
- - Logging in
- - Sending events
- - Creating rooms
- - Syncing
-
-Registration is also blocked for all new signups *unless* the user is registering with a threepid included in the `mau_limits_reserved_threepids`
-config value.
-
-When a request is blocked, the response will have the `errcode` `M_RESOURCE_LIMIT_EXCEEDED`.
-
-## Metrics
-
-Synapse records several different prometheus metrics for MAU.
-
-`synapse_admin_mau:current` records the current MAU figure for native (non-application-service) users.
-
-`synapse_admin_mau:max` records the maximum MAU as dictated by the `max_mau_value` config value.
-
-`synapse_admin_mau_current_mau_by_service` records the current MAU including application service users. The label `app_service` can be used
-to filter by a specific service ID. This *also* includes non-application-service users under `app_service=native` .
-
-`synapse_admin_mau:registered_reserved_users` records the number of users specified in `mau_limits_reserved_threepids` which have
-registered accounts on the homeserver.
diff --git a/docs/usage/administration/request_log.md b/docs/usage/administration/request_log.md
deleted file mode 100644
index 82f5ac7b96..0000000000
--- a/docs/usage/administration/request_log.md
+++ /dev/null
@@ -1,44 +0,0 @@
-# Request log format
-
-HTTP request logs are written by synapse (see [`site.py`](../synapse/http/site.py) for details).
-
-See the following for how to decode the dense data available from the default logging configuration.
-
-```
-2020-10-01 12:00:00,000 - synapse.access.http.8008 - 311 - INFO - PUT-1000- 192.168.0.1 - 8008 - {another-matrix-server.com} Processed request: 0.100sec/-0.000sec (0.000sec, 0.000sec) (0.001sec/0.090sec/3) 11B !200 "PUT /_matrix/federation/v1/send/1600000000000 HTTP/1.1" "Synapse/1.20.1" [0 dbevts]
--AAAAAAAAAAAAAAAAAAAAA- -BBBBBBBBBBBBBBBBBBBBBB- -C- -DD- -EEEEEE- -FFFFFFFFF- -GG- -HHHHHHHHHHHHHHHHHHHHHHH- -IIIIII- -JJJJJJJ- -KKKKKK-, -LLLLLL- -MMMMMMM- -NNNNNN- O -P- -QQ- -RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- -SSSSSSSSSSSS- -TTTTTT-
-```
-
-
-| Part | Explanation |
-| ----- | ------------ |
-| AAAA | Timestamp request was logged (not received) |
-| BBBB | Logger name (`synapse.access.(http\|https).<tag>`, where 'tag' is defined in the `listeners` config section, normally the port) |
-| CCCC | Line number in code |
-| DDDD | Log Level |
-| EEEE | Request Identifier (This identifier is shared by related log lines)|
-| FFFF | Source IP (Or X-Forwarded-For if enabled) |
-| GGGG | Server Port |
-| HHHH | Federated Server or Local User making request (blank if unauthenticated or not supplied).<br/>If this is of the form `@aaa:example.com|@bbb:example.com`, then that means that `@aaa:example.com` is authenticated but they are controlling `@bbb:example.com`, e.g. if `aaa` is controlling `bbb` [via the admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#login-as-a-user). |
-| IIII | Total Time to process the request |
-| JJJJ | Time to send response over network once generated (this may be negative if the socket is closed before the response is generated)|
-| KKKK | Userland CPU time |
-| LLLL | System CPU time |
-| MMMM | Total time waiting for a free DB connection from the pool across all parallel DB work from this request |
-| NNNN | Total time waiting for response to DB queries across all parallel DB work from this request |
-| OOOO | Count of DB transactions performed |
-| PPPP | Response body size |
-| QQQQ | Response status code<br/>Suffixed with `!` if the socket was closed before the response was generated.<br/>A `499!` status code indicates that Synapse also cancelled request processing after the socket was closed.<br/> |
-| RRRR | Request |
-| SSSS | User-agent |
-| TTTT | Events fetched from DB to service this request (note that this does not include events fetched from the cache) |
-
-
-MMMM / NNNN can be greater than IIII if there are multiple slow database queries
-running in parallel.
-
-Some actions can result in multiple identical http requests, which will return
-the same data, but only the first request will report time/transactions in
-`KKKK`/`LLLL`/`MMMM`/`NNNN`/`OOOO` - the others will be awaiting the first query to return a
-response and will simultaneously return with the first request, but with very
-small processing times.
diff --git a/docs/usage/administration/state_groups.md b/docs/usage/administration/state_groups.md
deleted file mode 100644
index f1dee7accf..0000000000
--- a/docs/usage/administration/state_groups.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# How do State Groups work?
-
-As a general rule, I encourage people who want to understand the deepest darkest secrets of the database schema to drop by #synapse-dev:matrix.org and ask questions.
-
-However, one question that comes up frequently is that of how "state groups" work, and why the `state_groups_state` table gets so big, so here's an attempt to answer that question.
-
-We need to be able to relatively quickly calculate the state of a room at any point in that room's history. In other words, we need to know the state of the room at each event in that room. This is done as follows:
-
-A sequence of events where the state is the same are grouped together into a `state_group`; the mapping is recorded in `event_to_state_groups`. (Technically speaking, since a state event usually changes the state in the room, we are recording the state of the room *after* the given event id: which is to say, to a handwavey simplification, the first event in a state group is normally a state event, and others in the same state group are normally non-state-events.)
-
-`state_groups` records, for each state group, the id of the room that we're looking at, and also the id of the first event in that group. (I'm not sure if that event id is used much in practice.)
-
-Now, if we stored all the room state for each `state_group`, that would be a huge amount of data. Instead, for each state group, we normally store the difference between the state in that group and some other state group, and only occasionally (every 100 state changes or so) record the full state.
-
-So, most state groups have an entry in `state_group_edges` (don't ask me why it's not a column in `state_groups`) which records the previous state group in the room, and `state_groups_state` records the differences in state since that previous state group.
-
-A full state group just records the event id for each piece of state in the room at that point.
-
-## Known bugs with state groups
-
-There are various reasons that we can end up creating many more state groups than we need: see https://github.com/matrix-org/synapse/issues/3364 for more details.
-
-## Compression tool
-
-There is a tool at https://github.com/matrix-org/rust-synapse-compress-state which can compress the `state_groups_state` on a room by-room basis (essentially, it reduces the number of "full" state groups). This can result in dramatic reductions of the storage used.
\ No newline at end of file
diff --git a/docs/usage/administration/understanding_synapse_through_grafana_graphs.md b/docs/usage/administration/understanding_synapse_through_grafana_graphs.md
deleted file mode 100644
index 6d6db0ebfe..0000000000
--- a/docs/usage/administration/understanding_synapse_through_grafana_graphs.md
+++ /dev/null
@@ -1,84 +0,0 @@
-## Understanding Synapse through Grafana graphs
-
-It is possible to monitor much of the internal state of Synapse using [Prometheus](https://prometheus.io)
-metrics and [Grafana](https://grafana.com/).
-A guide for configuring Synapse to provide metrics is available [here](../../metrics-howto.md)
-and information on setting up Grafana is [here](https://github.com/matrix-org/synapse/tree/master/contrib/grafana).
-In this setup, Prometheus will periodically scrape the information Synapse provides and
-store a record of it over time. Grafana is then used as an interface to query and
-present this information through a series of pretty graphs.
-
-Once you have grafana set up, and assuming you're using [our grafana dashboard template](https://github.com/matrix-org/synapse/blob/master/contrib/grafana/synapse.json), look for the following graphs when debugging a slow/overloaded Synapse:
-
-## Message Event Send Time
-
-
-
-This, along with the CPU and Memory graphs, is a good way to check the general health of your Synapse instance. It represents how long it takes for a user on your homeserver to send a message.
-
-## Transaction Count and Transaction Duration
-
-
-
-
-
-These graphs show the database transactions that are occurring the most frequently, as well as those are that are taking the most amount of time to execute.
-
-
-
-In the first graph, we can see obvious spikes corresponding to lots of `get_user_by_id` transactions. This would be useful information to figure out which part of the Synapse codebase is potentially creating a heavy load on the system. However, be sure to cross-reference this with Transaction Duration, which states that `get_users_by_id` is actually a very quick database transaction and isn't causing as much load as others, like `persist_events`:
-
-
-
-Still, it's probably worth investigating why we're getting users from the database that often, and whether it's possible to reduce the amount of queries we make by adjusting our cache factor(s).
-
-The `persist_events` transaction is responsible for saving new room events to the Synapse database, so can often show a high transaction duration.
-
-## Federation
-
-The charts in the "Federation" section show information about incoming and outgoing federation requests. Federation data can be divided into two basic types:
-
-- PDU (Persistent Data Unit) - room events: messages, state events (join/leave), etc. These are permanently stored in the database.
-- EDU (Ephemeral Data Unit) - other data, which need not be stored permanently, such as read receipts, typing notifications.
-
-The "Outgoing EDUs by type" chart shows the EDUs within outgoing federation requests by type: `m.device_list_update`, `m.direct_to_device`, `m.presence`, `m.receipt`, `m.typing`.
-
-If you see a large number of `m.presence` EDUs and are having trouble with too much CPU load, you can disable `presence` in the Synapse config. See also [#3971](https://github.com/matrix-org/synapse/issues/3971).
-
-## Caches
-
-
-
-
-
-This is quite a useful graph. It shows how many times Synapse attempts to retrieve a piece of data from a cache which the cache did not contain, thus resulting in a call to the database. We can see here that the `_get_joined_profile_from_event_id` cache is being requested a lot, and often the data we're after is not cached.
-
-Cross-referencing this with the Eviction Rate graph, which shows that entries are being evicted from `_get_joined_profile_from_event_id` quite often:
-
-
-
-we should probably consider raising the size of that cache by raising its cache factor (a multiplier value for the size of an individual cache). Information on doing so is available [here](https://github.com/matrix-org/synapse/blob/ee421e524478c1ad8d43741c27379499c2f6135c/docs/sample_config.yaml#L608-L642) (note that the configuration of individual cache factors through the configuration file is available in Synapse v1.14.0+, whereas doing so through environment variables has been supported for a very long time). Note that this will increase Synapse's overall memory usage.
-
-## Forward Extremities
-
-
-
-Forward extremities are the leaf events at the end of a DAG in a room, aka events that have no children. The more that exist in a room, the more [state resolution](https://spec.matrix.org/v1.1/server-server-api/#room-state-resolution) that Synapse needs to perform (hint: it's an expensive operation). While Synapse has code to prevent too many of these existing at one time in a room, bugs can sometimes make them crop up again.
-
-If a room has >10 forward extremities, it's worth checking which room is the culprit and potentially removing them using the SQL queries mentioned in [#1760](https://github.com/matrix-org/synapse/issues/1760).
-
-## Garbage Collection
-
-
-
-Large spikes in garbage collection times (bigger than shown here, I'm talking in the
-multiple seconds range), can cause lots of problems in Synapse performance. It's more an
-indicator of problems, and a symptom of other problems though, so check other graphs for what might be causing it.
-
-## Final Thoughts
-
-If you're still having performance problems with your Synapse instance and you've
-tried everything you can, it may just be a lack of system resources. Consider adding
-more CPU and RAM, and make use of [worker mode](../configuration/workers.md)
-to make use of multiple CPU cores / multiple machines for your homeserver.
-
diff --git a/docs/usage/administration/useful_sql_for_admins.md b/docs/usage/administration/useful_sql_for_admins.md
deleted file mode 100644
index f3b97f9576..0000000000
--- a/docs/usage/administration/useful_sql_for_admins.md
+++ /dev/null
@@ -1,207 +0,0 @@
-## Some useful SQL queries for Synapse Admins
-
-## Size of full matrix db
-```sql
-SELECT pg_size_pretty( pg_database_size( 'matrix' ) );
-```
-
-### Result example:
-```
-pg_size_pretty
-----------------
- 6420 MB
-(1 row)
-```
-
-## Show top 20 larger tables by row count
-```sql
-SELECT relname, n_live_tup AS "rows"
- FROM pg_stat_user_tables
- ORDER BY n_live_tup DESC
- LIMIT 20;
-```
-This query is quick, but may be very approximate, for exact number of rows use:
-```sql
-SELECT COUNT(*) FROM <table_name>;
-```
-
-### Result example:
-```
-state_groups_state - 161687170
-event_auth - 8584785
-event_edges - 6995633
-event_json - 6585916
-event_reference_hashes - 6580990
-events - 6578879
-received_transactions - 5713989
-event_to_state_groups - 4873377
-stream_ordering_to_exterm - 4136285
-current_state_delta_stream - 3770972
-event_search - 3670521
-state_events - 2845082
-room_memberships - 2785854
-cache_invalidation_stream - 2448218
-state_groups - 1255467
-state_group_edges - 1229849
-current_state_events - 1222905
-users_in_public_rooms - 364059
-device_lists_stream - 326903
-user_directory_search - 316433
-```
-
-## Show top 20 larger tables by storage size
-```sql
-SELECT nspname || '.' || relname AS "relation",
- pg_size_pretty(pg_total_relation_size(c.oid)) AS "total_size"
- FROM pg_class c
- LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace)
- WHERE nspname NOT IN ('pg_catalog', 'information_schema')
- AND c.relkind <> 'i'
- AND nspname !~ '^pg_toast'
- ORDER BY pg_total_relation_size(c.oid) DESC
- LIMIT 20;
-```
-
-### Result example:
-```
-public.state_groups_state - 27 GB
-public.event_json - 9855 MB
-public.events - 3675 MB
-public.event_edges - 3404 MB
-public.received_transactions - 2745 MB
-public.event_reference_hashes - 1864 MB
-public.event_auth - 1775 MB
-public.stream_ordering_to_exterm - 1663 MB
-public.event_search - 1370 MB
-public.room_memberships - 1050 MB
-public.event_to_state_groups - 948 MB
-public.current_state_delta_stream - 711 MB
-public.state_events - 611 MB
-public.presence_stream - 530 MB
-public.current_state_events - 525 MB
-public.cache_invalidation_stream - 466 MB
-public.receipts_linearized - 279 MB
-public.state_groups - 160 MB
-public.device_lists_remote_cache - 124 MB
-public.state_group_edges - 122 MB
-```
-
-## Show top 20 larger rooms by state events count
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#list-room-api)
-and set parameter `order_by=state_events`.
-
-```sql
-SELECT r.name, s.room_id, s.current_state_events
- FROM room_stats_current s
- LEFT JOIN room_stats_state r USING (room_id)
- ORDER BY current_state_events DESC
- LIMIT 20;
-```
-
-and by state_group_events count:
-```sql
-SELECT rss.name, s.room_id, COUNT(s.room_id)
- FROM state_groups_state s
- LEFT JOIN room_stats_state rss USING (room_id)
- GROUP BY s.room_id, rss.name
- ORDER BY COUNT(s.room_id) DESC
- LIMIT 20;
-```
-
-plus same, but with join removed for performance reasons:
-```sql
-SELECT s.room_id, COUNT(s.room_id)
- FROM state_groups_state s
- GROUP BY s.room_id
- ORDER BY COUNT(s.room_id) DESC
- LIMIT 20;
-```
-
-## Show top 20 rooms by new events count in last 1 day:
-```sql
-SELECT e.room_id, r.name, COUNT(e.event_id) cnt
- FROM events e
- LEFT JOIN room_stats_state r USING (room_id)
- WHERE e.origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 day') * 1000
- GROUP BY e.room_id, r.name
- ORDER BY cnt DESC
- LIMIT 20;
-```
-
-## Show top 20 users on homeserver by sent events (messages) at last month:
-Caution. This query does not use any indexes, can be slow and create load on the database.
-```sql
-SELECT COUNT(*), sender
- FROM events
- WHERE (type = 'm.room.encrypted' OR type = 'm.room.message')
- AND origin_server_ts >= DATE_PART('epoch', NOW() - INTERVAL '1 month') * 1000
- GROUP BY sender
- ORDER BY COUNT(*) DESC
- LIMIT 20;
-```
-
-## Show last 100 messages from needed user, with room names:
-```sql
-SELECT e.room_id, r.name, e.event_id, e.type, e.content, j.json
- FROM events e
- LEFT JOIN event_json j USING (room_id)
- LEFT JOIN room_stats_state r USING (room_id)
- WHERE sender = '@LOGIN:example.com'
- AND e.type = 'm.room.message'
- ORDER BY stream_ordering DESC
- LIMIT 100;
-```
-
-## Show rooms with names, sorted by events in this rooms
-
-**Sort and order with bash**
-```bash
-echo "SELECT event_json.room_id, room_stats_state.name FROM event_json, room_stats_state \
-WHERE room_stats_state.room_id = event_json.room_id" | psql -d synapse -h localhost -U synapse_user -t \
-| sort | uniq -c | sort -n
-```
-Documentation for `psql` command line parameters: https://www.postgresql.org/docs/current/app-psql.html
-
-**Sort and order with SQL**
-```sql
-SELECT COUNT(*), event_json.room_id, room_stats_state.name
- FROM event_json, room_stats_state
- WHERE room_stats_state.room_id = event_json.room_id
- GROUP BY event_json.room_id, room_stats_state.name
- ORDER BY COUNT(*) DESC
- LIMIT 50;
-```
-
-### Result example:
-```
- 9459 !FPUfgzXYWTKgIrwKxW:matrix.org | This Week in Matrix
- 9459 !FPUfgzXYWTKgIrwKxW:matrix.org | This Week in Matrix (TWIM)
- 17799 !iDIOImbmXxwNngznsa:matrix.org | Linux in Russian
- 18739 !GnEEPYXUhoaHbkFBNX:matrix.org | Riot Android
- 23373 !QtykxKocfZaZOUrTwp:matrix.org | Matrix HQ
- 39504 !gTQfWzbYncrtNrvEkB:matrix.org | ru.[matrix]
- 43601 !iNmaIQExDMeqdITdHH:matrix.org | Riot
- 43601 !iNmaIQExDMeqdITdHH:matrix.org | Riot Web/Desktop
-```
-
-## Lookup room state info by list of room_id
-You get the same information when you use the
-[admin API](../../admin_api/rooms.md#room-details-api).
-```sql
-SELECT rss.room_id, rss.name, rss.canonical_alias, rss.topic, rss.encryption,
- rsc.joined_members, rsc.local_users_in_room, rss.join_rules
- FROM room_stats_state rss
- LEFT JOIN room_stats_current rsc USING (room_id)
- WHERE room_id IN ( WHERE room_id IN (
- '!OGEhHVWSdvArJzumhm:matrix.org',
- '!YTvKGNlinIzlkMTVRl:matrix.org'
- );
-```
-
-## Show users and devices that have not been online for a while
-```sql
-SELECT user_id, device_id, user_agent, TO_TIMESTAMP(last_seen / 1000) AS "last_seen"
- FROM devices
- WHERE last_seen < DATE_PART('epoch', NOW() - INTERVAL '3 month') * 1000;
-```
diff --git a/docs/usage/configuration/config_documentation.md b/docs/usage/configuration/config_documentation.md
index 9add017ca1..20ed41e16a 100644
--- a/docs/usage/configuration/config_documentation.md
+++ b/docs/usage/configuration/config_documentation.md
@@ -427,9 +427,9 @@ Sub-options for each listener include:
* `type`: the type of listener. Normally `http`, but other valid options are:
- * `manhole`: (see the docs [here](../../manhole.md)),
+ * `manhole`: (see the docs [here](../../administration/manhole.md)),
- * `metrics`: (see the docs [here](../../metrics-howto.md)),
+ * `metrics`: (see the docs [here](../../administration/metrics-howto.md)),
* `tls`: set to true to enable TLS for this listener. Will use the TLS key/cert specified in tls_private_key_path / tls_certificate_path.
@@ -438,7 +438,7 @@ Sub-options for each listener include:
* `request_id_header`: The header extracted from each incoming request that is
used as the basis for the request ID. The request ID is used in
- [logs](../administration/request_log.md#request-log-format) and tracing to
+ [logs](../../administration/request_log.md#request-log-format) and tracing to
correlate and match up requests. When unset, Synapse will automatically
generate sequential request IDs. This option is useful when Synapse is behind
a [reverse-proxy](../../setup/reverse_proxy.md).
@@ -468,7 +468,7 @@ Valid resource names are:
* `media`: the media API (/_matrix/media).
-* `metrics`: the metrics interface. See [here](../../metrics-howto.md).
+* `metrics`: the metrics interface. See [here](../../administration/metrics-howto.md).
* `openid`: OpenID authentication. See [here](user_authentication/single_sign_on/openid.md).
@@ -525,7 +525,7 @@ listeners:
### `manhole_settings`
Connection settings for the manhole. You can find more information
-on the manhole [here](../../manhole.md). Manhole sub-options include:
+on the manhole [here](../../administration/manhole.md). Manhole sub-options include:
* `username` : the username for the manhole. This defaults to 'matrix'.
* `password`: The password for the manhole. This defaults to 'rabbithole'.
* `ssh_priv_key_path` and `ssh_pub_key_path`: The private and public SSH key pair used to encrypt the manhole traffic.
@@ -602,7 +602,7 @@ server owner wants to limit to the number of monthly active users. When enabled
reached the server returns a `ResourceLimitError` with error type `Codes.RESOURCE_LIMIT_EXCEEDED`.
Defaults to false. If this is enabled, a value for `max_mau_value` must also be set.
-See [Monthly Active Users](../administration/monthly_active_users.md) for details on how to configure MAU.
+See [Monthly Active Users](../../administration/monthly_active_users.md) for details on how to configure MAU.
Example configuration:
```yaml
@@ -1139,7 +1139,7 @@ number of entries that can be stored.
* `cache_autotuning` and its sub-options `max_cache_memory_usage`, `target_cache_memory_usage`, and
`min_cache_ttl` work in conjunction with each other to maintain a balance between cache memory
- usage and cache entry availability. You must be using [jemalloc](../administration/admin_faq.md#help-synapse-is-slow-and-eats-all-my-ramcpu)
+ usage and cache entry availability. You must be using [jemalloc](../../administration/admin_faq.md#help-synapse-is-slow-and-eats-all-my-ramcpu)
to utilize this option, and all three of the options must be specified for this feature to work. This option
defaults to off, enable it by providing values for the sub-options listed below. Please note that the feature will not work
and may cause unstable behavior (such as excessive emptying of caches or exceptions) if all of the values are not provided.
@@ -2498,7 +2498,7 @@ metrics_flags:
Whether or not to report homeserver usage statistics. This is originally
set when generating the config. Set this option to true or false to change the current
behavior. See
-[Reporting Homeserver Usage Statistics](../administration/monitoring/reporting_homeserver_usage_statistics.md)
+[Reporting Homeserver Usage Statistics](../../administration/reporting_homeserver_usage_statistics.md)
for information on what data is reported.
Statistics will be reported 5 minutes after Synapse starts, and then every 3 hours
|
