summary refs log tree commit diff
path: root/changelog.d/5171.misc (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-09-16Try caching blackDavid Robertson1-1/+1
2022-09-16Add an admin API endpoint to find a user based on its external ID in an auth ↵Quentin Gliech5-0/+155
provider. (#13810)
2022-09-16Update request log format documentation to mention the format used when the ↵reivilibre2-2/+3
authenticated user is controlling another user. (#13794)
2022-09-16Avoid putting rejected events in room state (#13723)Sean Quah3-0/+415
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-15Document common fix of Poetry problems by removing egg-info (#13785)Eric Eastwood2-0/+28
`matrix_synapse.egg-info/` Mentioned at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$aKy_IjrKwb70aTVZWeW_6zt0k7OIZ1YkyZpkP9uiRaM?via=matrix.org&via=element.io&via=beeper.com and many other places.
2022-09-15Be able to correlate timeouts in reverse-proxy layer in front of Synapse ↵Eric Eastwood5-5/+38
(pull request ID from header) (#13801) Fix https://github.com/matrix-org/synapse/issues/13685 New config: ```diff listeners: - port: 8008 tls: false type: http x_forwarded: true + request_id_header: "cf-ray" bind_addresses: ['::1', '127.0.0.1', '0.0.0.0'] ```
2022-09-15Record any exception when processing a pulled event (#13814)Eric Eastwood3-1/+12
Part of https://github.com/matrix-org/synapse/issues/13700 and https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13589
2022-09-15Support providing an index predicate for upserts. (#13822)Patrick Cloke3-7/+25
This is useful to upsert against a table which has a unique partial index while avoiding conflicts.
2022-09-15A third batch of Pydantic validation for rest/client/account.py (#13736)David Robertson4-45/+78
2022-09-15Add a `MXCUri` class to make working with mxc uri's easier. (#13162)Andrew Morgan6-74/+53
2022-09-14Keep track when we try and fail to process a pulled event (#13589)Eric Eastwood7-9/+329
We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
2022-09-14Update event push action and receipt tables to support threads. (#13753)Patrick Cloke11-20/+312
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` tables. This will notifications to be segmented by the thread in a future pull request. The `thread_id` column stores the root event ID or the special value `"main"`. The `thread_id` column for `event_push_actions` and `event_push_summary` is backfilled with `"main"` for all existing rows. New entries into `event_push_actions` and `event_push_actions_staging` will get the proper thread ID. `receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar, except `NULL` is a special value meaning the receipt is "unthreaded". See MSC3771 and MSC3773 for where this data will be useful.
2022-09-14Use partial indices on SQLIte. (#13802)Patrick Cloke4-5/+59
Partial indices have been supported since SQLite 3.8, but Synapse now requires >= 3.27, so we can enable support for them. This requires rebuilding previous indices which were partial on PostgreSQL, but not on SQLite.
2022-09-14Deduplicate `is_server_notices_room`. (#13780)reivilibre4-18/+20
2022-09-14Fix a memory leak when running the unit tests. (#13798)reivilibre3-6/+8
2022-09-14Remove unused method in `synapse.api.auth.Auth`. (#13795)Quentin Gliech2-9/+1
Clean-up from b19060a29b4f73897847db2aba5d03ec819086e0 (#13094) and 73af10f419346a5f2d70131ac1ed8e69942edca0 (#13093) which removed all callers.
2022-09-14Remove incorrect migration file from `state` logical DB (#13788)David Robertson2-37/+1
* Remove incorrect migration file from `state` logical DB The table `ex_outlier_stream` is part of the `main` logical DB; it should not have been created in the `state` logical DB. We remove this migration now as a tidy-up. Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new migration, because some (most) instances of Synapse host both of these logical DBs on the same DB cluster. * Changelog
2022-09-14Fix bug in device list caching when remote users leave rooms (#13749)Sean Quah5-15/+51
When a remote user leaves the last room shared with the homeserver, we have to mark their device list as unsubscribed, otherwise we would hold on to a stale device list in our cache. Crucially, the device list would remain cached even after the remote user rejoined the room, which could lead to E2EE failures until the next change to the remote user's device list. Fixes #13651. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14Fix a long-standing spec compliance bug where Synapse would accept a ↵reivilibre2-2/+2
trailing slash on the end of `/get_missing_events` federation requests. (#13789) * Don't accept a trailing slash on the end of /get_missing_events * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-09-13Make sequence `cache_invalidation_stream_seq` begin at `2` (#13766)Mathieu Velten3-0/+25
Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-09-13Updates to the schema dump script (#13770)David Robertson3-32/+21
2022-09-13Simplify CI tests DAG (#13784)David Robertson2-4/+1
* Simplify CI tests DAG * Changelog
2022-09-13installation.md: require libpq on M1 macs (#13480)Richard van der Hoff2-2/+4
2022-09-13Fixup changelog v1.67.0Erik Johnston1-3/+3
2022-09-131.67.0Erik Johnston3-3/+15
2022-09-13Add receipts event stream ordering (#13703)Nick Mills-Barrett4-1/+95
2022-09-12Fix GHA skippable syntax (#13778)Mathieu Velten2-1/+3
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-12Fix typo in ratelimiting documentation (#13727)Brendan Abolivier2-1/+2
2022-09-12Fix CI on non-PR builds (#13769)Erik Johnston2-0/+2
Mark cargo-test as skippable since it only runs on Rust code change.
2022-09-12Remove check current state membership up to date (#13745)Nick Mills-Barrett3-155/+100
* Remove checks for membership column in current_state_events * Add schema script to force through the `current_state_events_membership` background job Contributed by Nick @ Beeper (@fizzadar).
2022-09-12Check if Rust lib needs rebuilding. (#13759)Erik Johnston7-1/+149
This protects against the common mistake of failing to remember to rebuild Rust code after making changes.
2022-09-09Concurrently collect room unread counts for push badges (#13765)Nick Mills-Barrett2-3/+11
Most of the time this function is heavily cached, but when that isn't the case fetching the counts room by room slows down push delivery on users with many (thousands) of rooms. Signed off by Nick @ Beeper.
2022-09-09Tag trace with instance name (#13761)Eric Eastwood3-2/+12
We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load. As suggested by @clokep and @reivilibre in, - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
2022-09-09Strip number suffix from instance name to consolidate services that traces ↵Eric Eastwood2-1/+13
are spread over (#13729) The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985 Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through. With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇 Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.
2022-09-09Use an upsert for `receipts_graph`. (#13752)Patrick Cloke2-8/+5
Instead of a delete, then insert. This was previously done for `receipts_linearized` in 2dc430d36ef793b38d6d79ec8db4ea60588df2ee (#7607).
2022-09-09Add rust to CI (#13763)Erik Johnston3-1/+84
2022-09-09Require SQLite >= 3.27.0 (#13760)David Robertson9-208/+106
2022-09-08Re-type hint some collections in `/sync` code as read-only (#13754)Sean Quah2-10/+11
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Remove unused Prometheus recording rules from `synapse-v2.rules` and add ↵reivilibre2-23/+9
comments describing where the rest are used. (#13756)
2022-09-08Add timestamp to user's consent (#13741)Dirk Klimpel7-8/+52
Co-authored-by: reivilibre <olivier@librepush.net>
2022-09-08Update docstrings to explain the impact of partial state (#13750)Sean Quah2-1/+17
Update the docstrings for `get_users_in_room` and `get_current_hosts_in_room` to explain the impact of partial state. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Avoid raising errors due to malformed IDs in `get_current_hosts_in_room` ↵Sean Quah2-1/+5
(#13748) Handle malformed user IDs with no colons in `get_current_hosts_in_room`. It's not currently possible for a malformed user ID to join a room, so this error would never be hit. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Fix error in `is_mine_id` when encountering a malformed ID (#13746)Sean Quah3-2/+37
Previously, `is_mine_id` would raise an exception when passed an ID with no colons. Return `False` instead. Fixes #13040. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Fix cache metrics not being updated when not using the legacy exposition ↵reivilibre4-21/+81
module. (#13717)
2022-09-08Fix Prometheus recording rules to not use legacy metric names. (#13718)reivilibre6-37/+22
2022-09-08Fix latest deps (#13743)Erik Johnston2-0/+15
2022-09-08Also cite upgrade notesDavid Robertson1-1/+1
2022-09-08Add minimum version bump for sqlite to the release notes (#13742)David Robertson2-1/+21
* Notify that SQLite min version will be bumped * Mention in upgrade notes Co-authored-by: reivilibre <oliverw@matrix.org>
2022-09-08Fix a bug where Synapse fails to start if a signing key file contains an ↵reivilibre2-1/+13
empty line. (#13738)
2022-09-07Instrument `get_metadata_for_events` for tracing (#13730)Eric Eastwood2-0/+3
When backfilling, `_get_state_ids_after_missing_prev_event` calls [`get_metadata_for_events`](https://github.com/matrix-org/synapse/blob/26bc26586b4b95d63ce7e453e9312469843f796e/synapse/handlers/federation_event.py#L1133). For `#matrix:matrix.org`, it's called with 77k `state_events` which means 77 calls to the database and takes 28 seconds.
2022-09-07Fix latest deps CI (#13734)Erik Johnston3-3/+42
2022-09-07Add some rust caching to CI (#13735)Erik Johnston2-0/+6
2022-09-07Correct out-of-date doc for `event_cache_size` (#13726)David Robertson2-2/+5
2022-09-07Define SQLite compat policy (#13728)David Robertson2-4/+12
2022-09-07A second batch of Pydantic models for rest/client/account.py (#13687)David Robertson4-34/+64
2022-09-07Cancel the processing of key query requests when they time out. (#13680)reivilibre18-20/+110
2022-09-07Rename the `EventFormatVersions` enum values so that they line up with room ↵reivilibre13-41/+47
version numbers. (#13706)
2022-09-07Add Admin API to Fetch Messages Within a Particular Window (#13672)Connor Davis6-14/+435
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
2022-09-06Remove the unspecced room_id field in the /hierarchy response. (#13506)reivilibre2-1/+1
This is a re-do of 57d334a13d983406ea452dfa203bbe4837509c4e (#13365), which was backed out in 12abd724974a2311d5311272d26d2f8aa11734a9 (#13501). The `room_id` field represented the parent space for each room and was made redundant by changes in the API shape where the `children_state` is now nested underneath each `room`. The room ID of each child is in the `state_key` field and is still available.
2022-09-06Add a stub Rust crate (#12595)Erik Johnston23-11/+302
2022-09-06Fix trial-olddeps (#13725)Erik Johnston3-2/+3
2022-09-06Actually fix typechecking with latest types-jsonschema (#13724)David Robertson2-4/+5
2022-09-06Fixup changelog v1.67.0rc1Erik Johnston1-2/+2
2022-09-06Update trial old deps CI to use poetry 1.2.0 (#13707)Erik Johnston3-29/+42
2022-09-06Update Grafana dashboard to not use legacy metric names. (#13714)reivilibre4-73/+74
2022-09-06Update changelogErik Johnston2-0/+22
2022-09-061.67.0rc1Erik Johnston56-56/+91
2022-09-06Remove configuration options for direct TCP replication. (#13647)Patrick Cloke12-78/+63
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-05Reduce number of CI jobs run on PRs (#13713)Erik Johnston3-66/+160
* Reduce number of CI jobs run on PRs * Newsfile * Also limit sytest jobs * Fix typo * Fix up * Fixup
2022-09-05Extend the release script to wait for GitHub Actions to finish and to be ↵reivilibre2-4/+143
usable as a guide for the whole process. (#13483)
2022-09-05Fix typechecking with latest `types-jsonschema` (#13712)David Robertson2-4/+5
2022-09-05Share some metrics between the Prometheus exporter and the phone home stats ↵Brendan Abolivier6-4/+146
(#13671)
2022-09-03fix grammarMatthew Hodgson1-1/+1
2022-09-02Clarify `(room_id, event_id)` global uniqueness (#13701)Eric Eastwood2-0/+25
Summarized from @richvdh's reply at https://github.com/matrix-org/synapse/pull/13589#discussion_r961116999
2022-09-02Update docs to make enabling metrics more clear (#13678)Eric Eastwood2-4/+8
It was really easy to miss the `enable_metrics: True` step with the previous language.
2022-09-02Update lock file for Poetry v1.2.0 (#13689)Erik Johnston5-66/+99
2022-09-02Only run trial CI on all python versions on non-PRs (#13698)Erik Johnston2-0/+11
2022-09-01Fix two typos with colon in headlines (#13665)Dirk Klimpel2-2/+3
2022-09-01Add a schema delta to drop unstable private read receipts. (#13692)Patrick Cloke2-0/+20
Otherwise they'll be leaked due to the filtering code only respecting the stable identifiers for private read receipts.
2022-09-01Disable calculating unread counts unless the config flag is enabled. (#13694)Patrick Cloke4-23/+30
This avoids doing work that will never be used (since the resulting unread counts will never be sent in a /sync response). The negative of doing this is that unread counts will be incorrect when the feature is initially enabled.
2022-09-01Update the Grafana dashboard that is included with Synapse in the `contrib` ↵reivilibre2-20/+128
directory. (#13697) * Add missing graph to contrib * Update with minor but plausible changes, including positioning changes * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-09-01Cache `is_partial_state_room` (#13693)Erik Johnston2-4/+8
Fixes #13613.
2022-09-01Add monthly active users documentation (#13617)Will Hunt4-0/+88
* Add monthly active users documentation * changelog * Tidy up notes * more tidyup * Rewrite #1 * link back to mau docs * fix links * s/appservice|AS/application service * further review * a newline * Remove bit about shadow banned users. I think talking about them is confusing, and the current text doesn't imply they get any special treatment. * Update docs/usage/administration/monthly_active_users.md Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update docs/usage/administration/monthly_active_users.md Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-01Use poetry 1.2.0 rather than prerelease for deb packages (#13695)Erik Johnston2-1/+7
2022-09-01Add some logging to help track down #13444 (#13679)Erik Johnston2-0/+14
2022-09-01Return keys for unwhitelisted servers from `/_matrix/key/v2/query` (#13683)Richard van der Hoff2-20/+22
2022-09-01Bump docker image to use stable poetry version (#13688)Erik Johnston3-11/+2
2022-09-01Remove support for unstable private read receipts (#13653)Šimon Brandner13-122/+44
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-08-31Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without ↵Jacek Kuśnierz9-137/+81
an `id_access_token` (#13241) Fixes #13206 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-08-31Tweak changelog formatting v1.66.0Sean Quah1-6/+4
2022-08-31Move notice from 1.66.0rc1 to 1.66.0 section in changelogSean Quah1-6/+6
2022-08-31Tweak changelog wordingSean Quah1-4/+5
2022-08-31Improve clarity on deprecation of TCP replicationSean Quah2-2/+10
Borrows some text from https://github.com/matrix-org/synapse/pull/13647 for the changelog.
2022-08-31Remove cached wrap on `_get_joined_users_from_context` method (#13569)Nick Mills-Barrett3-85/+40
The method doesn't actually do any data fetching and the method that does, `_get_joined_profile_from_event_id`, has its own cache. Signed off by Nick @ Beeper (@Fizzadar).
2022-08-31Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre10-75/+75
other than just servlet methods. (#13662)
2022-08-31Fix dead link in 1.18.0 upgrade notesSean Quah1-1/+1
2022-08-31Fix admin List Room API return type on sqlite (#13509)David Robertson3-7/+19
2022-08-31Update changelog to link to the Synapse docs instead of markdownSean Quah1-1/+1
2022-08-311.66.0Sean Quah3-3/+12
2022-08-31Update changelog for v1.62.0Sean Quah1-0/+14
2022-08-31Change dpkg-statoverride to use --force-statoverride-add (#13638)Jörg Behrmann2-1/+2
The --force flag of dpkg-statoverride has been deprecated (apparently starting with the dpkg version in Debian buster). It offers --force-all as q quick fix, but the usage in the Debian postinst script is probably covered by --force-statoverride-add. Fixes: #8391 Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-08-31Remind people that direct TCP replication is disabled. (#13674)Patrick Cloke1-0/+8
2022-08-30Give the correct next event when the message timestamps are the same - ↵Eric Eastwood2-2/+11
MSC3030 (#13658) Discovered while working on https://github.com/matrix-org/synapse/pull/13589 and I had all the messages at the same timestamp in the tests. Part of https://github.com/matrix-org/matrix-spec-proposals/pull/3030 Complement tests: https://github.com/matrix-org/complement/pull/457
2022-08-30Drop unused column `application_services_state.last_txn` (#13627)Shay4-0/+59
2022-08-30Remove documentation of legacy `frontend_proxy` worker app (#13645)Richard van der Hoff3-41/+23
This has been the same as a generic_worker since #6964, so let's get rid of it. Fixes #3717
2022-08-30Update changes v1.66.0rc2David Robertson1-1/+1
2022-08-30Clarify documentation about replication traffic. (#13656)Patrick Cloke2-1/+5
It can be authenticated with the worker_replication_secret setting, but is always unencrypted.
2022-08-301.66.0rc2David Robertson4-2/+16
2022-08-30Fix bug where we wedge media plugins if clients disconnect early (#13660)Erik Johnston2-19/+22
We incorrectly didn't use the returned `Responder` if the client had disconnected, which meant that the resource used by the Responder wasn't correctly released. In particular, this exhausted the thread pools so that *all* requests timed out.
2022-08-30Do not wait for background updates to complete do expire URL cache. (#13657)Patrick Cloke2-4/+1
Media downloaded as part of a URL preview is normally deleted after two days. However, while a background database migration is running, the process is stopped. A long-running database migration can therefore cause the media store to fill up with old preview files. This logic was added in #2697 to make sure that we didn't try to run the expiry without an index on `local_media_repository.created_ts`; the original logic that needs that index was added in #2478 (in `get_url_cache_media_before`, as amended by 93247a424a5068b088567fa98b6990e47608b7cb), and is still present. Given that the background update was added before Synapse v1.0.0, just drop this check and assume the index exists.
2022-08-30Speed up inserting `event_push_actions_staging`. (#13634)Patrick Cloke2-20/+9
By using `execute_values` instead of `execute_batch`.
2022-08-30Fix rate limit metrics registering twice and misreporting (#13649)Eric Eastwood3-32/+128
* Fix rate limit metrics registering twice and misreporting Fix https://github.com/matrix-org/synapse/issues/13641 * Fix lints * Add changelog * Document `metrics_name=None`.
2022-08-30Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel4-6/+99
2022-08-30Optimize how we calculate `likely_domains` during backfill (#13575)Eric Eastwood5-70/+89
Optimize how we calculate `likely_domains` during backfill because I've seen this take 17s in production just to `get_current_state` which is used to `get_domains_from_state` (see case [*2. Loading tons of events* in the `/messages` investigation issue](https://github.com/matrix-org/synapse/issues/13356)). There are 3 ways we currently calculate hosts that are in the room: 1. `get_current_state` -> `get_domains_from_state` - Used in `backfill` to calculate `likely_domains` and `/timestamp_to_event` because it was cargo-culted from `backfill` - This one is being eliminated in favor of `get_current_hosts_in_room` in this PR 🕳 1. `get_current_hosts_in_room` - Used for other federation things like sending read receipts and typing indicators 1. `get_hosts_in_room_at_events` - Used when pushing out events over federation to other servers in the `_process_event_queue_loop` Fix https://github.com/matrix-org/synapse/issues/13626 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh) ### Query performance #### Before The query from `get_current_state` sucks just because we have to get all 80k events. And we see almost the exact same performance locally trying to get all of these events (16s vs 17s): ``` synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; Time: 16035.612 ms (00:16.036) synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; Time: 4243.237 ms (00:04.243) ``` But what about `get_current_hosts_in_room`: When there is 8M rows in the `current_state_events` table, the previous query in `get_current_hosts_in_room` took 13s from complete freshness (when the events were first added). But takes 930ms after a Postgres restart or 390ms if running back to back to back. ```sh $ psql synapse synapse=# \timing on synapse=# SELECT COUNT(DISTINCT substring(state_key FROM '@[^:]*:(.*)$')) FROM current_state_events WHERE type = 'm.room.member' AND membership = 'join' AND room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; count ------- 4130 (1 row) Time: 13181.598 ms (00:13.182) synapse=# SELECT COUNT(*) from current_state_events where room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; count ------- 80814 synapse=# SELECT COUNT(*) from current_state_events; count --------- 8162847 synapse=# SELECT pg_size_pretty( pg_total_relation_size('current_state_events') ); pg_size_pretty ---------------- 4702 MB ``` #### After I'm not sure how long it takes from complete freshness as I only really get that opportunity once (maybe restarting computer but that's cumbersome) and it's not really relevant to normal operating times. Maybe you get closer to the fresh times the more access variability there is so that Postgres caches aren't as exact. Update: The longest I've seen this run for is 6.4s and 4.5s after a computer restart. After a Postgres restart, it takes 330ms and running back to back takes 260ms. ```sh $ psql synapse synapse=# \timing on Timing is on. synapse=# SELECT substring(c.state_key FROM '@[^:]*:(.*)$') as host FROM current_state_events c /* Get the depth of the event from the events table */ INNER JOIN events AS e USING (event_id) WHERE c.type = 'm.room.member' AND c.membership = 'join' AND c.room_id = '!OGEhHVWSdvArJzumhm:matrix.org' GROUP BY host ORDER BY min(e.depth) ASC; Time: 333.800 ms ``` #### Going further To improve things further we could add a `limit` parameter to `get_current_hosts_in_room`. Realistically, we don't need 4k domains to choose from because there is no way we're going to query that many before we a) probably get an answer or b) we give up. Another thing we can do is optimize the query to use a index skip scan: - https://wiki.postgresql.org/wiki/Loose_indexscan - Index Skip Scan, https://commitfest.postgresql.org/37/1741/ - https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/
2022-08-28Print complement failure results last (#13639)Richard van der Hoff6-11/+31
Since github always scrolls to the bottom of any test output, let's put the failed tests last and hide any successful packages.
2022-08-26Improve documentation around user registration (#13640)Richard van der Hoff4-103/+134
Update a bunch of the documentation for user registration, add some cross links, etc.
2022-08-26Generate missing configuration files at startup (#13615)Richard van der Hoff3-12/+56
If things like the signing key file are missing, let's just try to generate them on startup. Again, this is useful for k8s-like deployments where we just want to generate keys on the first run.
2022-08-26Update debhelper (#13594)Jörg Behrmann7-41/+15
* Update debian packaging to debhelper version 12 Don't call dh_installinit anymore, because it has been deprecated, and use dh_installsystemd instead of dh_systemd_enable for the same reason. Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de> * Drop preinst script It was used for reasons of interactions of dh_systemd_start and dh_installinit, which have both be deprecated Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de> * Drop /etc/default file It was no longer being installed. * Remove debian/compat file This is managed by the control file nowadays
2022-08-26Move the execution of the retention purge_jobs to the main worker (#13632)Brad Murray2-4/+3
Fixes #9927 Signed-off-by: Brad Murray brad@beeper.com
2022-08-25Debian packaging: explicitly allocate a group for the system user (#13593)Jörg Behrmann2-2/+8
Otherwise the files of the synapse user are readable by the nobody user, which is unsafe. Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-08-25Support `registration_shared_secret` in a file (#13614)Richard van der Hoff4-5/+92
A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
2022-08-25register_new_matrix_user: read server url from config (#13616)Richard van der Hoff2-6/+52
Fixes https://github.com/matrix-org/synapse/issues/3672: `https://localhost:8448` is virtually never right.
2022-08-25Update automation for incoming issues (#13629)Kat Gerasimova1-4/+4
GitHub appears to be deprecating addProjectNextItem by not allowing it to be used alongside projectV2 to get the project ID, so switching to using addProjectV2ItemById instead.
2022-08-24Comment about a better future where we can get the state diff between two ↵Eric Eastwood2-0/+9
events (#13586) Split off from https://github.com/matrix-org/synapse/pull/13561 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh)
2022-08-24Rename `event_map` to `unpersisted_events` (#13603)David Robertson2-32/+38
2022-08-24Update `get_users_in_room` mis-use to get hosts with dedicated ↵Eric Eastwood6-17/+31
`get_current_hosts_in_room` (#13605) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24Directly lookup local membership instead of getting all members in a room ↵Eric Eastwood8-17/+60
first (`get_users_in_room` mis-use) (#13608) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24When loading current ids, sort by `stream_id` to avoid incorrect overwrite ↵Eric Eastwood2-2/+12
and avoid errors caused by sorting alphabetical instance name which can be `null` (#13585) When loading current ids, sort by stream ID so that we don't want to overwrite the `current_position` of an instance to a lower stream ID than we're actually at ([discussion](https://github.com/matrix-org/synapse/pull/13585#discussion_r951795379)). Previously, it sorted alphabeticall