summary refs log tree commit diff
path: root/MANIFEST.in (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-09-09More stuffErik Johnston3-33/+37
2022-09-09SNAPSHOTErik Johnston3-1/+239
2022-09-09Use an upsert for `receipts_graph`. (#13752)Patrick Cloke2-8/+5
Instead of a delete, then insert. This was previously done for `receipts_linearized` in 2dc430d36ef793b38d6d79ec8db4ea60588df2ee (#7607).
2022-09-09Add rust to CI (#13763)Erik Johnston3-1/+84
2022-09-09Require SQLite >= 3.27.0 (#13760)David Robertson9-208/+106
2022-09-08Re-type hint some collections in `/sync` code as read-only (#13754)Sean Quah2-10/+11
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Remove unused Prometheus recording rules from `synapse-v2.rules` and add ↵reivilibre2-23/+9
comments describing where the rest are used. (#13756)
2022-09-08Add timestamp to user's consent (#13741)Dirk Klimpel7-8/+52
Co-authored-by: reivilibre <olivier@librepush.net>
2022-09-08Update docstrings to explain the impact of partial state (#13750)Sean Quah2-1/+17
Update the docstrings for `get_users_in_room` and `get_current_hosts_in_room` to explain the impact of partial state. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Avoid raising errors due to malformed IDs in `get_current_hosts_in_room` ↵Sean Quah2-1/+5
(#13748) Handle malformed user IDs with no colons in `get_current_hosts_in_room`. It's not currently possible for a malformed user ID to join a room, so this error would never be hit. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Fix error in `is_mine_id` when encountering a malformed ID (#13746)Sean Quah3-2/+37
Previously, `is_mine_id` would raise an exception when passed an ID with no colons. Return `False` instead. Fixes #13040. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-08Fix cache metrics not being updated when not using the legacy exposition ↵reivilibre4-21/+81
module. (#13717)
2022-09-08Fix Prometheus recording rules to not use legacy metric names. (#13718)reivilibre6-37/+22
2022-09-08Fix latest deps (#13743)Erik Johnston2-0/+15
2022-09-08Fix a bug where Synapse fails to start if a signing key file contains an ↵reivilibre2-1/+13
empty line. (#13738)
2022-09-07Instrument `get_metadata_for_events` for tracing (#13730)Eric Eastwood2-0/+3
When backfilling, `_get_state_ids_after_missing_prev_event` calls [`get_metadata_for_events`](https://github.com/matrix-org/synapse/blob/26bc26586b4b95d63ce7e453e9312469843f796e/synapse/handlers/federation_event.py#L1133). For `#matrix:matrix.org`, it's called with 77k `state_events` which means 77 calls to the database and takes 28 seconds.
2022-09-07Fix latest deps CI (#13734)Erik Johnston3-3/+42
2022-09-07Add some rust caching to CI (#13735)Erik Johnston2-0/+6
2022-09-07Correct out-of-date doc for `event_cache_size` (#13726)David Robertson2-2/+5
2022-09-07Define SQLite compat policy (#13728)David Robertson2-4/+12
2022-09-07A second batch of Pydantic models for rest/client/account.py (#13687)David Robertson4-34/+64
2022-09-07Cancel the processing of key query requests when they time out. (#13680)reivilibre18-20/+110
2022-09-07Rename the `EventFormatVersions` enum values so that they line up with room ↵reivilibre13-41/+47
version numbers. (#13706)
2022-09-07Add Admin API to Fetch Messages Within a Particular Window (#13672)Connor Davis6-14/+435
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
2022-09-06Remove the unspecced room_id field in the /hierarchy response. (#13506)reivilibre2-1/+1
This is a re-do of 57d334a13d983406ea452dfa203bbe4837509c4e (#13365), which was backed out in 12abd724974a2311d5311272d26d2f8aa11734a9 (#13501). The `room_id` field represented the parent space for each room and was made redundant by changes in the API shape where the `children_state` is now nested underneath each `room`. The room ID of each child is in the `state_key` field and is still available.
2022-09-06Add a stub Rust crate (#12595)Erik Johnston23-11/+302
2022-09-06Fix trial-olddeps (#13725)Erik Johnston3-2/+3
2022-09-06Actually fix typechecking with latest types-jsonschema (#13724)David Robertson2-4/+5
2022-09-06Update trial old deps CI to use poetry 1.2.0 (#13707)Erik Johnston3-29/+42
2022-09-06Update Grafana dashboard to not use legacy metric names. (#13714)reivilibre4-73/+74
2022-09-06Remove configuration options for direct TCP replication. (#13647)Patrick Cloke12-78/+63
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-05Reduce number of CI jobs run on PRs (#13713)Erik Johnston3-66/+160
* Reduce number of CI jobs run on PRs * Newsfile * Also limit sytest jobs * Fix typo * Fix up * Fixup
2022-09-05Extend the release script to wait for GitHub Actions to finish and to be ↵reivilibre2-4/+143
usable as a guide for the whole process. (#13483)
2022-09-05Fix typechecking with latest `types-jsonschema` (#13712)David Robertson2-4/+5
2022-09-05Share some metrics between the Prometheus exporter and the phone home stats ↵Brendan Abolivier6-4/+146
(#13671)
2022-09-03fix grammarMatthew Hodgson1-1/+1
2022-09-02Clarify `(room_id, event_id)` global uniqueness (#13701)Eric Eastwood2-0/+25
Summarized from @richvdh's reply at https://github.com/matrix-org/synapse/pull/13589#discussion_r961116999
2022-09-02Update docs to make enabling metrics more clear (#13678)Eric Eastwood2-4/+8
It was really easy to miss the `enable_metrics: True` step with the previous language.
2022-09-02Update lock file for Poetry v1.2.0 (#13689)Erik Johnston5-66/+99
2022-09-02Only run trial CI on all python versions on non-PRs (#13698)Erik Johnston2-0/+11
2022-09-01Fix two typos with colon in headlines (#13665)Dirk Klimpel2-2/+3
2022-09-01Add a schema delta to drop unstable private read receipts. (#13692)Patrick Cloke2-0/+20
Otherwise they'll be leaked due to the filtering code only respecting the stable identifiers for private read receipts.
2022-09-01Disable calculating unread counts unless the config flag is enabled. (#13694)Patrick Cloke4-23/+30
This avoids doing work that will never be used (since the resulting unread counts will never be sent in a /sync response). The negative of doing this is that unread counts will be incorrect when the feature is initially enabled.
2022-09-01Update the Grafana dashboard that is included with Synapse in the `contrib` ↵reivilibre2-20/+128
directory. (#13697) * Add missing graph to contrib * Update with minor but plausible changes, including positioning changes * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-09-01Cache `is_partial_state_room` (#13693)Erik Johnston2-4/+8
Fixes #13613.
2022-09-01Add monthly active users documentation (#13617)Will Hunt4-0/+88
* Add monthly active users documentation * changelog * Tidy up notes * more tidyup * Rewrite #1 * link back to mau docs * fix links * s/appservice|AS/application service * further review * a newline * Remove bit about shadow banned users. I think talking about them is confusing, and the current text doesn't imply they get any special treatment. * Update docs/usage/administration/monthly_active_users.md Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update docs/usage/administration/monthly_active_users.md Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-01Use poetry 1.2.0 rather than prerelease for deb packages (#13695)Erik Johnston2-1/+7
2022-09-01Add some logging to help track down #13444 (#13679)Erik Johnston2-0/+14
2022-09-01Return keys for unwhitelisted servers from `/_matrix/key/v2/query` (#13683)Richard van der Hoff2-20/+22
2022-09-01Bump docker image to use stable poetry version (#13688)Erik Johnston3-11/+2
2022-09-01Remove support for unstable private read receipts (#13653)Šimon Brandner13-122/+44
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-08-31Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without ↵Jacek Kuśnierz9-137/+81
an `id_access_token` (#13241) Fixes #13206 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-08-31Tweak changelog formatting v1.66.0Sean Quah1-6/+4
2022-08-31Move notice from 1.66.0rc1 to 1.66.0 section in changelogSean Quah1-6/+6
2022-08-31Tweak changelog wordingSean Quah1-4/+5
2022-08-31Improve clarity on deprecation of TCP replicationSean Quah2-2/+10
Borrows some text from https://github.com/matrix-org/synapse/pull/13647 for the changelog.
2022-08-31Remove cached wrap on `_get_joined_users_from_context` method (#13569)Nick Mills-Barrett3-85/+40
The method doesn't actually do any data fetching and the method that does, `_get_joined_profile_from_event_id`, has its own cache. Signed off by Nick @ Beeper (@Fizzadar).
2022-08-31Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre10-75/+75
other than just servlet methods. (#13662)
2022-08-31Fix dead link in 1.18.0 upgrade notesSean Quah1-1/+1
2022-08-31Fix admin List Room API return type on sqlite (#13509)David Robertson3-7/+19
2022-08-31Update changelog to link to the Synapse docs instead of markdownSean Quah1-1/+1
2022-08-311.66.0Sean Quah3-3/+12
2022-08-31Update changelog for v1.62.0Sean Quah1-0/+14
2022-08-31Change dpkg-statoverride to use --force-statoverride-add (#13638)Jörg Behrmann2-1/+2
The --force flag of dpkg-statoverride has been deprecated (apparently starting with the dpkg version in Debian buster). It offers --force-all as q quick fix, but the usage in the Debian postinst script is probably covered by --force-statoverride-add. Fixes: #8391 Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-08-31Remind people that direct TCP replication is disabled. (#13674)Patrick Cloke1-0/+8
2022-08-30Give the correct next event when the message timestamps are the same - ↵Eric Eastwood2-2/+11
MSC3030 (#13658) Discovered while working on https://github.com/matrix-org/synapse/pull/13589 and I had all the messages at the same timestamp in the tests. Part of https://github.com/matrix-org/matrix-spec-proposals/pull/3030 Complement tests: https://github.com/matrix-org/complement/pull/457
2022-08-30Drop unused column `application_services_state.last_txn` (#13627)Shay4-0/+59
2022-08-30Remove documentation of legacy `frontend_proxy` worker app (#13645)Richard van der Hoff3-41/+23
This has been the same as a generic_worker since #6964, so let's get rid of it. Fixes #3717
2022-08-30Update changes v1.66.0rc2David Robertson1-1/+1
2022-08-30Clarify documentation about replication traffic. (#13656)Patrick Cloke2-1/+5
It can be authenticated with the worker_replication_secret setting, but is always unencrypted.
2022-08-301.66.0rc2David Robertson4-2/+16
2022-08-30Fix bug where we wedge media plugins if clients disconnect early (#13660)Erik Johnston2-19/+22
We incorrectly didn't use the returned `Responder` if the client had disconnected, which meant that the resource used by the Responder wasn't correctly released. In particular, this exhausted the thread pools so that *all* requests timed out.
2022-08-30Do not wait for background updates to complete do expire URL cache. (#13657)Patrick Cloke2-4/+1
Media downloaded as part of a URL preview is normally deleted after two days. However, while a background database migration is running, the process is stopped. A long-running database migration can therefore cause the media store to fill up with old preview files. This logic was added in #2697 to make sure that we didn't try to run the expiry without an index on `local_media_repository.created_ts`; the original logic that needs that index was added in #2478 (in `get_url_cache_media_before`, as amended by 93247a424a5068b088567fa98b6990e47608b7cb), and is still present. Given that the background update was added before Synapse v1.0.0, just drop this check and assume the index exists.
2022-08-30Speed up inserting `event_push_actions_staging`. (#13634)Patrick Cloke2-20/+9
By using `execute_values` instead of `execute_batch`.
2022-08-30Fix rate limit metrics registering twice and misreporting (#13649)Eric Eastwood3-32/+128
* Fix rate limit metrics registering twice and misreporting Fix https://github.com/matrix-org/synapse/issues/13641 * Fix lints * Add changelog * Document `metrics_name=None`.
2022-08-30Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel4-6/+99
2022-08-30Optimize how we calculate `likely_domains` during backfill (#13575)Eric Eastwood5-70/+89
Optimize how we calculate `likely_domains` during backfill because I've seen this take 17s in production just to `get_current_state` which is used to `get_domains_from_state` (see case [*2. Loading tons of events* in the `/messages` investigation issue](https://github.com/matrix-org/synapse/issues/13356)). There are 3 ways we currently calculate hosts that are in the room: 1. `get_current_state` -> `get_domains_from_state` - Used in `backfill` to calculate `likely_domains` and `/timestamp_to_event` because it was cargo-culted from `backfill` - This one is being eliminated in favor of `get_current_hosts_in_room` in this PR 🕳 1. `get_current_hosts_in_room` - Used for other federation things like sending read receipts and typing indicators 1. `get_hosts_in_room_at_events` - Used when pushing out events over federation to other servers in the `_process_event_queue_loop` Fix https://github.com/matrix-org/synapse/issues/13626 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh) ### Query performance #### Before The query from `get_current_state` sucks just because we have to get all 80k events. And we see almost the exact same performance locally trying to get all of these events (16s vs 17s): ``` synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; Time: 16035.612 ms (00:16.036) synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; Time: 4243.237 ms (00:04.243) ``` But what about `get_current_hosts_in_room`: When there is 8M rows in the `current_state_events` table, the previous query in `get_current_hosts_in_room` took 13s from complete freshness (when the events were first added). But takes 930ms after a Postgres restart or 390ms if running back to back to back. ```sh $ psql synapse synapse=# \timing on synapse=# SELECT COUNT(DISTINCT substring(state_key FROM '@[^:]*:(.*)$')) FROM current_state_events WHERE type = 'm.room.member' AND membership = 'join' AND room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; count ------- 4130 (1 row) Time: 13181.598 ms (00:13.182) synapse=# SELECT COUNT(*) from current_state_events where room_id = '!OGEhHVWSdvArJzumhm:matrix.org'; count ------- 80814 synapse=# SELECT COUNT(*) from current_state_events; count --------- 8162847 synapse=# SELECT pg_size_pretty( pg_total_relation_size('current_state_events') ); pg_size_pretty ---------------- 4702 MB ``` #### After I'm not sure how long it takes from complete freshness as I only really get that opportunity once (maybe restarting computer but that's cumbersome) and it's not really relevant to normal operating times. Maybe you get closer to the fresh times the more access variability there is so that Postgres caches aren't as exact. Update: The longest I've seen this run for is 6.4s and 4.5s after a computer restart. After a Postgres restart, it takes 330ms and running back to back takes 260ms. ```sh $ psql synapse synapse=# \timing on Timing is on. synapse=# SELECT substring(c.state_key FROM '@[^:]*:(.*)$') as host FROM current_state_events c /* Get the depth of the event from the events table */ INNER JOIN events AS e USING (event_id) WHERE c.type = 'm.room.member' AND c.membership = 'join' AND c.room_id = '!OGEhHVWSdvArJzumhm:matrix.org' GROUP BY host ORDER BY min(e.depth) ASC; Time: 333.800 ms ``` #### Going further To improve things further we could add a `limit` parameter to `get_current_hosts_in_room`. Realistically, we don't need 4k domains to choose from because there is no way we're going to query that many before we a) probably get an answer or b) we give up. Another thing we can do is optimize the query to use a index skip scan: - https://wiki.postgresql.org/wiki/Loose_indexscan - Index Skip Scan, https://commitfest.postgresql.org/37/1741/ - https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/
2022-08-28Print complement failure results last (#13639)Richard van der Hoff6-11/+31
Since github always scrolls to the bottom of any test output, let's put the failed tests last and hide any successful packages.
2022-08-26Improve documentation around user registration (#13640)Richard van der Hoff4-103/+134
Update a bunch of the documentation for user registration, add some cross links, etc.
2022-08-26Generate missing configuration files at startup (#13615)Richard van der Hoff3-12/+56
If things like the signing key file are missing, let's just try to generate them on startup. Again, this is useful for k8s-like deployments where we just want to generate keys on the first run.
2022-08-26Update debhelper (#13594)Jörg Behrmann7-41/+15
* Update debian packaging to debhelper version 12 Don't call dh_installinit anymore, because it has been deprecated, and use dh_installsystemd instead of dh_systemd_enable for the same reason. Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de> * Drop preinst script It was used for reasons of interactions of dh_systemd_start and dh_installinit, which have both be deprecated Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de> * Drop /etc/default file It was no longer being installed. * Remove debian/compat file This is managed by the control file nowadays
2022-08-26Move the execution of the retention purge_jobs to the main worker (#13632)Brad Murray2-4/+3
Fixes #9927 Signed-off-by: Brad Murray brad@beeper.com
2022-08-25Debian packaging: explicitly allocate a group for the system user (#13593)Jörg Behrmann2-2/+8
Otherwise the files of the synapse user are readable by the nobody user, which is unsafe. Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-08-25Support `registration_shared_secret` in a file (#13614)Richard van der Hoff4-5/+92
A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
2022-08-25register_new_matrix_user: read server url from config (#13616)Richard van der Hoff2-6/+52
Fixes https://github.com/matrix-org/synapse/issues/3672: `https://localhost:8448` is virtually never right.
2022-08-25Update automation for incoming issues (#13629)Kat Gerasimova1-4/+4
GitHub appears to be deprecating addProjectNextItem by not allowing it to be used alongside projectV2 to get the project ID, so switching to using addProjectV2ItemById instead.
2022-08-24Comment about a better future where we can get the state diff between two ↵Eric Eastwood2-0/+9
events (#13586) Split off from https://github.com/matrix-org/synapse/pull/13561 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh)
2022-08-24Rename `event_map` to `unpersisted_events` (#13603)David Robertson2-32/+38
2022-08-24Update `get_users_in_room` mis-use to get hosts with dedicated ↵Eric Eastwood6-17/+31
`get_current_hosts_in_room` (#13605) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24Directly lookup local membership instead of getting all members in a room ↵Eric Eastwood8-17/+60
first (`get_users_in_room` mis-use) (#13608) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24When loading current ids, sort by `stream_id` to avoid incorrect overwrite ↵Eric Eastwood2-2/+12
and avoid errors caused by sorting alphabetical instance name which can be `null` (#13585) When loading current ids, sort by stream ID so that we don't want to overwrite the `current_position` of an instance to a lower stream ID than we're actually at ([discussion](https://github.com/matrix-org/synapse/pull/13585#discussion_r951795379)). Previously, it sorted alphabetically by instance name which can be `null` and throw errors but more importantly, accomplishes nothing. Fixes the following startup error which is why I started looking into this area: ``` $ poetry run synapse_homeserver --config-path homeserver.yaml **************************************************************** Error during initialisation: '<' not supported between instances of 'NoneType' and 'str' There may be more information in the logs. **************************************************************** ``` Somehow my database ended up looking like the following, notice the `instance_name` is `null` in the db, and we can't sort `NoneType` things. Another question is why do we see the `instance_name` as `null` sometimes instead of `master` in monolith mode? ``` $ psql synapse synapse=# SELECT * FROM stream_positions; stream_name | instance_name | stream_id -----------------+---------------+----------- account_data | master | 1242 events | master | 1787 to_device | master | 58 presence_stream | master | 485638 receipts | master | 341 backfill | master | -139106 (6 rows) synapse=# SELECT instance_name, stream_id FROM receipts_linearized; instance_name | stream_id ---------------+----------- | 211 | 3 | 4 | 212 | 213 | 224 | 228 |