summary refs log tree commit diff
path: root/docs/replication.md (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-10-24Fixup broken linksDavid Robertson13-15/+15
``` dmr on titan in synapse/book on  dmr/docs-tidy via  v16.14.0 via 🐍 v3.10.7 (matrix-synapse-py3.10) via 🐏 12GiB/15GiB | 5GiB/8GiB took 8s 2022-10-24 15:15:53 ✔ $ linkchecker **.html INFO linkcheck.cmdline 2022-10-24 15:16:33,785 MainThread Checking intern URLs only; use --check-extern to check extern URLs. LinkChecker 10.1.0 Copyright (C) 2000-2016 Bastian Kleineidam, 2010-2021 LinkChecker Authors LinkChecker comes with ABSOLUTELY NO WARRANTY! This is free software, and you are welcome to redistribute it under certain conditions. Look at the file `LICENSE' within this distribution. Read the documentation at https://linkchecker.github.io/linkchecker/ Write comments and bugs to https://github.com/linkchecker/linkchecker/issues Start checking at 2022-10-24 15:16:33+001 /usr/lib/python3.10/site-packages/bs4/__init__.py:435: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup. warnings.warn( 10 threads active, 88 links queued, 241 links in 339 URLs checked, runtime 1 seconds Statistics: Downloaded: 4.04MB. Content types: 7 image, 106 text, 0 video, 0 audio, 22 application, 3 mail and 508 other. URL lengths: min=16, max=256, avg=67. That's it. 646 links in 646 URLs checked. 0 warnings found. 0 errors found. Stopped checking at 2022-10-24 15:16:39+001 (5 seconds) ```
2022-10-24Fixup dead linksDavid Robertson8-12/+16
2022-10-24Move old architecture docsDavid Robertson2-0/+0
2022-10-24Move usage docs to usage dirDavid Robertson4-8/+8
2022-10-24move other doc to other dirDavid Robertson3-5/+5
2022-10-24Rearrange admin API fs heirarchy to match sidebarDavid Robertson18-30/+30
I've kept the RST redirect file in-place though
2022-10-24Move admin docs to admin section (excluding admin api)David Robertson14-42/+42
2022-10-24Move config pages into config dirDavid Robertson34-52/+52
2022-10-24Move dev pages into dev dirDavid Robertson18-21/+21
2022-10-24Move setup pages into setup dirDavid Robertson12-26/+26
2022-10-24Bump peaceiris/actions-gh-pages from 3.8.0 to 3.9.0 (#14276)dependabot[bot]2-1/+2
2022-10-24Bump peaceiris/actions-mdbook from 1.1.14 to 1.2.0 (#14275)dependabot[bot]2-1/+2
2022-10-24Add debugging to help diagnose lost device-list-update (#14268)Richard van der Hoff2-17/+38
2022-10-24Bump pysaml2 from 7.1.2 to 7.2.1 (#14270)dependabot[bot]2-4/+5
2022-10-24Bump jinja2 from 3.0.3 to 3.1.2 (#14271)dependabot[bot]2-4/+5
2022-10-24Bump types-requests from 2.28.11 to 2.28.11.2 (#14272)dependabot[bot]2-3/+4
2022-10-24Bump setuptools-rust from 1.5.1 to 1.5.2 (#14273)dependabot[bot]2-3/+4
2022-10-24Bump prometheus-client from 0.14.0 to 0.15.0 (#14274)dependabot[bot]2-3/+4
2022-10-24Bump anyhow from 1.0.65 to 1.0.66 (#14278)dependabot[bot]3-3/+4
2022-10-24Bump serde from 1.0.145 to 1.0.147 (#14277)dependabot[bot]3-5/+6
2022-10-21Add initial power level event to batch of bulk persisted events when ↵Shay8-58/+82
creating a new room. (#14228)
2022-10-21Improve aesthetics and reusability of HTML templates. (#13652)Germain31-740/+691
Use a base template to create a cohesive feel across the HTML templates provided by Synapse. Adds basic styling to the base template for a more user-friendly look and feel.
2022-10-21Fix logging error on SIGHUP (#14258)Richard van der Hoff2-1/+3
2022-10-21Show erasure status when listing users in the Admin API (#14205)Tadeusz Sośnierz5-3/+51
* Show erasure status when listing users in the Admin API * Use USING when joining erased_users * Add changelog entry * Revert "Use USING when joining erased_users" This reverts commit 30bd2bf106415caadcfdbdd1b234ef2b106cc394. * Make the erased check work on postgres * Add a testcase for showing erased user status * Appease the style linter * Explicitly convert `erased` to bool to make SQLite consistent with Postgres This also adds us an easy way in to fix the other accidentally integered columns. * Move erasure status test to UsersListTestCase * Include user erased status when fetching user info via the admin API * Document the erase status in user_admin_api * Appease the linter and mypy * Signpost comments in tests Co-authored-by: Tadeusz Sośnierz <tadeusz@sosnierz.com> Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-10-21Fix event size checks (#13710)DeepBlueV7.X2-5/+6
2022-10-20Build wheels on macos 11, not 10.15 (#14249)David Robertson2-3/+4
2022-10-20Use servlets for /key/ endpoints. (#14229)Patrick Cloke9-83/+86
To fix the response for unknown endpoints under that prefix. See MSC3743.
2022-10-20Stop returning `unsigned.invite_room_state` in `PUT ↵Andrew Morgan2-0/+6
/_matrix/federation/v2/invite/{roomId}/{eventId}` responses (#14064) Co-authored-by: David Robertson <davidr@element.io>
2022-10-19Explain `SynapseError` and `FederationError` better (#14191)Eric Eastwood3-3/+30
Explain `SynapseError` and `FederationError` better Spawning from https://github.com/matrix-org/synapse/pull/13816#discussion_r993262622
2022-10-19Bump types-opentracing from 2.4.7 to 2.4.10 (#14133)dependabot[bot]2-3/+4
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: reivilibre <oliverw@matrix.org>
2022-10-19Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042)dependabot[bot]8-15/+22
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
2022-10-19Document encryption_enabled_by_default_for_room_type under the right name ↵Matthew Hodgson2-1/+2
(#14110) * document encryption_enabled_by_default_for_room_type under the right name * add changelog * Update changelog.d/14110.doc
2022-10-19let update_synapse_database run on a multi-database configurations (#13422)Finn2-8/+1
* Allow sharded database in db migrate script Signed-off-by: Finn Herzfeld <finn@beeper.com> * Update changelog.d/13422.bugfix Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Remove check entirely * remove unused import Signed-off-by: Finn Herzfeld <finn@beeper.com> Co-authored-by: finn <finn@beeper.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-10-19Remove poetry.lock updates v1.70.0rc1Erik Johnston1-22/+0
2022-10-19Update lockfileErik Johnston1-9/+8
2022-10-191.70.0rc1Erik Johnston105-104/+126
2022-10-19Use backend-meta edition of issue triage workflow (#14230)Will Hunt2-21/+9
2022-10-18Fix MSC3030 `/timestamp_to_event` returning `outliers` that it has no idea ↵Eric Eastwood3-21/+104
whether are near a gap or not (#14215) Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](https://github.com/matrix-org/synapse/blob/2c63cdcc3f1aa4625e947de3c23e0a8133c61286/docs/development/room-dag-concepts.md#outliers). Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake. Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation. Fix https://github.com/matrix-org/synapse/issues/13944 ### Why did this fail before? Why was it flakey? Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.` This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](https://github.com/matrix-org/synapse/blob/cb20b885cb4bd1648581dd043a184d86fc8c7a00/synapse/handlers/room.py#L1470-L1496) so it always runs after `sync_partial_state` completes. ```py from twisted.internet import task as twisted_task d = twisted_task.deferLater(self.hs.get_reactor(), 3.5) await d ``` In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
2022-10-18Fix hiding devices names over federation (#10015)Aaron Raimist2-2/+9
And don't include blank opentracing stuff in device list updates. Signed-off-by: Aaron Raimist <aaron@raim.ist>
2022-10-18Fix docstring in EventContext (#14145)Shay2-1/+4
2022-10-18Add aarch64 wheels to CI (#14212)MichaIng2-5/+26
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-10-18Bump twisted from 22.4.0 to 22.8.0 (#14207)dependabot[bot]2-13/+15
* Bump twisted from 22.4.0 to 22.8.0 Bumps [twisted](https://github.com/twisted/twisted) from 22.4.0 to 22.8.0. - [Release notes](https://github.com/twisted/twisted/releases) - [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst) - [Commits](https://github.com/twisted/twisted/compare/twisted-22.4.0...twisted-22.8.0) --- updated-dependencies: - dependency-name: twisted dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18Prepatory work for adding power level event to batched events (#14214)Shay6-17/+47
2022-10-18Bump pygithub from 1.55 to 1.56 (#14206)dependabot[bot]2-4/+5
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18Bump types-setuptools from 65.4.0.0 to 65.5.0.1 (#14208)dependabot[bot]2-3/+4
* Bump types-setuptools from 65.4.0.0 to 65.5.0.1 Bumps [types-setuptools](https://github.com/python/typeshed) from 65.4.0.0 to 65.5.0.1. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-setuptools dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)Hugh Nimmo-Smith16-45/+257
Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18Don't pin dev-deps in pyproject; use lower bounds (#14227)David Robertson3-7/+8
* Don't pin dev-deps in pyproject; use lower bounds This makes it slightly less tedious to update these things via successive dependabot updates, by reducing the likelihood of a merge conflict. * Changelog * Changelog
2022-10-18Fixes to release-artifacts warnings (#14224)David Robertson2-0/+5
2022-10-18Update the thread_id right before use (in case the bg update hasn't ↵Patrick Cloke5-134/+112
finished) (#14222) This avoids running a forced-update of a null thread_id rows. An index is added (in the background) to hopefully make this easier in the future.
2022-10-18Enable URL previews in complement homeserver config. (#14198)Jonathan de Jong2-0/+3
2022-10-18Fix `track_memory_usage` on poetry-core 1.3.x installations (#14221)David Robertson3-3/+4
* Fix `track_memory_usage` on poetry-core 1.3.x installations The same kind of problem as discussed in #14085: 1. we defined an extra with an underscore 2. we look it up at runtime with an underscore 3. but poetry-core 1.3.x. installs it with a dash, causing (2) to fail. Fix by using a dash everywhere. * Changelog
2022-10-18Deal with some GHA deprecation warnings (#14216)David Robertson4-4/+12
2022-10-18Flush stdout/err in Dockerfile-workers before replacing the current process ↵realtyem3-17/+22
(#14195) Also update `subprocess.check_output` to the slightly newer `subprocess.run`. Signed-off-by: Jason Little <realtyem@gmail.com>
2022-10-18When restarting a partial join resync, prioritise the server which actioned ↵David Robertson6-31/+95
a partial join (#14126)
2022-10-18Allow poetry-core 1.3.2 (#14217)Ivan Shapovalov2-1/+2
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2022-10-18Avoid checking the event cache when backfilling events (#14164)Andrew Morgan4-15/+140
2022-10-18Remove `_get_events_cache` check optimisation from `_have_seen_events_dict` ↵Andrew Morgan3-30/+14
(#14161)
2022-10-17Add debug logs to figure out why an event was filtered (#14095)Eric Eastwood2-1/+32
Spawned while investigating https://github.com/matrix-org/synapse/issues/13944 This way we might get some more context whenever an `403 Forbidden - body: {"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}` error is produced. `log_config.yaml` ```yaml loggers: synapse: level: INFO synapse.visibility: level: DEBUG ```
2022-10-17Cache Rust build cache when building docker images (#14130)Erik Johnston3-1/+5
2022-10-17Bump click from 8.1.1 to 8.1.3 (#14201)dependabot[bot]3-5/+6
2022-10-17Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke9-177/+212
Gated behind an experimental configuration flag.
2022-10-17Bump psycopg2 from 2.9.3 to 2.9.4 (#14200)dependabot[bot]2-12/+13
2022-10-17Bump bcrypt from 3.2.0 to 4.0.1 (#14186)dependabot[bot]2-15/+23
2022-10-17Use Pydantic when PUTting room aliases (#14179)David Robertson3-31/+47
2022-10-17Invalidate rooms for user caches when receiving membership events (#14155)Nick Mills-Barrett2-0/+5
This should fix a race where the event notification comes in over replication before the state replication, leaving a window during which a sync may get an incorrect list of rooms for the user.
2022-10-17Fix dead link to admin registration API (#14189)David Robertson2-1/+2
* Fix dead link to admin registration API * Changelog
2022-10-17Remove not needed `replication` listener in docker compose example (#14107)Dirk Klimpel4-25/+2
2022-10-171.69.0 v1.69.0Erik Johnston3-3/+15
2022-10-15Stop getting missing `prev_events` after we already know their signature is ↵Eric Eastwood7-2/+386
invalid (#13816) While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place. Related to - https://github.com/matrix-org/synapse/issues/13622 - https://github.com/matrix-org/synapse/pull/13635 - https://github.com/matrix-org/synapse/issues/13676 Part of https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures. With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now. For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761 To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid. - `backfill` - `outgoing-federation-request` `/backfill` - `_check_sigs_and_hash_and_fetch` - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill - ❗ `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - ❗ `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - `_process_pulled_events` - `_process_pulled_event` for each validated event - ❗ Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it - `_get_state_ids_after_missing_prev_event` - `outgoing-federation-request` `/state_ids` - ❗ `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again - ❗ `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-14Bump types-bleach from 4.1.4 to 5.0.3 (#14188)dependabot[bot]2-3/+4
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-15Bump types-pillow from 9.2.2 to 9.2.2.1 (#14187)dependabot[bot]2-20/+21
* Bump types-pillow from 9.2.2 to 9.2.2.1 Bumps [types-pillow](https://github.com/python/typeshed) from 9.2.2 to 9.2.2.1. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-pillow dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-14Strip whitespace from worker types in Dockerfile-workers (#14165)realtyem2-4/+3
2022-10-14Accept threaded receipts for events related to the root event. (#14174)Patrick Cloke5-8/+247
The root node of a thread (and events related to it) are considered "part of a thread" when validating receipts. This allows clients which show the root node in both the main timeline and the threaded timeline to easily send receipts in either. Note that threaded notifications are not created for these events, these events created notifications on the main timeline.
2022-10-141.69.0rc4 v1.69.0rc4Erik Johnston6-5/+23
2022-10-14Fix background update to use an index (#14181)Erik Johnston2-11/+52
2022-10-14Advertise support for Matrix 1.4. (#14184)Patrick Cloke3-1/+3
All features / changes in Matrix 1.4 are now supported in Synapse.
2022-10-14Fix changelog for #14171 to match #13394/#14175.Patrick Cloke1-1/+1
2022-10-14Don't require optional `invite_room_state` field on fed v2 invite (#14083)Andrew Morgan2-1/+2
2022-10-14Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke16-50/+29
The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
2022-10-14Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182)realtyem2-1/+12
2022-10-14Stabilize the threads API. (#14175)Patrick Cloke7-29/+35
Stabilize the threads API (MSC3856) by supporting (only) the v1 path for the endpoint. This also marks the API as safe for workers since it is a read-only API.
2022-10-14Properly invalidate get_thread_id cache. (#14163)Patrick Cloke2-0/+2
This was missed in 2b6d41ebd685fb546e52acdbcb0024dfcf5a5db1 (#13824).
2022-10-14Revert "Disable push of docker images"Erik Johnston1-1/+1
This reverts commit f3f303aa22b9681c21468fb0bdce1b21d1bdbd92.
2022-10-14Fix docker workflowErik Johnston1-3/+5
2022-10-14Disable push of docker imagesErik Johnston1-1/+1
2022-10-13Fix docker build OOMing in CI for arm64 builds (#14173)Erik Johnston3-1/+11
Co-authored-by: David Robertson <davidr@element.io>
2022-10-13Optimise the event_push_backfill_thread_id bg job (#14172)David Robertson2-2/+9
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-10-13Fix sqlite syntax for upserts. (#14171)Patrick Cloke2-1/+2
2022-10-13Properly return the thread ID down sync. (#14159)Patrick Cloke1-2/+2
Fix a broken conflict in e6e876b9b158f47811b6dfedd8783f658ce960a4, by not stomping over a field right after creating it.
2022-10-13Add an API for listing threads in a room. (#13394)Patrick Cloke10-6/+522
Implement the /threads endpoint from MSC3856. This is currently unstable and behind an experimental configuration flag. It includes a background update to backfill data, results from the /threads endpoint will be partial until that finishes.
2022-10-12Fix a bug where the joined hosts for a given event were not being properly ↵Shay3-45/+51
cached (#14125)
2022-10-12Return the thread ID properly down sync. (#14159)Patrick Cloke2-0/+3
A receipt's thread ID, if one exists, should be added to the body of a receipt.
2022-10-12Return the main timeline for events which are not part of a thread. (#14140)Patrick Cloke3-6/+9
Fixes a bug where threaded receipts could not be sent for the main timeline.
2022-10-12Bump typing-extensions from 4.1.1 to 4.4.0 (#14134)dependabot[bot]2-5/+6
2022-10-12Correct field name for stripped state events when knocking. ↵Andrew Morgan4-6/+26
`knock_state_events` -> `knock_room_state` (#14102)
2022-10-12Bump bleach from 4.1.0 to 5.0.1 (#14116)dependabot[bot]2-5/+9
2022-10-12Bump idna from 3.3 to 3.4 (#14115)dependabot[bot]2-3/+4
2022-10-12Fixup changelog v1.69.0rc3Erik Johnston1-6/+6
2022-10-121.69.0rc3Erik Johnston7-5/+24
2022-10-12Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke6-2/+37
Co-authored-by: Eric Eastwood <erice@element.io>
2022-10-12Batch up calls to `get_rooms_for_users` (#14109)Nick Mills-Barrett2-1/+17
2022-10-12Remove the experimental implementation of MSC3772. (#14094)Patrick Cloke12-365/+22
MSC3772 has been abandoned.
2022-10-12Use minimal Rust installation in docker images and CI (#14141)David Robertson5-3/+5
2022-10-11Fix a bug where redactions were not being sent over federation if we did not ↵Shay6-38/+62
have the original event. (#13813)
2022-10-11Handle `gottestfmt` repository move (#14144)David Robertson3-2/+3
2022-10-11Remove the groups config code. (#14142)Patrick Cloke2-27/+1
This has been unused for a long time, but missed removal in #11584.
2022-10-11Fix rotating existing notifications in push summary (#14138)Erik Johnston2-5/+16
Broke by #14045. Fixes #14120. Introduced in v1.69.0rc2.
2022-10-11Making parse_server_name more consistent (#14007)Abdullah Osama3-2/+6
Fixes #12122
2022-10-11Fallback if 'approved' isn't included in a registration replication request ↵Brendan Abolivier2-1/+18
(#14135)
2022-10-11Enable dependabot for Rust dependencies (#14132)Erik Johnston3-28/+34
2022-10-11Fix pinning Rust deps in docker images (#14129)Erik Johnston3-1/+3
2022-10-10Fix name of "alias_creation_rules" option in config manual (#14124)Andrew Morgan2-2/+3
2022-10-10Bump types-psycopg2 from 2.9.9 to 2.9.21.1 (#14114)dependabot[bot]2-3/+4
* Bump types-psycopg2 from 2.9.9 to 2.9.21.1 Bumps [types-psycopg2](https://github.com/python/typeshed) from 2.9.9 to 2.9.21.1. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-psycopg2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-10Bump types-pillow from 9.0.15 to 9.2.2 (#14113)dependabot[bot]2-5/+6
* Bump types-pillow from 9.0.15 to 9.2.2 Bumps [types-pillow](https://github.com/python/typeshed) from 9.0.15 to 9.2.2. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-pillow dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-07Indicate what endpoint came back with a JSON response we were unable to ↵Eric Eastwood3-2/+12
parse (#14097) **Before:** ``` WARNING - POST-11 - Unable to parse JSON: Expecting value: line 1 column 1 (char 0) (b'') ``` **After:** ``` WARNING - POST-11 - Unable to parse JSON from POST /_matrix/client/v3/join/%21ZlmJtelqFroDRJYZaq:hs1?server_name=hs1 response: Expecting value: line 1 column 1 (char 0) (b'') ``` --- It's possible to figure out which endpoint these warnings were coming from before but you had to follow the request ID `POST-11` to the log line that says `Completed request [...]`. Including this key information next to the JSON parsing error makes it much easier to reason whether it matters or not. ``` 2022-09-29T08:23:25.7875506Z synapse_main | 2022-09-29 08:21:10,336 - synapse.http.matrixfederationclient - 299 - INFO - POST-11 - {GET-O-13} [hs1] Completed request: 200 OK in 0.53 secs, got 450 bytes - GET matrix://hs1/_matrix/federation/v1/make_join/%21ohtKoQiXlPePSycXwp%3Ahs1/%40charlie%3Ahs2?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=org.matrix.msc2176&ver=7&ver=8&ver=9&ver=org.matrix.msc3787&ver=10&ver=org.matrix.msc2716v4 ``` --- As a note, having no `body` is normal for the `/join` endpoint and it can handle it. https://github.com/matrix-org/synapse/blob/0c853e09709d52783efd37060ed9e8f55a4fc704/synapse/rest/client/room.py#L398-L403 Alternatively we could remove these extra logs but they are probably more usually helpful to figure out what went wrong.
2022-10-07Remove support for the unstable dir flag on relations. (#14106)Patrick Cloke5-57/+31
From MSC3715, this was unused by clients (and there was no way for clients to know it was supported). Matrix 1.4 defines the stable field.
2022-10-07Apply & bundle edits for non-message events. (#14034)Patrick Cloke2-7/+5
Fixes two related bugs: * No edit information was bundled for events which aren't `m.room.message`. * `m.new_content` was not applied for those events.
2022-10-07Document Google OpenID Connect email attribute (#14081)Paul Tötterman2-1/+3
2022-10-07Parse SYNAPSE_ASYNC_IO_REACTOR env variable & log the reactor on startup ↵Quentin Gliech3-14/+16
(#14092)
2022-10-07Bump phonenumbers from 8.12.44 to 8.12.56 (#14043)dependabot[bot]2-3/+4
* Bump phonenumbers from 8.12.44 to 8.12.56 Bumps [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) from 8.12.44 to 8.12.56. - [Release notes](https://github.com/daviddrysdale/python-phonenumbers/releases) - [Commits](https://github.com/daviddrysdale/python-phonenumbers/compare/v8.12.44...v8.12.56) --- updated-dependencies: - dependency-name: phonenumbers dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2022-10-07Unpin build-system requirements, but impose an upper-bound (#14085)David Robertson3-4/+10
* Revert to prior build-system requirements This reverts #14080. * Use normalised extra name, which poetry-core 1.3 will generate anyway * Changelog * Upper bound build-system requirements * Remove upgrade note; expand changelog entry a little. * Fix typo in build-system comment Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-10-07Bump pydantic from 1.9.1 to 1.10.2 (#14044)dependabot[bot]2-38/+40
* Bump pydantic from 1.9.1 to 1.10.2 Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.9.1 to 1.10.2. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](https://github.com/pydantic/pydantic/compare/v1.9.1...v1.10.2) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2022-10-07Bump msgpack from 1.0.3 to 1.0.4 (#14040)dependabot[bot]2-36/+55
* Bump msgpack from 1.0.3 to 1.0.4 Bumps [msgpack](https://github.com/msgpack/msgpack-python) from 1.0.3 to 1.0.4. - [Release notes](https://github.com/msgpack/msgpack-python/releases) - [Changelog](https://github.com/msgpack/msgpack-python/blob/main/ChangeLog.rst) - [Commits](https://github.com/msgpack/msgpack-python/compare/v1.0.3...v1.0.4) --- updated-dependencies: - dependency-name: msgpack dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2022-10-07Improve the listener example on the metrics documentation (#14078)Dirk Klimpel2-12/+30
Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
2022-10-07Be more lenient in the oEmbed response parsing. (#14089)Patrick Cloke3-51/+160
Attempt to parse any valid information from an oEmbed response (instead of bailing at the first unexpected data). This should allow for more partial oEmbed data to be returned, resulting in better / more URL previews, even if those URL previews are only partial.
2022-10-07Use stable identifiers for MSC3771 & MSC3773. (#14050)Patrick Cloke12-52/+49
These are both part of Matrix 1.4 which has now been released. For now, support both the unstable and stable identifiers.
2022-10-07Add sample worker files for `pusher` and `federation_sender` (#14077)Dirk Klimpel4-0/+29
Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
2022-10-07Use Pydantic to validate /devices endpoints (#14054)David Robertson2-46/+53
2022-10-07Catch BrokenPipeError from metrics server, and log as a warning (#14072)David Robertson2-6/+13
2022-10-07Update frozendict 2.3.3 -> 2.3.4 (#13955)David Robertson2-18/+19
* Update frozendict 2.3.3 -> 2.3.4 This claims to fix more memory leaks. Could have automated this upgrade with #11828 if we wanted. * Changelog
2022-10-06Fix broken links to README (#14093)David Robertson3-2/+3
2022-10-06Always close _all_ `ijson` coroutines, even if doing so raises Exceptions ↵David Robertson4-5/+76
(#14065)
2022-10-06The changelog entry ending in a `.` or `!` is not optional (#14087)Andre Klärner2-1/+2
2022-10-06Update 1.69.0rc2 changelog v1.69.0rc2Sean Quah1-2/+2
2022-10-06Replace incorrect 1.69.0rc3 version with 1.69.0rc2Sean Quah1-1/+1
2022-10-061.69.0rc3Sean Quah5-5/+10
2022-10-06Fix sending events into rooms with non-integer power levels (#14073)David Robertson4-1/+86
2022-10-06Pin build-system requirements (#14080)David Robertson2-1/+2
* Pin build-system requirements * Changelog
2022-10-06Deprecate the `generate_short_term_login_token` method in favor of an async ↵Quentin Gliech3-0/+76
`create_login_token` method in the Module API. (#13842) Signed-off-by: Quentin Gliech <quenting@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-10-061.69.0rc2Sean Quah4-4/+17
2022-10-06openid.md: fix a typo in the facebook exampleRichard van der Hoff1-1/+1
2022-10-05Batch up notifications after event persistence (#14033)Shay5-58/+66
2022-10-05Complement test image: capture logs from nginx (#14063)Richard van der Hoff2-1/+6
Have nginx send its logs to stderr/out, so that we can debug https://github.com/matrix-org/synapse/issues/13334.
2022-10-05Fix handling of public rooms filter with a network tuple. (#14053)Patrick Cloke3-27/+58
Fixes two related bugs: * The handling of `[null]` for a `room_types` filter was incorrect. * The ordering of arguments when providing both a network tuple and room type field was incorrect.
2022-10-05Fix backwards compatibility with upcoming threads schema changes. (#14045)Patrick Cloke2-11/+24
Ensure that the upsert will work properly by first updating any existing rows (in the same way that the background update to backfill data works).
2022-10-04Use threaded receipts when fetching events for push. (#13878)Patrick Cloke3-41/+97
Update the HTTP and email pushers to consider threaded read receipts when fetching unread events.
2022-10-04Bump types-pyyaml from 6.0.4 to 6.0.12 (#14041)dependabot[bot]2-5/+6
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Remove get rooms for user with stream ordering (#13991)Nick Mills-Barrett2-80/+70
By getting the joined rooms before the current token we avoid any reading history to confirm a user *was* in a room. We can then use any membership change events, which we already fetch during sync, to determine the final list of joined room IDs.
2022-10-04Recursively fetch the thread for receipts & notifications. (#13824)Patrick Cloke5-2/+162
Consider an event to be part of a thread if you can follow a chain of relations up to a thread root. Part of MSC3773 & MSC3771.
2022-10-04update authlib from 0.15.5 to 1.1.0 (#14006)ffaex2-10/+8
Signed-off-by: Fyodor Fomichev <fyodor.fomichev@gmail.com>
2022-10-04Mark events as read using threaded read receipts from MSC3771. (#13877)Patrick Cloke5-62/+504
Applies the proper logic for unthreaded and threaded receipts to either apply to all events in the room or only events in the same thread, respectively.
2022-10-04Advertise supporting version 1.3 of the Matrix spec. (#14032)Patrick Cloke2-0/+2
Now that all features / changes in 1.3 are supported in Synapse.
2022-10-04Fix building wheels on OSX (#14046)David Robertson2-0/+5
* Fix building wheels on OSX Follow-up to #13983. I missed a breaking change in setup-python v4. Serves me right for rushing to cut through the dependabot spam. * Changelog * Merge changelog
2022-10-04Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke17-93/+514
When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
2022-10-04Bump isort from 5.7.0 to 5.10.1 (#13988)dependabot[bot]3-6/+8
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Update changelog v1.69.0rc1Sean Quah1-20/+18
2022-10-04Bump jsonschema from 4.4.0 to 4.16.0 (#13985)dependabot[bot]2-4/+18
* Bump jsonschema from 4.4.0 to 4.16.0 Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema) from 4.4.0 to 4.16.0. - [Release notes](https://github.com/python-jsonschema/jsonschema/releases) - [Changelog](https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst) - [Commits](https://github.com/python-jsonschema/jsonschema/compare/v4.4.0...v4.16.0) --- updated-dependencies: - dependency-name: jsonschema dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Bump types-pyopenssl from 22.0.0 to 22.0.10 (#13984)dependabot[bot]2-5/+6
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Bump actions/setup-python from 2 to 4 (#13983)dependabot[bot]4-8/+9
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Bump types-requests from 2.27.11 to 2.28.11 (#13987)dependabot[bot]2-3/+4
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-04Bump types-setuptools from 57.4.9 to 65.4.0.0 (#13986)dependabot[bot]2-28/+11
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-10-04Disable pushing for server ACL events (MSC3786). (#13997)Patrick Cloke6-25/+5
Switches to the stable identifier for MSC3786 and enables it by default. This disables pushes of m.room.server_acl events.
2022-10-04Send the appservice access token as a header. (#13996)Patrick Cloke3-6/+26
Implements MSC2832 by sending application service access tokens in the Authorization header. The access token is also still sent as a query parameter until the application service ecosystem has fully migrated to using headers. In the future this could be made opt-in, or removed completely.
2022-10-041.69.0rc1Sean Quah102-102/+111
2022-10-04Linkify config documentation. (#14003)Akshit Tyagi2-5/+8
2022-10-03Track when the pulled event signature fails (#13815)Eric Eastwood5-15/+140
Because we're doing the recording in `_check_sigs_and_hash_for_pulled_events_and_fetch` (previously named `_check_sigs_and_hash_and_fetch`), this means we will track signature failures for `backfill`, `get_room_state`, `get_event_auth`, and `get_missing_events` (all pulled event scenarios). And we also record signature failures from `get_pdu`. Part of https://github.com/matrix-org/synapse/issues/13700 Part of https://github.com/matrix-org/synapse/issues/13676 and https://github.com/matrix-org/synapse/issues/13356 This PR will be especially important for https://github.com/matrix-org/synapse/pull/13816 so we can avoid the costly `_get_state_ids_after_missing_prev_event` down the line when `/messages` calls backfill.
2022-10-03Dependabot changelog: don't require a changelog in lintsDavid Robertson2-16/+7
2022-10-03Bump actions/upload-artifact from 2 to 3 (#14023)dependabot[bot]5-5/+6
* Bump actions/upload-artifact from 2 to 3 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2 to 3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-03Dependabot changelog: try using branch nameDavid Robertson1-2/+2
Gonna hope this doesn't use the merge ref and is good enough
2022-10-03Clear out old rows from `event_push_actions_staging` (#14020)Erik Johnston5-1/+105
On matrix.org we have ~5 million stale rows in `event_push_actions_staging`, let's add a background job to make sure we clear them out.
2022-10-03Announce that legacy metric names are deprecated, will be turned off by ↵reivilibre5-27/+64
default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. (#14024)
2022-10-03Bump actions/cache from 2 to 3 (#13980)dependabot[bot]2-1/+2
* Bump actions/cache from 2 to 3 Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-03Dependabot changelog: allow dependabot to trigger actionsDavid Robertson1-0/+3
2022-10-03Bump actions/download-artifact from 2 to 3 (#13979)dependabot[bot]2-1/+2
* Bump actions/download-artifact from 2 to 3 Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-03Do not return unspecced original_event field when using the stable ↵Patrick Cloke4-17/+28
/relations endpoint. (#14025) Keep the old behavior (of including the original_event field) for any requests to the /unstable version of the endpoint, but do not include the field when the /v1 version is used. This should avoid new clients from depending on this field, but will not help with current dependencies.
2022-10-03Bump docker/login-action from 1 to 2 (#13978)dependabot[bot]2-1/+2
* Bump docker/login-action from 1 to 2 Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/login-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-03Dependabot changelog: pass GITHUB_TOKEN to ghDavid Robertson1-0/+2
2022-10-03Dependabot changelogs: trigger CI explicitly (#14027)David Robertson4-4/+22
* Dependabot changelogs: trigger CI explicitly * Changelog * Use merge ref, not head ref ref ref ref ref ref * explanatory note
2022-10-03Bump docker/build-push-action from 2 to 3 (#14022)dependabot[bot]2-1/+2
* Bump docker/build-push-action from 2 to 3 Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v2...v3) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
2022-10-03More WIP dependabot changelog CIDavid Robertson1-1/+2
Fixes a typo in #14021, and allows the workflow to trigger on `reopen` for easier debugging.
2022-10-03Bump docker/setup-qemu-action from 1 to 2 (#14019)dependabot[bot]2-1/+2
* Bump docker/setup-qemu-action from 1 to 2 Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v1...v2) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
2022-10-03yet another attempt to make dependabot changelogs work (#14021)David Robertson2-0/+6
* yet another attempt to make dependabot changelogs work * Changelog
2022-10-03Bump docker/setup-buildx-action from 1 to 2 (#14015)dependabot[bot]3-2/+3
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
2022-10-03Fix bug where we didn't delete staging push actions (#14014)Erik Johnston2-1/+2
Introduced in #13719
2022-10-03Try again to automate dependabot changelogs (#14017)David Robertson2-0/+7
* Try again to automate dependabot changelogs * Changelog
2022-10-03Add query parameter `ts` to allow appservices set the `origin_server_ts` for ↵lukasdenk4-15/+152
state events. (#11866) MSC3316 declares that both /rooms/{roomId}/send and /rooms/{roomId}/state should accept a ts parameter for appservices. This change expands support to /state and adds tests.
2022-10-03Fix twisted trunk mypy errors (#14012)David Robertson3-0/+7
2022-10-03Bump actions/checkout from 2 to 3 (#13982)dependabot[bot]6-32/+33
* Bump actions/checkout from 2 to 3 Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Changelog Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: David Robertson <davidr@element.io>
2022-10-03Add cache to `get_partial_state_servers_at_join` (#14013)Erik Johnston2-0/+8
2022-10-03Refactor `_get_e2e_device_keys_txn` to split large queries (#13956)Sean Quah3-29/+115
Instead of running a single large query, run a single query for user-only lookups and additional queries for batches of user device lookups. Resolves #13580. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-10-03Change dependabot CL suffix (#14011)David Robertson2-1/+2
* Change dependabot CL suffix * Changelog
2022-10-03Protoype automatic changelogs for dependabot PRs (#13998)David Robertson2-0/+24
2022-10-03Always build Rust extension in release mode (#14009)Erik Johnston2-0/+4
Fixes #13990
2022-10-03Revert the general exception recording introduced in #13814 (#13969)Eric Eastwood2-10/+1
* Maybe not catch all errors to avoid things in the nature-of CancelledError See https://github.com/matrix-org/synapse/pull/13815#discussion_r983384698 * Remove general exception tracking * Add changelog
2022-09-30Fix `get_users_in_room` mis-use in `transfer_room_state_on_room_upgrade` ↵Eric Eastwood2-2/+3
(#13960) Spawning from looking into `get_users_in_room` while investigating https://github.com/matrix-org/synapse/issues/13942#issuecomment-1262787050. See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755 for the original exploration around finding `get_users_in_room` mis-uses. Related to the following PRs where we also cleaned up some `get_users_in_room` mis-uses: - https://github.com/matrix-org/synapse/pull/13605 - https://github.com/matrix-org/synapse/pull/13608 - https://github.com/matrix-org/synapse/pull/13606 - https://github.com/matrix-org/synapse/pull/13958
2022-09-30Clarifications in user directory for users who share rooms tracking (#13966)Eric Eastwood2-12/+25
Spawned while working on [`get_users_in_room` mis-uses](https://github.com/matrix-org/synapse/pull/13958#discussion_r984074897) and thinking we could use `get_local_users_in_room` here but we can't. From first glance, it seemed like this was only using local users from all of the `is_mine_id(user_id)` checks but I see that it does actually use remote users. Just making things a little more clear here what it does and mentions remote users so maybe that will be more obvious in the future.
2022-09-30Skip filtering during push if there are no push actions (#13992)Erik Johnston4-2/+12
2022-09-30Update mypy and mypy-zope, attempt 3 (#13993)David Robertson11-68/+61
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30Revert "Update mypy and mypy-zope (#13925)"David Robertson10-60/+67
This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
2022-09-30Update mypy and mypy-zope (#13925)David Robertson10-67/+60
* Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30Enable dependabot updates (#13976)Richard van der Hoff2-0/+18
Fixes https://github.com/matrix-org/synapse/issues/11828
2022-09-30Speed up calculating push actions in large rooms (#13973)Erik Johnston3-12/+96
We move the expensive check of visibility to after calculating push actions, avoiding the expensive check for users who won't get pushed anyway. I think this should have a big impact on rooms with large numbers of local users that have pushed disabled.
2022-09-30Discourage automatic replies to Synapse's emails (#13957)David Robertson2-0/+14
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30Add some cross references to worker documentation (#13974)Dirk Klimpel3-15/+28
Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
2022-09-30Fix performance regression in `get_users_in_room` (#13972)Erik Johnston5-70/+98
Fixes #13942. Introduced in #13575. Basically, let's only get the ordered set of hosts out of the DB if we need an ordered set of hosts. Since we split the function up the caching won't be as good, but I think it will still be fine as e.g. multiple backfill requests for the same room will hit the cache.
2022-09-30Fix overflows in /messages backfill calculation (#13936)David Robertson3-41/+103
* Reproduce bug * Compute `least_function` first * Substitute `least_function` with an f-string * Bugfix: avoid overflow Co-authored-by: Eric Eastwood <erice@element.io>
2022-09-30Add missing version information in the ModuleApi (#13947)Brendan Abolivier2-0/+5
2022-09-29Don't require `setuptools_rust` at runtime (#13952)David Robertson3-3/+35
2022-09-29Update UPSERT comment now that native upserts are the default (#13924)David Robertson2-10/+51
2022-09-29Implement push rule evaluation in Rust. (#13838)Erik Johnston14-403/+894
2022-09-29Optimise get_rooms_for_user (drop with_stream_ordering) (#13787)Nick Mills-Barrett7-75/+66
2022-09-29Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier21-34/+731
used (using MSC3866) (#13556)
2022-09-29Explicit cast to enforce type hints. (#13939)Patrick Cloke2-4/+5
2022-09-29Expose MSC3882 only be under an unstable endpoint. (#13868)Hugh Nimmo-Smith3-8/+13
2022-09-29Clarify that a method returns only unthreaded receipts. (#13937)Patrick Cloke4-76/+47
By renaming it and updating the docstring. Additionally, refactors a method which is used only by tests.
2022-09-29Always send default and rule_id to clients (#13904)Nicolas Werner2-4/+3
2022-09-29Improve backfill robustness by trying more servers. (#13890)reivilibre2-2/+32
Co-authored-by: Eric Eastwood <erice@element.io>
2022-09-28Handle local device list updates during partial join (#13934)Erik Johnston4-15/+141
2022-09-28Limit and filter the number of backfill points to get from the database (#13879)Eric Eastwood4-82/+198
There is no need to grab thousands of backfill points when we only need 5 to make the `/backfill` request with. We need to grab a few extra in case the first few aren't visible in the history. Previously, we grabbed thousands of backfill points from the database, then sorted and filtered them in the app. Fetching the 4.6k backfill points for `#matrix:matrix.org` from the database takes ~50ms - ~570ms so it's not like this saves a lot of time 🤷. But it might save us more time now that `get_backfill_points_in_room`/`get_insertion_event_backward_extremities_in_room` are more complicated after https://github.com/matrix-org/synapse/pull/13635 This PR moves the filtering and limiting to the SQL query so we just have less data to work with in the first place. Part of https://github.com/matrix-org/synapse/issues/13356
2022-09-28Add upgrade notes for changes to receipts replication. (#13932)Patrick Cloke2-0/+13
2022-09-28Revert "Stop returning an unused column when handling new receipts. ↵Patrick Cloke2-3/+2
(#13933)" (#13935) This reverts commit 7766bd5b354cd4ea1a33351ba320e54a14d3aeac (#13933). The unused column is actually used, but much further down in the function.
2022-09-28Stop returning an unused column when handling new receipts. (#13933)Patrick Cloke2-2/+3
2022-09-28Document that the 'auto_join_rooms' option works with Spaces (#13931)Andrew Morgan2-2/+6
2022-09-28Correct default number of bcrypt hash rounds in .1 man page (#13930)Andrew Morgan2-1/+2
2022-09-28Handle remote device list updates during partial join (#13913)Erik Johnston6-0/+170
c.f. #12993 (comment), point 3 This stores all device list updates that we receive while partial joins are ongoing, and processes them once we have the full state. Note: We don't actually process the device lists in the same ways as if we weren't partially joined. Instead of updating the device list remote cache, we simply notify local users that a change in the remote user's devices has happened. I think this is safe as if the local user requests the keys for the remote user and we don't have them we'll simply fetch them as normal.
2022-09-28fix: Push notifications for invite over federation (#13719)Kateřina Churanová8-23/+42
2022-09-28Add instruction for running unit tests in parallel (#13928)Ashish Kumar2-0/+7
2022-09-28Persist CreateRoom events to DB in a batch (#13800)Shay12-339/+563
2022-09-28Prepatory work for batching events to send (#13487)Shay5-108/+290
This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
2022-09-27Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood5-67/+165
Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
2022-09-27Emphasize the right reasons to use `(room_id, event_id)` in a schema (#13915)Eric Eastwood2-15/+17
* Emphasize the right reasons to use (room_id, event_id) Follow-up to: - https://github.com/matrix-org/synapse/pull/13701 - https://github.com/matrix-org/synapse/pull/13771
2022-09-27Add new columns tracking when we partial-joined (#13892)David Robertson4-3/+87
2022-09-27Support the stable dir parameter for /relations. (#13920)Patrick Cloke3-13/+18
Since MSC3715 has passed FCP, the stable parameter can be used. This currently falls back to the unstable parameter if the stable parameter is not provided (and MSC3715 support is enabled in the configuration).
2022-09-27Prioritize outbound to-device over device list updates (#13922)Erik Johnston2-13/+17
Otherwise device list changes for large accounts can temporarily delay to-device messages.
2022-09-27Update upgrade notes (#13923)David Robertson1-6/+23
* Update upgrade notes * Synapse Synapse Synapse Synapse Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-09-27Dockerfile for tests: align comments with current behavior (#13867)Mathieu Velten2-13/+10
Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-09-27Carry IdP Session IDs through user-mapping sessions. (#13839)Quentin Gliech2-0/+10
Since #11482, we're saving sessions IDs from upstream IdPs, but we've been losing them when the user goes through a user mapping session on account registration.
2022-09-27Handle the case of remote users leaving a partial join room for device lists ↵Erik Johnston7-107/+85
(#13885)
2022-09-27Clarify who isn't affected by rust v1.68.0David Robertson1-2/+6
2022-09-27s/Synapse v1/Synapse 1/gDavid Robertson1-18/+18
2022-09-27tweak upgrade notesDavid Robertson1-3/+5
2022-09-27typo fixesDavid Robertson1-3/+3
2022-09-27`SQLite earlier than` -> `SQLite older than`David Robertson1-1/+1
2022-09-271.68.0David Robertson4-7/+20
2022-09-27Faster room joins: Fix spurious error when joining a room (#13872)Sean Quah2-3/+20
During a `lazy_load_members` `/sync`, we look through auth events in rooms with partial state to find prior membership events. When such a membership is not found, an error is logged. Since the first join event for a user never has a prior membership event to cite, the error would always be logged when one appeared in the room timeline. Avoid logging errors for such events. Introduced in #13477. Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-26Complement image: propagate SIGTERM to all workers (#13914)Richard van der Hoff2-2/+31
This should mean that logs from worker processes are flushed before shutdown. When a test completes, Complement stops the docker container, which means that synapse will receive a SIGTERM. Currently, the `complement_fork_starter` exits immediately (without notifying the worker processes), which means that the workers never get a chance to flush their logs before the whole container is vaped. We can fix this by propagating the SIGTERM to the children.
2022-09-26Improve tests for get_unread_push_actions_for_user_in_range_*. (#13893)Patrick Cloke3-30/+97
* Adds a docstring. * Reduces a small amount of duplicated code. * Improves tests.
2022-09-26Update the manpage documentation for the hash_password script (#13911)Andrew Morgan3-1/+9
2022-09-26Snapshot schema 72 (#13873)David Robertson13-22/+2165
Including another batch of fixes to the schema dump script
2022-09-26typing: check origin server of typing event against room's servers (#13830)Mathieu Velten3-2/+10
This is also using the partial state approximation if needed so we do not block here during a fast join. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-04 10:31:07 +0000