summary refs log tree commit diff
path: root/synapse/_scripts (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Remove 3pid and identity serversRory&5 days1-7/+0
|
* Bump ruff from 0.7.3 to 0.11.10 (#18451)dependabot[bot]2025-05-202-4/+4
| | | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Delete unreferenced state groups in background (#18254)Devon Hudson2025-03-211-0/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR fixes #18154 to avoid de-deltaing state groups which resulted in DB size temporarily increasing until the DB was `VACUUM`'ed. As a result, less state groups will get deleted now. It also attempts to improve performance by not duplicating work when processing state groups it has already processed in previous iterations. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erikj@element.io>
* Add a column `participant` to `room_memberships` table (#18068)Shay2025-03-181-0/+1
|
* Revert the background job to clear unreferenced state groups (that was ↵reivilibre2025-03-071-30/+0
| | | | | | | | | | | | | | | | | introduced in v1.126.0rc1), due to a suspected issue that causes increased disk usage. (#18222) Revert "Add background job to clear unreferenced state groups (#18154)" This mechanism is suspected of inserting large numbers of rows into `state_groups_state`, thus unreasonably increasing disk usage. See: https://github.com/element-hq/synapse/issues/18217 This reverts commit 5121f9210c989fcc909e78195133876dff3bc9b9 (#18154). --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* Add background job to clear unreferenced state groups (#18154)Devon Hudson2025-02-251-0/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #18150 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erikj@element.io>
* Cleanup Python 3.8 leftovers (#17967)V024602025-02-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some small cleanups after Python3.8 became EOL. - Move some type imports from `typing_extensions` to `typing` - Remove the `abi3-py38` feature from pyo3 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Quentin Gliech <quenting@element.io>
* Speed up slow initial sliding syncs on large servers (#17946)Erik Johnston2024-11-191-0/+2
| | | | This was due to a missing index, which meant that deleting previous connections associated with the device and `conn_id` took a long time.
* Format files with Ruff (#17643)Quentin Gliech2024-09-023-17/+20
| | | | | | I thought ruff check would also format, but it doesn't. This runs ruff format in CI and dev scripts. The first commit is just a run of `ruff format .` in the root directory.
* Sliding Sync: Pre-populate room data for quick filtering/sorting (#17512)Eric Eastwood2024-08-291-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pre-populate room data for quick filtering/sorting in the Sliding Sync API Spawning from https://github.com/element-hq/synapse/pull/17450#discussion_r1697335578 This PR is acting as the Synapse version `N+1` step in the gradual migration being tracked by https://github.com/element-hq/synapse/issues/17623 Adding two new database tables: - `sliding_sync_joined_rooms`: A table for storing room meta data that the local server is still participating in. The info here can be shared across all `Membership.JOIN`. Keyed on `(room_id)` and updated when the relevant room current state changes or a new event is sent in the room. - `sliding_sync_membership_snapshots`: A table for storing a snapshot of room meta data at the time of the local user's membership. Keyed on `(room_id, user_id)` and only updated when a user's membership in a room changes. Also adds background updates to populate these tables with all of the existing data. We want to have the guarantee that if a row exists in the sliding sync tables, we are able to rely on it (accurate data). And if a row doesn't exist, we use a fallback to get the same info until the background updates fill in the rows or a new event comes in triggering it to be fully inserted. This means we need a couple extra things in place until we bump `SCHEMA_COMPAT_VERSION` and run the foreground update in the `N+2` part of the gradual migration. For context on why we can't rely on the tables without these things see [1]. 1. On start-up, block until we clear out any rows for the rooms that have had events since the max-`stream_ordering` of the `sliding_sync_joined_rooms` table (compare to max-`stream_ordering` of the `events` table). For `sliding_sync_membership_snapshots`, we can compare to the max-`stream_ordering` of `local_current_membership` - This accounts for when someone downgrades their Synapse version and then upgrades it again. This will ensure that we don't have any stale/out-of-date data in the `sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` tables since any new events sent in rooms would have also needed to be written to the sliding sync tables. For example a new event needs to bump `event_stream_ordering` in `sliding_sync_joined_rooms` table or some state in the room changing (like the room name). Or another example of someone's membership changing in a room affecting `sliding_sync_membership_snapshots`. 1. Add another background update that will catch-up with any rows that were just deleted from the sliding sync tables (based on the activity in the `events`/`local_current_membership`). The rooms that need recalculating are added to the `sliding_sync_joined_rooms_to_recalculate` table. 1. Making sure rows are fully inserted. Instead of partially inserting, we need to check if the row already exists and fully insert all data if not. All of this extra functionality can be removed once the `SCHEMA_COMPAT_VERSION` is bumped with support for the new sliding sync tables so people can no longer downgrade (the `N+2` part of the gradual migration). <details> <summary><sup>[1]</sup></summary> For `sliding_sync_joined_rooms`, since we partially insert rows as state comes in, we can't rely on the existence of the row for a given `room_id`. We can't even rely on looking at whether the background update has finished. There could still be partial rows from when someone reverted their Synapse version after the background update finished, had some state changes (or new rooms), then upgraded again and more state changes happen leaving a partial row. For `sliding_sync_membership_snapshots`, we insert items as a whole except for the `forgotten` column ~~so we can rely on rows existing and just need to always use a fallback for the `forgotten` data. We can't use the `forgotten` column in the table for the same reasons above about `sliding_sync_joined_rooms`.~~ We could have an out-of-date membership from when someone reverted their Synapse version. (same problems as outlined for `sliding_sync_joined_rooms` above) Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) </details> ### TODO - [x] Update `stream_ordering`/`bump_stamp` - [x] Handle remote invites - [x] Handle state resets - [x] Consider adding `sender` so we can filter `LEAVE` memberships and distinguish from kicks. - [x] We should add it to be able to tell leaves from kicks - [x] Consider adding `tombstone` state to help address https://github.com/element-hq/synapse/issues/17540 - [x] We should add it `tombstone_successor_room_id` - [x] Consider adding `forgotten` status to avoid extra lookup/table-join on `room_memberships` - [x] We should add it - [x] Background update to fill in values for all joined rooms and non-join membership - [x] Clean-up tables when room is deleted - [ ] Make sure tables are useful to our use case - First explored in https://github.com/element-hq/synapse/compare/erikj/ss_use_new_tables - Also explored in https://github.com/element-hq/synapse/commit/76b5a576eb363496315dfd39510cad7d02b0fc73 - [x] Plan for how can we use this with a fallback - See plan discussed above in main area of the issue description - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) - [x] Plan for how we can rely on this new table without a fallback - Synapse version `N+1`: (this PR) Bump `SCHEMA_VERSION` to `87`. Add new tables and background update to backfill all rows. Since this is a new table, we don't have to add any `NOT VALID` constraints and validate them when the background update completes. Read from new tables with a fallback in cases where the rows aren't filled in yet. - Synapse version `N+2`: Bump `SCHEMA_VERSION` to `88` and bump `SCHEMA_COMPAT_VERSION` to `87` because we don't want people to downgrade and miss writes while they are on an older version. Add a foreground update to finish off the backfill so we can read from new tables without the fallback. Application code can now rely on the new tables being populated. - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.hh7shg4cxdhj) ### Dev notes ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase ``` ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.handlers.test_sliding_sync.FilterRoomsTestCase ``` Reference: - [Development docs on background updates and worked examples of gradual migrations ](https://github.com/element-hq/synapse/blob/1dfa59b238cee0dc62163588cc9481896c288979/docs/development/database_schema.md#background-updates) - A real example of a gradual migration: https://github.com/matrix-org/synapse/pull/15649#discussion_r1213779514 - Adding `rooms.creator` field that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/10697 - Adding `rooms.room_version` that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/6729 - Adding `room_stats_state.room_type` that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/13031 - Tables from MSC2716: `insertion_events`, `insertion_event_edges`, `insertion_event_extremities`, `batch_events` - `current_state_events` updated in `synapse/storage/databases/main/events.py` --- ``` persist_event (adds to queue) _persist_event_batch _persist_events_and_state_updates (assigns `stream_ordering` to events) _persist_events_txn _store_event_txn _update_metadata_tables_txn _store_room_members_txn _update_current_state_txn ``` --- > Concatenated Indexes [...] (also known as multi-column, composite or combined index) > > [...] key consists of multiple columns. > > We can take advantage of the fact that the first index column is always usable for searching > > *-- https://use-the-index-luke.com/sql/where-clause/the-equals-operator/concatenated-keys* --- Dealing with `portdb` (`synapse/_scripts/synapse_port_db.py`), https://github.com/element-hq/synapse/pull/17512#discussion_r1725998219 --- <details> <summary>SQL queries:</summary> Both of these are equivalent and work in SQLite and Postgres Options 1: ```sql WITH data_table (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) AS ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT * FROM data_table WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` Option 2: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT column1 as room_id, column2 as user_id, column3 as membership_event_id, column4 as membership, column5 as event_stream_ordering, {", ".join("column" + str(i) for i in range(6, 6 + len(insert_keys)))} FROM ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) as v WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` If we don't need the `membership` condition, we could use: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, membership_event_id, user_id, membership, event_stream_ordering, {", ".join(insert_keys)}) VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` </details> ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* hash_password accepts stdin now (#17608)eyJhb2024-08-271-1/+3
| | | | | | `hash_password` now actually accepts password from stdin. The `getpass` reads from TTY, and does NOT accept stdin in any way. The manpage has been updated to reflect that.
* Prepare for authenticated media freeze (#17433)Shay2024-07-221-2/+3
| | | | | | As part of the rollout of [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) this PR adds support for designating authenticated media and ensuring that authenticated media is not served over unauthenticated endpoints.
* Bump ruff from 0.3.7 to 0.5.0 (#17381)dependabot[bot]2024-07-052-2/+2
|
* register-new-matrix-user: add a flag to ignore already existing users (#17304)Jörg Thalheim2024-06-191-2/+28
| | | | Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
* register_new_matrix_user: add password-file flag (#17294)Jörg Thalheim2024-06-181-5/+15
| | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
* Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator` ↵Erik Johnston2024-05-301-10/+61
| | | | | (#17229) Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
* Add support for MSC3823 - Account Suspension (#17051)Shay2024-05-011-1/+1
|
* Add support for moving `/push_rules` off of main process (#17037)Erik Johnston2024-03-281-6/+3
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-131-11/+11
|
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-239-0/+11
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1310-105/+160
|\
| * Update license headersPatrick Cloke2023-11-2110-105/+160
| |
* | Write signing keys with file mode 0640 (#16740)elara-leitstellentechnik2023-12-081-3/+10
|/ | | | Co-authored-by: Fabian Klemp <fabian.klemp@frequentis.com>
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-091-2/+1
|
* Portdb: don't copy a table that gets rebuilt (#16563)David Robertson2023-10-271-1/+1
|
* Force TLS certificate verification in registration script. (#16530)Denis Kasak2023-10-231-2/+2
| | | | | | | | | | If using the script remotely, there's no particularly convincing reason to disable certificate verification, as this makes the connection interceptible. If on the other hand, the script is used locally (the most common use case), you can simply target the HTTP listener and avoid TLS altogether. This is what the script already attempts to do if passed a homeserver configuration YAML file.
* Update ruff config (#16283)Patrick Cloke2023-09-081-1/+0
| | | Enable additional checks & clean-up unneeded configuration.
* Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-291-2/+7
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-152-4/+4
|
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-101-1/+1
|
* Properly setup the additional sequences in the portdb script. (#16043)Patrick Cloke2023-08-011-3/+15
| | | | | The un_partial_stated_event_stream_sequence and application_services_txn_id_seq were never properly configured in the portdb script, resulting in an error on start-up.
* Add basic read/write lock (#15782)Erik Johnston2023-07-051-1/+8
|
* Add foreign key constraint to `event_forward_extremities`. (#15751)Erik Johnston2023-07-051-0/+2
|
* Fix harmless exception in port DB script (#15814)Erik Johnston2023-06-211-0/+3
| | | | | | | The port DB script would try and run database background tasks, which could fail if the data they acted on was in the process of being ported. These exceptions were non fatal. Fixes #15789
* Add an admin API endpoint to support per-user feature flags (#15344)Shay2023-04-281-0/+1
|
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-1/+4
|
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+2
|
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-241-1/+5
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Add a primitive helper script for listing worker endpoints. (#15243)reivilibre2023-03-231-0/+302
| | | | Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Add a check to SQLite port DB script to ensure that the sqlite database ↵Shay2023-03-221-0/+8
| | | | passed to the script exists before trying to port from it (#15306)
* Revert "check sqlite database file exists before porting/#14692" (#15301)Shay2023-03-211-1/+1
|
* Refactor media modules. (#15146)Patrick Cloke2023-02-271-1/+1
| | | | | | | * Removes the `v1` directory from `test.rest.media.v1`. * Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`. * Flatten the `v1` directory from `synapse.rest.media`, but leave compatiblity with 3rd party media repositories and spam checkers.
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-223-4/+0
|
* Teach portdb about `un_partial_stated_event_stream` (#15108)David Robertson2023-02-201-33/+52
| | | | | | | | | | | | * Sort BOOLEAN_COLUMNS and APPEND_ONLY_TABLES So I can see if a given table is present in logarithmic time, rather than linear. * Teach portdb about `un_partial_stated_event_streams` * Comments comments comments * Changelog
* Always notify replication when a stream advances (#14877)Erik Johnston2023-01-201-0/+4
| | | This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
* Check sqlite database file exists before porting. (#14692)Jeyachandran Rathnam2022-12-221-1/+1
| | | | To avoid creating an empty SQLite file if the given path is incorrect.
* Fix a bug introduced in v1.67.0 where not specifying a config file or a ↵reivilibre2022-12-071-2/+3
| | | | server URL would lead to the `register_new_matrix_user` script failing. (#14637)
* let update_synapse_database run on a multi-database configurations (#13422)Finn2022-10-191-8/+0
| | | | | | | | | | | | | | | | | * Allow sharded database in db migrate script Signed-off-by: Finn Herzfeld <finn@beeper.com> * Update changelog.d/13422.bugfix Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Remove check entirely * remove unused import Signed-off-by: Finn Herzfeld <finn@beeper.com> Co-authored-by: finn <finn@beeper.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add an API for listing threads in a room. (#13394)Patrick Cloke2022-10-131-0/+2
| | | | | | | | | Implement the /threads endpoint from MSC3856. This is currently unstable and behind an experimental configuration flag. It includes a background update to backfill data, results from the /threads endpoint will be partial until that finishes.
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-291-1/+1
| | | | used (using MSC3866) (#13556)
* Support enabling/disabling pushers (from MSC3881) (#13799)Brendan Abolivier2022-09-211-0/+1
| | | Partial implementation of MSC3881
* Generate separate snapshots for logical databases (#13792)David Robertson2022-09-201-4/+10
| | | | | | | * Generate separate snapshots for sqlite, postgres and common * Cleanup postgres dbs in the TRAP * Say which logical DB we're applying updates to * Run background updates on the state DB * Add new option for accepting a SCHEMA_NUMBER
* Add receipts event stream ordering (#13703)Nick Mills-Barrett2022-09-131-0/+2
|
* Support `registration_shared_secret` in a file (#13614)Richard van der Hoff2022-08-251-3/+42
| | | | A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
* register_new_matrix_user: read server url from config (#13616)Richard van der Hoff2022-08-251-6/+51
| | | | Fixes https://github.com/matrix-org/synapse/issues/3672: `https://localhost:8448` is virtually never right.
* Merge branch 'master' into developBrendan Abolivier2022-07-191-1/+1
|\
| * Remove 'anonymised' from the phone home stats documentation (#13321)Andrew Morgan2022-07-191-1/+1
| |
* | Drop unused tables from groups/communities. (#12967)Patrick Cloke2022-07-131-16/+0
|/ | | | These tables have been unused since Synapse v1.61.0, although schema version 72 was added in Synapse v1.62.0.
* Ensure portdb selects _all_ rows with negative rowids (#13226)David Robertson2022-07-111-1/+4
|
* Check that `auto_vacuum` is disabled when porting a SQLite database to ↵reivilibre2022-07-071-0/+34
| | | | Postgres, as `VACUUM`s must not be performed between runs of the script. (#13195)
* Merge remote-tracking branch 'origin/release-v1.62' into developPatrick Cloke2022-06-301-4/+2
|\
| * Add index to help delete old push actions (#13141)Erik Johnston2022-06-301-4/+2
| |
* | Improve performance of getting unread counts in rooms (#13119)Erik Johnston2022-06-291-0/+3
|/
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-151-0/+4
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Replace noop background updates with DELETE. (#12954)Patrick Cloke2022-06-131-2/+0
| | | | Removes the `register_noop_background_update` and deletes the background updates directly in a delta file.
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-072-8/+4
|
* Remove remaining pieces of groups code. (#12966)Patrick Cloke2022-06-061-0/+1
| | | | | * Remove an unused stream ID generator. * Remove the now unused remote profile cache.
* Remove groups code from synapse_port_db. (#12899)Patrick Cloke2022-06-031-8/+15
|
* Remove most groups datastore code. (#12895)Patrick Cloke2022-05-311-2/+2
| | | | The remaining piece is a background update that is needed for backwards compatibility.
* hash_password: raise an error if no config file is specified (#12789)Jae Lo Presti2022-05-191-5/+5
|
* Add missing type hints to config classes. (#12402)Patrick Cloke2022-04-111-3/+1
|
* Disallow untyped defs in synapse._scripts (#12422)David Robertson2022-04-1110-109/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Of note: * No untyped defs in `register_new_matrix_user` This one might be contraversial. `request_registration` has three dependency-injection arguments used for testing. I'm removing the injection of the `requests` module and using `unitest.mock.patch` in the test cases instead. Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the change: ``` synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any" synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any" ``` And after: ``` synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType" synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response" ``` * Drive-by comment in `synapse.storage.types` * No untyped defs in `synapse_port_db` This was by far the most painful. I'm happy to break this up into smaller pieces for review if it's not managable as-is.
* Make `synapse._scripts` pass typechecks (#12421)David Robertson2022-04-084-38/+49
|
* Track device list updates per room. (#12321)Erik Johnston2022-04-041-0/+1
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Move synctl into `synapse._scripts` and expose as an entrypoint (#12140)David Robertson2022-03-041-0/+360
|
* Move scripts directory inside synapse, exposing as setuptools entry_points ↵David Robertson2022-03-028-0/+1851
| | | | | | | | | | | | (#12118) * Two scripts are basically entry_points already * Move and rename scripts/* to synapse/_scripts/*.py * Delete sync_room_to_group.pl * Expose entry points in setup.py * Update linter script and config * Fixup scripts & docs mentioning scripts that moved Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add a flag to the `synapse_review_recent_signups` script to ignore and ↵lukasdenk2022-01-171-3/+15
| | | | filter appservice users. (#11675)
* Add type hints to synapse._scripts (#11297)Patrick Cloke2021-11-102-18/+28
|
* Fixed config parse bug in review_recent_signups (#11191)Samuel Philipp2021-10-271-2/+7
|
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Add script for getting info about recently registered users (#10290)Erik Johnston2021-07-061-0/+175
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Strip trailing / from server_url in register_new_matrix_user (#8823)Dmitry Borodaenko2020-11-261-1/+1
| | | | | | | | | When server URL provided to register_new_matrix_user includes path component (e.g. "http://localhost:8008/"), the command fails with "ERROR! Received 400 Bad Request". Stripping trailing slash from the server_url command argument makes sure combined endpoint URL remains valid. Signed-off-by: Dmitry Borodaenko angdraug@debian.org
* Remove deprecated `/_matrix/client/*/admin` endpoints (#8785)Dirk Klimpel2020-11-251-1/+1
| | | These are now only available via `/_synapse/admin/v1`.
* Remove obsolete __future__ imports (#8337)Jonathan de Jong2020-09-171-2/+0
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-2/+0
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-3/+3
|
* Run Black. (#5482)Amber Brown2019-06-201-11/+13
|
* Fix flake8 (#4519)Amber Brown2019-01-301-2/+2
|
* create support user (#4141)Neil Johnson2018-12-141-3/+16
| | | | | | Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
* Port register_new_matrix_user to Python 3 and add tests (#4085)Amber Brown2018-10-262-0/+215
generated by cgit-magenta 1.5.1 (git 2.48.1) at 2025-07-03 08:48:46 +0000