summary refs log tree commit diff
path: root/synapse/storage/databases/main/schema (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-07264-10027/+0
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Split presence out of master (#9820)Erik Johnston2021-04-232-0/+38
|
* Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-191-0/+18
| | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-142-2/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Remove outdated constraint on remote_media_cache_thumbnails (#9725)Richard van der Hoff2021-04-061-0/+22
| | | | | | | | | | | The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.
* Add a background task to purge unused chain IDs. (#9542)Patrick Cloke2021-03-091-0/+17
| | | | | This is a companion change to apply the fix in #9498 / 922788c6043138165c025c78effeda87de842bab to previously purged rooms.
* Make deleting stale pushers a background update (#9536)Erik Johnston2021-03-031-1/+2
|
* Re-run rejected metadata background update. (#9503)Erik Johnston2021-03-021-0/+9
| | | | | | | It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.
* Fix SQL delta file taking a long time to run (#9516)Erik Johnston2021-03-021-5/+4
| | | | Fixes #9504
* Ensure pushers are deleted for deactivated accounts (#9285)Erik Johnston2021-02-251-0/+21
|
* Add SQL delta for deleting stale pushers (#9479)Erik Johnston2021-02-241-0/+19
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-1/+2
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) ↵Eric Eastwood2021-02-101-10/+0
| | | | | | | (#9003) Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests. Fix #8996
* Merge remote-tracking branch 'origin/release-v1.26.0' into developErik Johnston2021-01-211-0/+18
|\
| * Add schema update to fix existing DBs affected by #9193 (#9195)Erik Johnston2021-01-211-0/+18
| |
* | Use execute_batch in more places (#9188)Erik Johnston2021-01-211-1/+1
|/ | | | | * Use execute_batch in more places * Newsfile
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-182-0/+52
|
* Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-0/+17
|
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-112-0/+68
|
* Remove old tables after schema version bump (#9055)Erik Johnston2021-01-112-0/+34
| | | | | These tables are unused, and can be dropped now the schema version has been bumped.
* Ensure rejected events get added to some metadata tables (#9016)Erik Johnston2021-01-111-0/+17
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Drop last_used column from access_tokens (#9025)Jerin J Titus2021-01-112-0/+78
| | | | | * Dropped last_used column from access_tokens Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-072-0/+43
|
* Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-0/+82
| | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* Drop the unused local_invites table. (#8979)Patrick Cloke2020-12-291-0/+18
| | | This table has been unused since Synapse v1.17.0.
* Allow re-using a UI auth validation for a period of time (#8970)Patrick Cloke2020-12-181-0/+18
|
* UIA: offer only available auth flowsRichard van der Hoff2020-12-021-0/+17
| | | | | | | During user-interactive auth, do not offer password auth to users with no password, nor SSO auth to users with no SSO. Fixes #7559.
* Add missing `ordering` to background updates (#8850)Richard van der Hoff2020-12-014-12/+12
| | | | | | | It's important that we make sure our background updates happen in a defined order, to avoid disasters like #6923. Add an ordering to all of the background updates that have landed since #7190.
* Drop (almost) unused index on event_json (#8845)Richard van der Hoff2020-11-301-0/+19
|
* Improve start time by adding index to e2e_cross_signing_keys (#8694)Erik Johnston2020-11-021-0/+17
| | | We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-0/+17
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Add admin API to list users' local media (#8647)Dirk Klimpel2020-10-271-0/+2
| | | Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
* Move schema file for as_device_stream (#8590)Will Hunt2020-10-201-3/+2
| | | | | | | | | * Move schema file * Add a . * Add matching changelog entry * Fix sqlite
* Drop unused `device_max_stream_id` table (#8589)Vasilis Gerakaris2020-10-191-0/+1
| | | Signed-off-by: Vasilis Gerakaris <vasilis.gerakaris@navarino.gr>
* Send some ephemeral events to appservices (#8437)Will Hunt2020-10-151-0/+18
| | | Optionally sends typing, presence, and read receipt information to appservices.
* Include user agent in user daily visits table (#8503)Neil Johnson2020-10-151-0/+18
| | | Include user agent in user daily visits table.
* Fix message duplication if something goes wrong after persisting the event ↵Erik Johnston2020-10-131-0/+40
| | | | | (#8476) Should fix #3365.
* Only send RDATA for instance local events. (#8496)Erik Johnston2020-10-091-0/+17
| | | | | When pulling events out of the DB to send over replication we were not filtering by instance name, and so we were sending events for other instances.
* Use vector clocks for room stream tokens. (#8439)Erik Johnston2020-10-071-0/+25
| | | | | | | | | | Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down. To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things: 1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and 2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that. The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
* Add support for MSC2697: Dehydrated devices (#8380)Hubert Chathi2020-10-071-0/+20
| | | | This allows a user to store an offline device on the server and then restore it at a subsequent login.
* Add support for MSC2732: olm fallback keys (#8312)Hubert Chathi2020-10-061-0/+24
|
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-0210-41/+24
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* typoMatthew Hodgson2020-09-281-1/+1
|
* Fix schema delta for servers that have not backfilled (#8396)Erik Johnston2020-09-251-1/+3
| | | | | Fixes #8395.
* Fix MultiWriteIdGenerator's handling of restarts. (#8374)Erik Johnston2020-09-241-0/+22
| | | | | | | | | | | | | | | | | | | On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.
* Add experimental support for sharding event persister. Again. (#8294)Erik Johnston2020-09-142-0/+42
| | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* Don't remember `enabled` of deleted push rules and properly return 404 for ↵reivilibre2020-09-091-0/+28
| | | | | | | missing push rules in `.../actions` and `.../enabled` (#7796) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Include method in thumbnail media name (#7124)DeepBlueV7.X2020-09-082-0/+77
| | | | | | | | | | | | | | This fixes an issue where different methods (crop/scale) overwrite each other. This first tries the new path. If that fails and we are looking for a remote thumbnail, it tries the old path. If that still isn't found, it continues as normal. This should probably be removed in the future, after some of the newer thumbnails were generated with the new path on most deployments. Then the overhead should be minimal if the other thumbnails need to be regenerated. Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Rename 'populate_stats_process_rooms_2' background job back to ↵Andrew Morgan2020-09-081-0/+22
| | | | | | | 'populate_stats_process_rooms' again (#8243) Fixes https://github.com/matrix-org/synapse/issues/8238 Alongside the delta file, some changes were also necessary to the codebase to remove references to the now defunct `populate_stats_process_rooms_2` background job. Thankfully the latter doesn't seem to have made it into any documentation yet :)
* Merge branch 'release-v1.20.0' into developRichard van der Hoff2020-09-061-3/+3
|\
| * Unread counts fixes (#8254)Brendan Abolivier2020-09-041-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fixup `ALTER TABLE` database queries Make the new columns nullable, because doing otherwise can wedge a server with a big database, as setting a default value rewrites the table. * Switch back to using the notifications count in the push badge Clients are likely to be confused if we send a push but the badge count is the unread messages one, and not the notifications one. * Changelog
* | In light of #8255, use BIGINTs for destination_rooms (#8256)reivilibre2020-09-041-1/+1
| |
* | Catch up after Federation Outage (split, 2): Track last successful stream ↵reivilibre2020-09-041-0/+21
| | | | | | | | | | ordering after transmission (#8247) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Catch-up after Federation Outage (split, 1) (#8230)reivilibre2020-09-041-0/+42
|/ | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Revert "Add experimental support for sharding event persister. (#8170)" (#8242)Brendan Abolivier2020-09-042-42/+0
| | | | | | | * Revert "Add experimental support for sharding event persister. (#8170)" This reverts commit 82c1ee1c22a87b9e6e3179947014b0f11c0a1ac3. * Changelog
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-0/+26
|
* Add experimental support for sharding event persister. (#8170)Erik Johnston2020-09-022-0/+42
| | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* Allow denying or shadow banning registrations via the spam checker (#8034)Patrick Cloke2020-08-201-0/+25
|
* Add a shadow-banned flag to users. (#8092)Patrick Cloke2020-08-141-0/+18
|
* Remove some unused database functions. (#8085)Patrick Cloke2020-08-141-0/+17
|
* Revert #7736 (#8039)Brendan Abolivier2020-08-061-18/+0
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-05216-0/+8908