summary refs log tree commit diff
path: root/synapse/storage/prepare_database.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add type hints to schema deltas (#15497)Patrick Cloke2023-04-271-3/+5
| | | | | | | | | | Cleans-up the schema delta files: * Removes no-op functions. * Adds missing type hints to function parameters. * Fixes any issues with type hints. This also renames one (very old) schema delta to avoid a conflict that mypy complains about.
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-221-2/+2
|
* Do not always start a db txn on Postgres (#14840)Andrew Morgan2023-02-091-4/+9
|
* Snapshot schema 72 (#13873)David Robertson2022-09-261-4/+4
| | | Including another batch of fixes to the schema dump script
* Use dummy fallback engines if imports fail (#12979)David Robertson2022-06-071-2/+1
|
* Reduce the number of "untyped defs" (#12716)David Robertson2022-05-121-1/+1
|
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Minor updates, and docs, for schema delta files (#11823)Richard van der Hoff2022-01-251-3/+6
| | | | | | | | | * Make functions in python deltas optional It's annoying to always have to write stubs for these. * Documentation for delta files * changelog
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-3/+3
|
* Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-18/+22
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix typo in comment from #11255. (#11276)Patrick Cloke2021-11-081-1/+1
|
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-11/+12
| | | Fixes #11252
* Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-0/+2
| | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-0/+4
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-1/+1
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-231-1/+1
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-3/+3
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-151-3/+3
|
* Support for database schema version ranges (#9933)Richard van der Hoff2021-06-111-39/+82
| | | This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-071-23/+25
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-221-2/+1
| | | This is no longer required, since we have dropped support for Python 3.5.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Deprecate imp (#9718)Cristina2021-03-311-3/+8
| | | | | Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-2/+6
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Update type hints for Cursor to match PEP 249. (#9299)Jonathan de Jong2021-02-051-2/+2
|
* Remove old tables after schema version bump (#9055)Erik Johnston2021-01-111-3/+0
| | | | | These tables are unused, and can be dropped now the schema version has been bumped.
* Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-1/+1
| | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* Handle updating schema version without any deltas. (#9033)Erik Johnston2021-01-071-7/+10
| | | | This can happen when using a split out state database and we've upgraded the schema version without there being any changes in the state schema.
* Add additional type hints to the storage module. (#8980)Patrick Cloke2020-12-301-48/+56
|
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-23/+10
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Merge remote-tracking branch 'origin/release-v1.20.0' into developRichard van der Hoff2020-09-081-17/+70
|\
| * Add more logging to debug slow startup (#8264)Richard van der Hoff2020-09-071-0/+10
| | | | | | | | I'm hoping this will provide some pointers for debugging https://github.com/matrix-org/synapse/issues/7968.
| * Refuse to upgrade database on worker processes (#8266)Richard van der Hoff2020-09-071-17/+61
| |
* | Run database updates in a transaction (#8265)Richard van der Hoff2020-09-071-5/+22
|/ | | Fixes: #6467
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-25/+23
|
* Fix bug in account data replication stream. (#7656)Erik Johnston2020-06-091-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | * Ensure account data stream IDs are unique. The account data stream is shared between three tables, and the maximum allocated ID was tracked in a dedicated table. Updating the max ID happened outside the transaction that allocated the ID, leading to a race where if the server was restarted then the same ID could be allocated but the max ID failed to be updated, leading it to be reused. The ID generators have support for tracking across multiple tables, so we may as well use that instead of a dedicated table. * Fix bug in account data replication stream. If the same stream ID was used in both global and room account data then the getting updates for the replication stream would fail due to `heapq.merge(..)` trying to compare a `str` with a `None`. (This is because you'd have two rows like `(534, '!room')` and `(534, None)` from the room and global account data tables). Fix is just to order by stream ID, since we don't rely on the ordering beyond that. The bug where stream IDs can be reused should be fixed now, so this case shouldn't happen going forward. Fixes #7617
* Fix incorrect placeholder syntax in database prepartion code (#7575)Andrew Morgan2020-05-271-3/+2
| | | We were using `logger` syntax which isn't supported by `Exception`s.
* Replace device_27_unique_idx bg update with a fg one (#7562)Richard van der Hoff2020-05-261-4/+9
| | | | | | The bg update never managed to complete, because it kept being interrupted by transactions which want to take a lock. Just doing it in the foreground isn't that bad, and is a good deal simpler.
* Support any process writing to cache invalidation stream. (#7436)Erik Johnston2020-05-071-0/+2
|
* Only run one background update at a timeRichard van der Hoff2020-03-311-1/+1
|
* Sanity-check database before running upgrades (#6982)Richard van der Hoff2020-02-251-2/+13
| | | | | | Some of the database deltas rely on `config.server_name` being set correctly, so we should check that it is before running the deltas. Fixes #6870.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-3/+3
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Warn if postgres database has non-C locale. (#6734)Erik Johnston2020-01-281-0/+5
| | | As using non-C locale can cause issues on upgrading OS.
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Split state groups into a separate data store (#6296)Erik Johnston2019-12-201-1/+1
|
* Explode on duplicate delta file names. (#6565)Erik Johnston2019-12-191-0/+18
|
* Improve diagnostics on database upgrade failure (#6570)Richard van der Hoff2019-12-191-1/+4
| | | | `Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst has anything useful.
* Add database config class (#6513)Erik Johnston2019-12-181-4/+3
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Clean up prepare_database.py a bit and add commentsErik Johnston2019-10-221-7/+30
|
* Fix schema management to work with multiple data stores.Erik Johnston2019-10-211-41/+108
|
* Code style (Black)Olivier Wilkinson (reivilibre)2019-08-271-3/+1
| | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Allow schema deltas to be engine-specificOlivier Wilkinson (reivilibre)2019-08-271-5/+21
| | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Add membership column to current_state_events table.Erik Johnston2019-07-181-1/+1
| | | | | It turns out that doing a join is surprisingly expensive for the DB to do when room_membership table is larger than the disk cache.
* Fix "Unexpected entry in 'full_schemas'" log warning (#5509)Richard van der Hoff2019-06-211-1/+1
| | | | | There is a README.txt which always sets off this warning, which is a bit alarming when you first start synapse. I don't think we need to warn about this.
* Run Black. (#5482)Amber Brown2019-06-201-6/+7
|
* Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1Erik Johnston2019-06-061-2/+14
|\ | | | | Make a full SQL schema
| * fix schemasAmber Brown2019-06-031-2/+2
| |
| * prepareAmber Brown2019-06-031-2/+14
| |
* | Add ability to perform password reset via email without trusting the ↵Andrew Morgan2019-06-061-1/+1
|/ | | | | | | | | | | | identity server (#5377) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368
* Remove presence lists (#4989)Neil Johnson2019-04-031-1/+1
| | | Remove presence list support as per MSC 1819
* Run black on the rest of the storage module (#4996)Amber Brown2019-04-031-38/+29
|
* Drop sent_transactionsAaron Raimist2018-12-011-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Ignore __pycache__ directories in schema delta dirRichard van der Hoff2018-11-201-1/+1
| | | | Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
* Add state_group index to event_to_state_groupsErik Johnston2018-10-041-1/+1
| | | | | This is needed to efficiently check for unreferenced state groups during purge.
* api into monthly_active_users tableNeil Johnson2018-07-311-1/+1
|
* run isortAmber Brown2018-07-091-1/+0
|
* bump versionNeil Johnson2018-05-291-1/+1
|
* Generate user daily statsNeil Johnson2018-04-251-1/+1
|
* Add joinability for groupsDavid Baker2018-03-281-1/+2
| | | | | Adds API to set the 'joinable' flag, and corresponding flag in the table.
* Keep track of last access time for local mediaErik Johnston2018-01-171-1/+1
|
* DB schema interface for password auth providersRichard van der Hoff2017-10-311-0/+70
| | | | | Provide an interface by which password auth providers can register db schema files to be run at startup
* Bump schema version to 46Luke Barnard2017-10-261-1/+1
|
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* Fix schema delta versionsErik Johnston2017-10-111-1/+1
|
* Delete expired url cache dataErik Johnston2017-09-281-1/+1
|
* Add DB schema for tracking users who share roomsErik Johnston2017-06-151-1/+1
|
* Add background task to clear out old event_authErik Johnston2017-05-241-1/+1
|
* User Cursor.__iter__ instead of fetchallErik Johnston2017-03-231-1/+1
| | | | This prevents unnecessary construction of lists
* Add stream_id index to device_lists_outbound_pokesErik Johnston2017-02-281-1/+1
| | | | As this is used for replication streaming
* Change device_inbox stream index to include userErik Johnston2017-01-101-1/+1
| | | | | This makes fetching the nost recently changed users much tricker, and brings it in line with e.g. presence_stream indices.
* Handle sending events and device messages over federationErik Johnston2016-11-171-1/+1
|
* Replace postgres GIN with GISTErik Johnston2016-11-031-1/+1
| | | | | This is because GIN can be slow to write too, especially when the table gets large.
* Bump schema versionErik Johnston2016-10-171-1/+1
|
* Readd entries to public_room_list_stream that were deletedErik Johnston2016-09-211-1/+1
|
* Log delta files we're applyingErik Johnston2016-09-081-2/+2
|
* Move to storing state_groups_state as deltasErik Johnston2016-09-011-1/+1
|
* Add some invalidations to a cache_streamErik Johnston2016-08-151-1/+1
|
* Track approximate last access time for remote mediaErik Johnston2016-06-291-1/+1
|
* Add /report endpointErik Johnston2016-05-041-1/+1
|
* Remove some unused functions (#711)Mark Haines2016-04-081-12/+0
| | | | | | | | * Remove some unused functions * get_room_events_stream is only used in tests * is_exclusive_room might actually be something we want
* Don't require config to create databaseErik Johnston2016-04-061-43/+21
|
* Store invites in a separate table.Erik Johnston2016-04-041-1/+1
|
* Mark AS users with their AS's IDDaniel Wagner-Hall2016-02-111-7/+8
|
* Change a log from debug to infoErik Johnston2016-02-091-1/+1
|
* Change event_push_actions_rm_tokens schemaErik Johnston2016-02-031-1/+1
|
* Allow guests to upgrade their accountsDaniel Wagner-Hall2016-01-051-2/+2
|
* Bump schema version.Mark Haines2015-12-041-1/+1
| | | | As we released version 26 in v0.11.1
* s/private_user_data/account_data/Mark Haines2015-11-181-1/+1
|
* Move FTS to delta 25Erik Johnston2015-10-221-1/+1
|
* rename schema_prepare to prepare_databaseErik Johnston2015-10-131-0/+395
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-24 17:23:56 +0000