summary refs log tree commit diff
path: root/synapse/storage/engines (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add a timeout that aborts any Postgres statement taking more than 1 hour. ↵reivilibre2023-07-031-0/+13
| | | | | | | | | | | | | (#15853) * Add a timeout to Postgres statements * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Speed up SQLite unit test CI (#15334)Erik Johnston2023-03-301-1/+16
| | | Tests now take 40% of the time.
* Use mypy 1.0 (#15052)David Robertson2023-02-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Make sqlite database migrations transactional again, part two (#14926)Sean Quah2023-01-313-5/+12
| | | | | | | | | | | | | | | | #14910 fixed the regression introduced by #13873 where sqlite database migrations would no longer run inside a transaction. However, it committed the transaction before Synapse updated its bookkeeping of which migrations have been run, which means that migrations may be run again after they have completed successfully. Leave the transaction open at the end of `executescript`, to restore the old, correct behaviour. Also make the PostgreSQL behaviour consistent with SQLite. Fixes #14909. Signed-off-by: Sean Quah <seanq@matrix.org>
* Make sqlite database migrations transactional again (#14910)Sean Quah2023-01-252-2/+6
| | | | | | | | | | #13873 introduced a regression which causes sqlite database migrations to no longer run inside a transaction. Wrap them in a transaction again, to avoid database corruption when migrations are interrupted. Fixes #14909. Signed-off-by: Sean Quah <seanq@matrix.org>
* Enable `--warn-redundant-casts` option in mypy (#14671)David Robertson2022-12-121-1/+1
| | | | | | | | | | | | | | | | | | | * Enable `--warn-redundant-casts` option in mypy Doesn't do much but helps me sleep better at night. * Changelog * Fix name of the ignore * Fix one more missed cast Not sure why I didn't see this one locally, maybe I needed a poetry update * Remove old comment Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Drop support for Postgres 10 in full text search code. (#14397)Patrick Cloke2022-11-091-16/+0
|
* Remove support for PostgreSQL 10 (#14392)Sean Quah2022-11-081-2/+2
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Unified search query syntax using the full-text search capabilities of the ↵James Salter2022-10-251-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | underlying DB. (#11635) Support a unified search query syntax which leverages more of the full-text search of each database supported by Synapse. Supports, with the same syntax across Postgresql 11+ and Sqlite: - quoted "search terms" - `AND`, `OR`, `-` (negation) operators - Matching words based on their stem, e.g. searches for "dog" matches documents containing "dogs". This is achieved by - If on postgresql 11+, pass the user input to `websearch_to_tsquery` - If on sqlite, manually parse the query and transform it into the sqlite-specific query syntax. Note that postgresql 10, which is close to end-of-life, falls back to using `phraseto_tsquery`, which only supports a subset of the features. Multiple terms separated by a space are implicitly ANDed. Note that: 1. There is no escaping of full-text syntax that might be supported by the database; e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this as accidental functionality and depend on something we don't guarantee. 2. English text is assumed for stemming. To support other languages, either the target language needs to be known at the time of indexing the message (via room metadata, or otherwise), or a separate index for each language supported could be created. Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
* Enable WAL for SQLite (#13897)asymmetric2022-10-251-0/+4
| | | Signed-off-by: Lorenzo Manacorda <lorenzo@mailbox.org>
* Snapshot schema 72 (#13873)David Robertson2022-09-263-4/+52
| | | Including another batch of fixes to the schema dump script
* Require SQLite >= 3.27.0 (#13760)David Robertson2022-09-093-26/+2
|
* Use dummy fallback engines if imports fail (#12979)David Robertson2022-06-072-18/+44
|
* Tidy up and type-hint the database engine modules (#12734)David Robertson2022-05-174-94/+108
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Refuse to start if DB has an unsafe locale (#12262)Shay2022-03-231-15/+30
|
* Use psycopg2 type stubs (#12269)David Robertson2022-03-231-1/+1
|
* Db txn set isolation level (#11799)Nick Barrett2022-01-253-5/+50
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* Fix the inbound PDU metric (#10279)Erik Johnston2021-06-303-0/+16
| | | This broke in #10272
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-144-4/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-083-25/+5
|
* Require py36 and Postgres 9.6Richard van der Hoff2021-04-081-2/+2
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-163-9/+8
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add experimental support for PyPy. (#9123)Jonathan de Jong2021-02-042-6/+8
| | | | * Adds proper dependencies. * Minor fixes in database layer.
* Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-073-1/+36
| | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
* Use SequenceGenerator for state group ID allocationRichard van der Hoff2020-07-163-25/+0
|
* Fix incorrect error message when database CTYPE was set incorrectly. (#7760)Richard van der Hoff2020-07-011-1/+1
|
* Persist user interactive authentication sessions (#7302)Patrick Cloke2020-04-301-0/+1
| | | | | By persisting the user interactive authentication sessions to the database, this fixes situations where a user hits different works throughout their auth session and also allows sessions to persist through restarts of Synapse.
* Only import sqlite3 when type checking (#7155)David Vo2020-03-271-2/+5
| | | | Fixes: #7127 Signed-off-by: David Vo <david@vovo.id.au>
* Add some type annotations in `synapse.storage` (#6987)Richard van der Hoff2020-02-274-23/+117
| | | | | I cracked, and added some type definitions in synapse.storage.
* Updated warning for incorrect database collation/ctype (#6985)Uday Bansal2020-02-261-3/+7
| | | Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
* Warn if postgres database has non-C locale. (#6734)Erik Johnston2020-01-282-0/+47
| | | As using non-C locale can cause issues on upgrading OS.
* Refuse to start if sqlite is older than 3.11.0Richard van der Hoff2020-01-092-4/+7
|
* Check postgres version in check_databaseRichard van der Hoff2020-01-091-13/+12
| | | | | this saves doing it on each connection, and will allow us to pass extra options in.
* Modify check_database to take a connection rather than a cursorRichard van der Hoff2020-01-092-9/+10
| | | | We might not need the cursor at all.
* Add database config class (#6513)Erik Johnston2019-12-181-3/+13
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Merge pull request #6156 from matrix-org/erikj/postgres_anyErik Johnston2019-10-102-0/+12
|\ | | | | Use Postgres ANY for selecting many values.
| * Fix SQLiteErik Johnston2019-10-101-1/+1
| |
| * Remove add_in_list_sql_clauseErik Johnston2019-10-102-0/+12
| |
* | Disable bytes usage with postgresErik Johnston2019-10-081-0/+7
|/ | | | | | | | More often than not passing bytes to `txn.execute` is a bug (where we meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is set to `ESCAPE`. However, this is a bit of a footgun so we want to instead error when this happens, and force using `bytearray` if we actually want to use bytes.
* Fix `devices_last_seen` background update.Erik Johnston2019-09-302-0/+15
| | | | Fixes #6134.
* Run Black. (#5482)Amber Brown2019-06-201-1/+1
|
* Remove Postgres 9.4 support (#5448)Amber Brown2019-06-181-2/+6
|
* Run black on the rest of the storage module (#4996)Amber Brown2019-04-033-22/+10
|
* Add database version to phonehome stats. (#4753)Richard van der Hoff2019-02-272-0/+34
|
* Fix UPSERTs on SQLite 3.24+ (#4477)Amber Brown2019-01-281-7/+3
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-01-251-2/+1
|\ | | | | | | erikj/sqlite_native_upsert
| * Don't require sqlite3 when using postgres (#4466)Andrew Morgan2019-01-251-2/+1
| |
* | Disable native upserts for sqlite, as they don't workErik Johnston2019-01-251-3/+7
|/
* Use native UPSERTs where possible (#4306)Amber Brown2019-01-243-1/+24
|
* Port storage/ to Python 3 (#3725)Amber Brown2018-08-311-2/+7
|
* run isortAmber Brown2018-07-092-6/+5
|
* Fix pep8 error on psycopg2cffi hackRichard van der Hoff2018-04-101-6/+5
|
* Use psycopg2cffi module instead of psycopg2 if running on pypyVincent Breitmoser2018-04-101-1/+7
| | | | | | | The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.
* Store state groups separately from events (#2784)Erik Johnston2018-02-062-0/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints
* Optionally make committing to postgres asynchronous.Mark Haines2016-06-203-3/+14
| | | | | Useful when running tests when you don't care whether the server will lose data that it claims that it has committed.
* Don't require config to create databaseErik Johnston2016-04-063-20/+7
|
* Mark AS users with their AS's IDDaniel Wagner-Hall2016-02-113-6/+9
|
* Fix flake8 warnings for new flake8Daniel Wagner-Hall2016-02-021-1/+1
|
* copyrightsMatthew Hodgson2016-01-074-4/+4
|
* Implement rank function for SQLite FTSErik Johnston2015-10-231-0/+27
|
* rename schema_prepare to prepare_databaseErik Johnston2015-10-132-2/+2
|
* Expose error more nicelyErik Johnston2015-10-132-2/+2
|
* Split out the schema preparation and update logic into its own moduleErik Johnston2015-10-132-2/+4
|
* Remove race conditionErik Johnston2015-05-142-0/+4
|
* Don't call 'encode_parameter' no-opErik Johnston2015-05-052-6/+0
|
* Fix bug where we reconnected to the database on every query.Erik Johnston2015-05-011-1/+1
|
* Make postgres database error slightly more helpfulErik Johnston2015-04-291-1/+2
|
* Use __all__ instead of assert to stop pyflakes from warning about unused ↵Erik Johnston2015-04-291-2/+3
| | | | import in __init__
* Appease PEP8Erik Johnston2015-04-291-0/+2
|
* Check that postgres database has correct charset setErik Johnston2015-04-294-0/+33
|
* Shuffle operations so that locking upsert happens last in the txn. This ↵Erik Johnston2015-04-272-0/+6
| | | | ensures the lock is held for the least amount of time possible.
* Handle the fact that postgres databases can be restarted from under usErik Johnston2015-04-272-0/+6
|
* PEP8Erik Johnston2015-04-161-2/+0
|
* Go back to storing JSON in TEXTErik Johnston2015-04-162-8/+0
|
* Remove mysql/maria supportErik Johnston2015-04-152-52/+0
|
* Correctly identify deadlocksErik Johnston2015-04-151-1/+1
|
* Add postgres database engineErik Johnston2015-04-141-0/+44
|
* Add support for postgres instead of mysql. Change sql accourdingly. blob + ↵Erik Johnston2015-04-141-0/+2
| | | | varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
* Use case sensitive collationsErik Johnston2015-04-101-1/+1
|
* Handle the fact that in sqlite binary data might be stored as unicode or bytesErik Johnston2015-04-102-1/+11
|
* Fix maria engine to correctly recognize deadlocksErik Johnston2015-04-081-2/+2
|
* Retry on deadlockErik Johnston2015-04-072-0/+8
|
* Fix unicode supportErik Johnston2015-04-023-1/+24
|
* Make work in both Maria and SQLite. Fix testsErik Johnston2015-04-013-0/+90
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-15 13:30:47 +0000