summary refs log tree commit diff
path: root/synapse/storage/database.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Require Collections as the parameters for simple_* methods. (#11580)Patrick Cloke2021-12-151-18/+10
| | | | Instead of Iterable since the generators are not allowed due to the potential for their re-use.
* checks for generators in database functions (#11564)Richard van der Hoff2021-12-131-4/+46
| | | | | A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).
* Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-131-1/+1
|
* skip some dict munging in event persistence (#11560)Richard van der Hoff2021-12-101-5/+54
| | | | | | Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.
* Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-171-3/+3
|
* Add some background update admin APIs (#11263)Erik Johnston2021-11-081-0/+4
| | | Fixes #11259
* Add metrics to the threadpools (#11178)Erik Johnston2021-11-011-1/+6
|
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-221-1/+5
|
* Fix remove_stale_pushers job on SQLite. (#10843)reivilibre2021-09-201-9/+12
|
* Use `execute_values` more in PostgreSQL (#10754)Erik Johnston2021-09-031-19/+42
| | | `execute_values` is a faster version of `execute_batch`.
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-041-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Allow setting transaction limit for db connections (#10440)Toni Spets2021-08-021-0/+21
| | | | | | | Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
* Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442)Erik Johnston2021-07-221-31/+20
| | | | | | | | Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-151-7/+7
|
* Add script for getting info about recently registered users (#10290)Erik Johnston2021-07-061-1/+1
|
* Fix logging context when opening new DB connection (#10141)Erik Johnston2021-06-081-3/+9
| | | | Fixes #10140
* More database opentracing (#10136)Richard van der Hoff2021-06-071-2/+5
| | | Add a couple of extra logs/spans, to give a bit of a better idea.
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-031-33/+53
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Use the parent's logging context name for runWithConnection. (#9895)Patrick Cloke2021-04-281-1/+3
| | | | | This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-9/+6
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-221-1/+1
| | | This is no longer required, since we have dropped support for Python 3.5.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-4/+4
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-6/+14
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* remove unused param on `make_tuple_comparison_clause`Richard van der Hoff2021-04-081-4/+1
|
* Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-081-55/+7
|
* Add a storage method for returning all current presence from all users (#9650)Andrew Morgan2021-03-251-2/+9
| | | | | | | Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133 Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-1/+1
|
* Refactor to ensure we call check_consistency (#9470)Erik Johnston2021-02-241-12/+4
| | | The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-8/+19
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Update type hints for Cursor to match PEP 249. (#9299)Jonathan de Jong2021-02-051-5/+9
|
* Use execute_batch in more places (#9188)Erik Johnston2021-01-211-0/+6
| | | | | * Use execute_batch in more places * Newsfile
* Use execute_batch instead of executemany in places (#9181)Erik Johnston2021-01-211-3/+2
| | | `execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
* Fix chain cover background update to work with split out event persisters ↵Erik Johnston2021-01-141-0/+11
| | | | (#9115)
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-4/+18
|
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-3/+5
|
* Enable reconnection in DB pool (#8726)Erik Johnston2020-11-121-1/+6
| | | | | | `adbapi.ConnectionPool` let's you turn on auto reconnect of DB connections. This is off by default. As far as I can tell if its not enabled dead connections never get removed from the pool. Maybe helps #8574
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-2/+2
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Pre-emptively fix synapse.storage.types.Connection for future mypy release ↵Jonathan de Jong2020-10-171-1/+1
| | | | | (#8577) Fix the Connection protocol according to typeshed's assertions about sqlite3.Connection
* Use autocommit mode for single statement DB functions. (#8542)Erik Johnston2020-10-141-8/+91
| | | | | | | | | | | | | Autocommit means that we don't wrap the functions in transactions, and instead get executed directly. Introduced in #8456. This will help: 1. reduce the number of `could not serialize access due to concurrent delete` errors that we see (though there are a few functions that often cause serialization errors that we don't fix here); 2. improve the DB performance, as it no longer needs to deal with the overhead of `REPEATABLE READ` isolation levels; and 3. improve wall clock speed of these functions, as we no longer need to send `BEGIN` and `COMMIT` to the DB. Some notes about the differences between autocommit mode and our default `REPEATABLE READ` transactions: 1. Currently `autocommit` only applies when using PostgreSQL, and is ignored when using SQLite (due to silliness with [Twisted DB classes](https://twistedmatrix.com/trac/ticket/9998)). 2. Autocommit functions may get retried on error, which means they can get applied *twice* (or more) to the DB (since they are not in a transaction the previous call would not get rolled back). This means that the functions need to be idempotent (or otherwise not care about being called multiple times). Read queries, simple deletes, and updates/upserts that replace rows (rather than generating new values from existing rows) are all idempotent. 3. Autocommit functions no longer get executed in [`REPEATABLE READ`](https://www.postgresql.org/docs/current/transaction-iso.html) isolation level, and so data can change queries, which is fine for single statement queries.
* Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-071-6/+63
| | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-15/+74
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Catch-up after Federation Outage (split, 1) (#8230)reivilibre2020-09-041-2/+2
| | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Fix type signature in simple_select_one_onecol and friends (#8241)reivilibre2020-09-041-10/+7
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-2/+2
|
* Add an overload for simple_select_one_onecol_txn. (#8235)Patrick Cloke2020-09-021-0/+24
|
* Convert runInteraction to async/await (#8156)Patrick Cloke2020-09-021-15/+14
|
* Add StreamStore to mypy (#8232)Erik Johnston2020-09-021-0/+34
|
* Convert simple_delete to async/await. (#8191)Patrick Cloke2020-08-271-8/+55
|
* simple_search_list_txn should return None, not 0. (#8187)Patrick Cloke2020-08-271-4/+3
|
* Convert additional database methods to async (select list, search, ↵Patrick Cloke2020-08-271-62/+37
| | | | insert_many, delete_*) (#8168)
* Convert simple_update* and simple_select* to async (#8173)Patrick Cloke2020-08-271-14/+15
|
* Convert simple_select_one and simple_select_one_onecol to async (#8162)Patrick Cloke2020-08-261-6/+30
|
* Add more types to synapse.storage.database. (#8127)Patrick Cloke2020-08-201-219/+358
|
* Convert runWithConnection to async. (#8121)Patrick Cloke2020-08-191-14/+13
|
* Convert some of the general database methods to async (#8100)Patrick Cloke2020-08-171-14/+9
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Move some log lines from default logger to sql/transaction loggers (#7952)Andrew Morgan2020-07-281-8/+10
| | | Idea from matrix-org/synapse-dinsic#49
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-2/+1
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-7/+6
|
* Replace device_27_unique_idx bg update with a fg one (#7562)Richard van der Hoff2020-05-261-1/+0
| | | | | | The bg update never managed to complete, because it kept being interrupted by transactions which want to take a lock. Just doing it in the foreground isn't that bad, and is a good deal simpler.
* Remove `exception_to_unicode`Richard van der Hoff2020-05-151-12/+3
| | | | this is a no-op on python 3.
* Fix new flake8 errors (#7470)Erik Johnston2020-05-121-2/+2
|
* use an upsert to update device_lists_outbound_last_successRichard van der Hoff2020-05-061-0/+1
|
* Better type annotations for simple_upsert_txnRichard van der Hoff2020-05-061-30/+43
| | | | most of these params don't really need to be lists.
* bg update to clear out duplicate outbound_device_list_pokes (#7193)Richard van der Hoff2020-04-071-1/+82
| | | | We seem to have some duplicates, which could do with being cleared out.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-6/+5
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Hopefully mypy is happy nowBrendan Abolivier2020-03-101-2/+8
|
* Add some type annotations in `synapse.storage` (#6987)Richard van der Hoff2020-02-271-59/+84
| | | | | I cracked, and added some type definitions in synapse.storage.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-2/+2
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Minor perf fixes to `get_auth_chain_ids`.Erik Johnston2020-02-191-1/+1
|
* Reduce performance logging to DEBUG (#6833)Michael Kaye2020-02-051-1/+1
| | | | | * Reduce tnx performance logging to DEBUG * Changelog.d
* Add database config class (#6513)Erik Johnston2019-12-181-3/+42
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Pass Database into the data storeErik Johnston2019-12-061-22/+16
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-12-061-23/+28
| | | | erikj/make_database_class
* Remove unused varErik Johnston2019-12-061-2/+0
|
* Move background update handling out of storeErik Johnston2019-12-051-0/+3
|
* CommentsErik Johnston2019-12-051-0/+5
|
* Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-051-0/+1485
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-22 06:47:40 +0000