summary refs log tree commit diff
path: root/synapse/storage/background_updates.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add a unique index to `state_group_edges` to prevent duplicates being ↵reivilibre2022-05-191-0/+15
| | | | accidentally introduced and the consequential impact to performance. (#12687)
* Do not keep going if there are 5 back-to-back background update failures. ↵reivilibre2022-05-181-0/+8
| | | | (#12781)
* Another batch of type annotations (#12726)David Robertson2022-05-131-5/+14
|
* Add config settings for background update parameters (#11980)Shay2022-03-111-14/+25
|
* Fix a bug in background updates wherein background updates are never run ↵Shay2022-03-071-3/+5
| | | | using the default batch size (#12157)
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-291-23/+169
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-1/+1
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add an admin API to run background jobs. (#11352)Dirk Klimpel2021-11-191-0/+2
| | | | | | Instead of having admins poke into the database directly. Can currently run jobs to populate stats and to populate the user directory.
* Add some background update admin APIs (#11263)Erik Johnston2021-11-081-18/+47
| | | Fixes #11259
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-151-9/+7
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-1/+1
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-4/+4
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add additional type hints to the storage module. (#8980)Patrick Cloke2020-12-301-47/+64
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-2/+2
|
* Convert additional databases to async/await part 3 (#8201)Patrick Cloke2020-09-011-2/+2
|
* Convert additional database methods to async (select list, search, ↵Patrick Cloke2020-08-271-4/+5
| | | | insert_many, delete_*) (#8168)
* Use the JSON encoder without whitespace in more places. (#8124)Patrick Cloke2020-08-201-3/+2
|
* Convert misc database code to async (#8087)Patrick Cloke2020-08-141-9/+5
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-11/+11
|
* Consistently use `db_to_json` to convert from database values to JSON ↵Patrick Cloke2020-07-161-1/+4
| | | | objects. (#7849)
* Update docstring per review commentsRichard van der Hoff2020-04-031-1/+1
|
* review commentRichard van der Hoff2020-04-021-1/+1
|
* Only run one background update at a timeRichard van der Hoff2020-03-311-26/+48
|
* Make do_next_background_update return a boolRichard van der Hoff2020-03-311-7/+5
| | | | returning a None or an int that we don't use is confusing.
* Make `has_completed_background_updates` asyncRichard van der Hoff2020-03-311-4/+3
| | | | (Almost) everywhere that uses it is happy with an awaitable.
* Remove unused `start_background_update`Richard van der Hoff2020-03-311-21/+0
| | | | This was only used in a unit test, so let's just inline it in the test.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Add a background update to clear tombstoned rooms from the directory (#6648)Richard van der Hoff2020-01-071-0/+15
| | | | | | * Add a background update to clear tombstoned rooms from the directory * use the ABC metaclass
* Async/await for background updates (#6647)Richard van der Hoff2020-01-071-16/+20
| | | | so that bg update routines can be async
* Pass Database into the data storeErik Johnston2019-12-061-1/+1
|
* Move background update handling out of storeErik Johnston2019-12-051-8/+7
|
* Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-051-8/+8
|
* Remove underscore from SQLBaseStore functionsErik Johnston2019-12-041-7/+7
|
* Add CI for synapse_port_db (#6140)Brendan Abolivier2019-10-281-3/+6
| | | | | | | | | | | | | | | | | | This adds: * a test sqlite database * a configuration file for the sqlite database * a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`) as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that: 1. installs Synapse 2. updates the test sqlite database to the latest schema and runs background updates on it 3. creates an empty postgresql database 4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage) Step `2` is done via a new script located at `scripts-dev/update_database`. The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-251-1/+1
|\ | | | | | | erikj/cleanup_user_ips_2
| * Fix loggingBrendan Abolivier2019-09-241-1/+1
| |
* | Review commentsErik Johnston2019-09-251-4/+1
| |
* | Add has_completed_background_updateErik Johnston2019-09-241-1/+24
|/ | | | This allows checking if a specific background update has completed.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-10/+10
|
* Run Black. (#5482)Amber Brown2019-06-201-1/+1
|
* Run black on the rest of the storage module (#4996)Amber Brown2019-04-031-15/+20
|
* Migrate the user directory initial population to a background task (#4864)Amber Brown2019-03-191-2/+6
|
* Fix typosAndrew Morgan2019-01-221-1/+1
|
* Run things as background processesRichard van der Hoff2018-07-181-2/+8
| | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
* run isortAmber Brown2018-07-091-5/+5
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+2
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-221-2/+1
|
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+1
|\
| * Replace ujson with simplejsonErik Johnston2018-03-151-1/+1
| |
* | Reinstate event_search_postgres_gist handlerRichard van der Hoff2018-02-021-0/+19
|/ | | | People may have queued updates for this, so we can't just delete it.
* Check database in has_completed_background_updatesRichard van der Hoff2017-11-221-2/+25
| | | | so that the right thing happens on workers.
* Fix error on sqlite 3.7Richard van der Hoff2017-11-211-1/+11
| | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.
* Make __init__ consitstent across Store heirarchyRichard van der Hoff2017-11-131-2/+2
| | | | | | Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* Don't create event_search index on sqliteRichard van der Hoff2017-05-111-3/+10
| | | | ... because the table is virtual
* Add an index to event_searchRichard van der Hoff2017-05-111-3/+7
| | | | - to make the purge API quicker
* Fix bgupdate error if index already exists (#2167)Richard van der Hoff2017-04-271-30/+53
| | | | | | | | | When creating a new table index in the background, guard against it existing already. Fixes https://github.com/matrix-org/synapse/issues/2135. Also, make sure we restore the autocommit flag when we're done, otherwise we get more failures from other operations later on. Fixes https://github.com/matrix-org/synapse/issues/1890 (hopefully).
* Remove broken use of clock.call_laterRichard van der Hoff2017-03-181-12/+3
| | | | | | | | | background_updates was using `call_later` in a way that leaked the logcontext into the reactor. We could have rewritten it to do it properly, but given that we weren't using the fancier facilities provided by `call_later`, we might as well just use `async.sleep`, which does the logcontext stuff properly.
* Add WHERE clause support to index creationErik Johnston2016-09-121-9/+15
|
* Reindex state_groups_state after pruningErik Johnston2016-09-081-2/+4
|
* fix: defer.returnValue takes one argumentMark Haines2016-07-251-1/+1
|
* background updates: fix assert againRichard van der Hoff2016-07-251-1/+1
|
* background updates: Fix assertion to do somethingRichard van der Hoff2016-07-251-2/+2
|
* Fix background_update testsRichard van der Hoff2016-07-251-7/+20
| | | | | | A bit of a cleanup for background_updates, and make sure that the real background updates have run before we start the unit tests, so that they don't interfere with the tests.
* Create index on user_ips in the backgroundRichard van der Hoff2016-07-221-7/+66
| | | | | | | | user_ips is kinda big, so really we want to add the index in the background once we're running. Replace the schema delta with one which will do that. I've done this in a way that's reasonably easy to reuse as there a few other indexes I need, and I don't suppose they will be the last.
* Simplify query and handle finishing correctlyErik Johnston2016-04-221-1/+2
|
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Fix the background updateMark Haines2015-11-111-5/+8
|
* Run the background updates when starting synapse.Mark Haines2015-11-101-7/+50
|
* Add storage module for tracking background updates.Mark Haines2015-11-091-0/+210
The progress for each background update is stored as a JSON blob in the database. Each background update is broken up into separate batches. The batch size is automatically tuned to try avoid blocking single threaded databases for too long.