| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
| |
We do this by marking the constraint as deferrable.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Quick & dirty metric for background update status
* Changelog
* Remove debug
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* Actually write to _aborted
---------
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
event_push_{actions,actions_staging,summary} (#15597)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.
For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.
Each table is updated as a separate schema delta to avoid
deadlocks between them.
For SQLite we simply rebuild the table & copy the data.
|
| |
|
|
|
| |
event_push_{actions,actions_staging,summary} (#15437)" (#15580)
This reverts commit a7b3e9ce65335e452de216cb42b9e724e8f3ad1d.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
event_push_{actions,actions_staging,summary} (#15437)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.
For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.
For SQLite we simply rebuild the table & copy the data.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
When Synapse is terminated while running the background update to create
the `receipts_graph` or `receipts_linearized` indexes, the indexes may
be successfully created (or marked as invalid on postgres) while the
background update remains unfinished. When Synapse next starts up, the
background update will fail because the index already exists, or exists
but is invalid on postgres.
Use the existing code to create indices in background updates, since it
handles these edge cases.
Signed-off-by: Sean Quah <seanq@matrix.org>
|
| |
|
|
|
|
|
| |
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
|
| |
|
|
| |
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
|
| |
|
|
|
|
|
| |
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.
This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
|
| |
|
|
| |
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
|
| |
|
|
| |
accidentally introduced and the consequential impact to performance. (#12687)
|
| |
|
|
| |
(#12781)
|
| | |
|
| | |
|
| |
|
|
| |
using the default batch size (#12157)
|
| |
|
| |
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
| |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
|
| |
Instead of having admins poke into the database directly.
Can currently run jobs to populate stats and to populate
the user directory.
|
| |
|
| |
Fixes #11259
|
| | |
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
insert_many, delete_*) (#8168)
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
objects. (#7849)
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
returning a None or an int that we don't use is confusing.
|
| |
|
|
| |
(Almost) everywhere that uses it is happy with an awaitable.
|
| |
|
|
| |
This was only used in a unit test, so let's just inline it in the test.
|
| |
|
|
| |
Ensure good comprehension hygiene using flake8-comprehensions.
|
| |
|
|
|
|
| |
* Add a background update to clear tombstoned rooms from the directory
* use the ABC metaclass
|
| |
|
|
| |
so that bg update routines can be async
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds:
* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)
as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:
1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)
Step `2` is done via a new script located at `scripts-dev/update_database`.
The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
|
| |\
| |
| |
| | |
erikj/cleanup_user_ips_2
|
| | | |
|
| | | |
|
| |/
|
|
| |
This allows checking if a specific background update has completed.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| |/
|
|
| |
People may have queued updates for this, so we can't just delete it.
|
| |
|
|
| |
so that the right thing happens on workers.
|
| |
|
|
|
|
|
|
|
|
|
| |
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572.
|
| |
|
|
|
|
| |
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
|
| |
|
|
| |
what could possibly go wrong
|
| |
|
|
| |
... because the table is virtual
|
| |
|
|
| |
- to make the purge API quicker
|
| |
|
|
|
|
|
|
|
| |
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135.
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
|
| |
|
|
|
|
|
|
|
| |
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
|
| |
|
|
|
|
|
|
| |
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.
I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
|
| | |
|
| | |
|
| | |
|
| | |
|
|
|
The progress for each background update is stored as a JSON blob in the
database. Each background update is broken up into separate batches.
The batch size is automatically tuned to try avoid blocking single
threaded databases for too long.
|
