summary refs log tree commit diff
path: root/scripts/synapse_port_db (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move scripts directory inside synapse, exposing as setuptools entry_points ↵David Robertson2022-03-021-1253/+0
| | | | | | | | | | | | (#12118) * Two scripts are basically entry_points already * Move and rename scripts/* to synapse/_scripts/*.py * Delete sync_room_to_group.pl * Expose entry points in setup.py * Update linter script and config * Fixup scripts & docs mentioning scripts that moved Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-3/+4
| | | | * Require latest matrix-common * Use the common function
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+4
| | | | users. (#11655)
* Add index to `local_group_updates.stream_id` (#11231)Erik Johnston2021-11-021-0/+2
| | | | This should speed up startup times and generally increase performance of groups.
* Remove unnecessary list comprehension in `synapse_port_db` to fix linting in ↵reivilibre2021-10-111-1/+1
| | | | CI (#11043)
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-061-2/+2
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Add a partial index to `presence_stream` to speed up startups (#10748)Sean2021-09-031-0/+2
| | | Signed-off-by: Sean Quah <seanq@element.io>
* MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-241-1/+3
| | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* Fix off-by-one-error in synapse_port_db (#9991)Richard van der Hoff2021-05-241-1/+1
| | | fixes #9979
* Run `black` on the scripts (#9981)Richard van der Hoff2021-05-141-21/+25
| | | | Turns out these scripts weren't getting linted.
* Fix port_db on empty db (#9930)Richard van der Hoff2021-05-101-8/+10
| | | | ... and test it.
* Split presence out of master (#9820)Erik Johnston2021-04-231-2/+5
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Fix SQL delta file taking a long time to run (#9516)Erik Johnston2021-03-021-0/+2
| | | | Fixes #9504
* Include newly added sequences in the port DB script. (#9449)Patrick Cloke2021-02-231-24/+41
| | | And ensure the consistency of `event_auth_chain_id`.
* Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-1/+1
|
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-071-0/+27
|
* Fix port script so that it can be run again after failure. (#8755)Erik Johnston2020-11-131-30/+54
| | | | | | | If the script fails (or is CTRL-C'ed) between porting some of the events table and copying of the sequences then the port script will immediately die if run again due to the postgres DB having inconsistencies between sequences and tables. The fix is to move the porting of sequences to before porting the tables, so that there is never a period where the Postgres DB is inconsistent. To do that we need to change how we port the sequences so that it calculates the values from the SQLite DB rather than the Postgres DB. Fixes #8619
* Fix port script fails when DB has no backfilled events. (#8729)Erik Johnston2020-11-111-7/+5
| | | Fixes #8618
* Fix port script to handle foreign key constraints (#8730)Erik Johnston2020-11-111-6/+62
|
* Improve start time by adding index to e2e_cross_signing_keys (#8694)Erik Johnston2020-11-021-0/+2
| | | We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
* Cast errors generated during synapse_port_db to str (#8585)Andrew Morgan2020-10-201-2/+3
| | | | | | | | | I noticed in https://github.com/matrix-org/synapse/issues/8575 that the `end_error` variable in `synapse_port_db` is set to an `Exception`, even though later we expect it to be a `str`. This PR simply casts an exception raised to a string. I'm doing this instead of having `end_error` be of type exception as we explicitly set `end_error` to a str here: https://github.com/matrix-org/synapse/blob/d25eb8f3709965d0face01a041d5292490bf0139/scripts/synapse_port_db#L542-L547 This whole file could probably use some heavy refactoring, but until then at least this fix will prevent exception contents from being hidden from us and users.
* Add support for MSC2732: olm fallback keys (#8312)Hubert Chathi2020-10-061-0/+1
|
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-1/+1
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410)Dagfinn Ilmari Mannsåker2020-09-281-0/+1
| | | | | | This table was created in #8034 (1.20.0). It references `ui_auth_sessions`, which is ignored, so this one should be too. Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
* Add new sequences to port DB script (#8387)Erik Johnston2020-09-241-0/+24
|
* Mark the shadow_banned column as boolean in synapse_port_db. (#8386)Patrick Cloke2020-09-241-0/+1
|
* Revert #7736 (#8039)Brendan Abolivier2020-08-061-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-39/+39
|
* Re-implement unread counts (#7736)Brendan Abolivier2020-07-291-1/+1
|
* Use a postgres sequence to generate guest user IDsRichard van der Hoff2020-07-161-1/+11
|
* Sync ignored table names in synapse_port_db to current database schema (#7717)Patrick Cloke2020-06-231-4/+12
|
* Allow local media to be marked as safe from being quarantined. (#7718)Patrick Cloke2020-06-221-0/+1
|
* Ignore the UI Auth sessions when porting from sqlite to postgresql (#7711)Patrick Cloke2020-06-171-7/+12
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-3/+1
|
* Add option to move event persistence off master (#7517)Erik Johnston2020-05-221-0/+3
|
* Support any process writing to cache invalidation stream. (#7436)Erik Johnston2020-05-071-2/+2
|
* Port synapse_port_db to async/await (#6718)Brendan Abolivier2020-01-211-82/+112
| | | | | | | | | | | | | | | | | | | | | | | | | | * Raise an exception if there are pending background updates So we return with a non-0 code * Changelog * Port synapse_port_db to async/await * Port update_database to async/await * Add version string to mocked homeservers * Remove unused imports * Convert overseen bits to async/await * Fixup logging contexts * Fix imports * Add a way to print an error without raising an exception * Incorporate review
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Refuse to start if sqlite is older than 3.11.0Richard van der Hoff2020-01-091-4/+12
|
* Modify check_database to take a connection rather than a cursorRichard van der Hoff2020-01-091-18/+7
| | | | We might not need the cursor at all.
* Add a background update to clear tombstoned rooms from the directory (#6648)Richard van der Hoff2020-01-071-0/+5
| | | | | | * Add a background update to clear tombstoned rooms from the directory * use the ABC metaclass
* Split state groups into a separate data store (#6296)Erik Johnston2019-12-201-4/+4
|
* Add database config class (#6513)Erik Johnston2019-12-181-34/+24
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Fix port db scriptErik Johnston2019-12-061-34/+2
|
* Fix DB scriptsErik Johnston2019-12-051-5/+9
|
* Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-051-36/+37
|
* Merge pull request #6464 from matrix-org/erikj/make_public_sql_baseErik Johnston2019-12-051-9/+9
|\ | | | | Clean up SQLBaseStore private function usage
| * Remove underscore from SQLBaseStore functionsErik Johnston2019-12-041-9/+9
| |
* | Fix background updates for synapse_port_dbBrendan Abolivier2019-12-041-0/+2
| |
* | Make synapse_port_db exit with a non-0 code if something failedBrendan Abolivier2019-12-041-0/+1
|/
* Fix error when using synapse_port_db on a vanilla synapse dbAndrew Morgan2019-12-021-1/+4
|
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-3/+3
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Merge pull request #6247 from matrix-org/babolivier/port_db_columnBrendan Abolivier2019-10-241-0/+1
|\ | | | | Add new boolean column to synapse_port_db
| * Add new boolean column to synapse_port_dbBrendan Abolivier2019-10-241-0/+1
| |
* | Fix up changelog v1.5.0rc1Andrew Morgan2019-10-241-7/+9
|/
* Fix import paths in synapse_port_dbBrendan Abolivier2019-10-231-11/+19
|
* Make synapse_port_db correctly create indexes (#6102)Brendan Abolivier2019-10-231-52/+130
| | | | | | | Make `synapse_port_db` correctly create indexes in the PostgreSQL database, by having it run the background updates on the database before migrating the data. To ensure we're migrating the right data, also block the port if the SQLite3 database still has pending or ongoing background updates. Fixes #4877
* Add missing BOOLEAN_COLUMNs to synapse_port_db (#6216)Bart Noordervliet2019-10-181-0/+2
| | | | Small fix to synapse_port_db to be able to convert from database schema v56.
* Add account_validity's email_sent column to the list of boolean columns in ↵Brendan Abolivier2019-06-031-0/+1
| | | | | | synapse_port_db Fixes #5306
* drop tables listed in #1830 (#4992)Neil Johnson2019-04-081-5/+0
| | | | | | | | | | | | | | | Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities
* Fix issue #4596Anders2019-04-031-1/+1
| | | | | Make synapse_port_db --curses work with Python 3. Signed-off-by: Anders Jensen-Waud <anders@jensenwaud.com>
* Consider e2e_room_keys.is_verified column as booleanEric2019-02-201-0/+1
| | | | | | | This column was considered as an int, crashing the whole migration process Signed-off-by: Eric <eric@pedr0.net>
* Fix a number of flake8 errorsRichard van der Hoff2018-10-241-1/+2
| | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.
* Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase ↵Amber Brown2018-10-201-150/+122
| | | | on py3) (#4068)
* Replace some type checks with six type checksAdrian Tschira2018-04-071-1/+3
| | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* Port script: Set up state_group_id_seqRichard van der Hoff2018-04-061-0/+13
| | | | Fixes https://github.com/matrix-org/synapse/issues/3050.
* Port script: clean up a bitRichard van der Hoff2018-04-061-25/+29
| | | | | Improve logging and comments. Group all the stuff to do with inspecting tables together rather than creating the port tables in the middle.
* Port script: avoid nasty errors when setting upRichard van der Hoff2018-04-061-8/+6
| | | | We really shouldn't spit out "Failed to create port table", it looks scary.
* Less s'sErik Johnston2017-11-141-2/+2
|
* Add new boolean columns to port scriptErik Johnston2017-11-141-0/+7
|
* Make the port script drop NUL values in all tablesDavid Baker2017-10-311-7/+19
| | | | | | Postgres doesn't support NULs in strings so it makes the script throw an exception and stop if any values contain \0. Drop them with appropriate warning.
* Add groups table to BOOLEAN_COLUMNS in synapse_port_dbLuke Barnard2017-10-261-0/+1
|
* Fix port scriptErik Johnston2017-10-261-0/+1
| | | | | | | We changed _simple_update_one_txn to use _simple_update_txn but didn't yank it out in the port script. Fixes #2565
* Drop search values with nul charactersRichard van der Hoff2017-10-021-4/+7
| | | | | | https://github.com/matrix-org/synapse/issues/2187 contains a report of a port failing due to nul characters somewhere in the search table. Let's try dropping the offending rows.
* Extend commentErik Johnston2017-07-201-1/+2
|
* Fix port script for user directory tablesErik Johnston2017-07-201-0/+18
|
* add missing boolean to synapse_port_dbMatthew2017-06-161-0/+1
|
* fix synapse_port scriptMatthew2017-06-161-1/+1
|
* Don't assume postgres tables are in the public schema during db portJohn Kristensen2017-03-171-3/+1
| | | | | | | When fetching the list of tables from the postgres database during the db port, it is assumed that the tables are in the public schema. This is not always the case, so lets just rely on postgres to determine the default schema to use.
* Fix typoKevin Liu2017-02-111-1/+1
| | | This is what I get for not proofreading
* Fix synapse_port_db failure (fixes #1902)Kevin Liu2017-02-111-0/+1
| | | | | See https://matrix.to/#/!cURbafjkfsMDVwdRDQ:matrix.org/$148686272020hCgRD:potatofrom.space Signed-off-by: Kevin Liu <kevin@potatofrom.space>
* Update port script with recently added tablesErik Johnston2016-09-271-0/+9
| | | | | | This also fixes a bug where the port script would explode when it encountered the newly added boolean column `public_room_list_stream.visibility`
* Allow upgrading from old port_from_sqlite3 formatErik Johnston2016-08-041-0/+21
|
* Handle the fact that some tables have negative rowid rowsErik Johnston2016-08-041-43/+111
|
* Update port scriptErik Johnston2016-04-221-42/+81
|
* Don't require config to create databaseErik Johnston2016-04-061-8/+5
|
* Mark AS users with their AS's IDDaniel Wagner-Hall2016-02-111-2/+7
|
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* SpellingErik Johnston2015-11-191-1/+1
|
* Fix database port script to work with new event_search tableErik Johnston2015-11-181-11/+44
|
* Remove unused _execute_and_decode from scripts/synapse_port_dbMark Haines2015-09-231-2/+0
|
* Merge branch 'master' into developMark Haines2015-09-071-6/+9
|\ | | | | | | | | Conflicts: setup.py
| * Make port script handle empty sent_transactions tableErik Johnston2015-09-021-6/+9
| |
* | update logger to match new ambiguous script name...Matthew Hodgson2015-09-031-1/+1
|/
* Clean up scripts/Erik Johnston2015-08-271-0/+760
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-13 19:39:20 +0000