summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix the SQL SELECT query in _paginate_room_events_txnBrendan Abolivier2019-11-071-2/+13
| | | | Doing a SELECT DISTINCT when paginating is quite expensive, because it requires the engine to do sorting on the entire events table. However, we only need to run it if we're filtering on 2+ labels, so this PR is changing the request so that DISTINCT is only used then.
* Fix bug which caused rejected events to be stored with the wrong room state ↵Richard van der Hoff2019-11-061-1/+1
| | | | | | | | | | | | (#6320) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.
* Merge pull request #6301 from matrix-org/babolivier/msc2326Brendan Abolivier2019-11-013-2/+75
|\ | | | | Implement MSC2326 (label based filtering)
| * Incorporate reviewBrendan Abolivier2019-11-012-1/+7
| |
| * Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sqlBrendan Abolivier2019-11-011-1/+1
| | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Add more data to the event_labels table and fix the indexesBrendan Abolivier2019-11-013-5/+21
| |
| * LintBrendan Abolivier2019-11-011-1/+1
| |
| * Incorporate reviewBrendan Abolivier2019-11-011-2/+10
| |
| * LintBrendan Abolivier2019-10-301-7/+1
| |
| * Add more integration testingBrendan Abolivier2019-10-301-1/+1
| |
| * Add index on labelBrendan Abolivier2019-10-301-1/+3
| |
| * Implement filteringBrendan Abolivier2019-10-301-0/+9
| |
| * Store labels for new eventsBrendan Abolivier2019-10-301-1/+19
| |
| * Add database table for keeping track of labels on eventsBrendan Abolivier2019-10-291-0/+20
| |
* | Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schemaHubert Chathi2019-11-011-0/+42
|\ \ | | | | | | fix hidden field in devices table for older sqlite
| * | fix hidden field in devices table for older sqliteHubert Chathi2019-10-311-0/+42
| | |
* | | Update synapse/storage/data_stores/main/devices.pyHubert Chathi2019-10-311-1/+1
| | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-3119-40/+300
|\| |
| * | Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notifyHubert Chathi2019-10-3118-39/+272
| |\ \
| | * | Update black to 19.10b0 (#6304)Amber Brown2019-11-0113-27/+29
| | | | | | | | | | | | * update version of black and also fix the mypy config being overridden
| | * | Merge pull request #6294 from matrix-org/erikj/add_state_storageErik Johnston2019-10-313-1/+236
| | |\ \ | | | | | | | | | | Add StateGroupStorage interface
| | | * | Apply suggestions from code review Erik Johnston2019-10-311-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | Fix docstring Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | | * | Add StateGroupStorage interfaceErik Johnston2019-10-303-1/+235
| | | | |
| | * | | Fix /purge_room API.Erik Johnston2019-10-311-1/+0
| | | | | | | | | | | | | | | | | | | | It fails trying to clean the `topic` table which was recently removed.
| | * | | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-313-5/+5
| | | | | | | | | | | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
| | * | | fix delete_existing for _persist_events (#6300)Richard van der Hoff2019-10-302-5/+2
| | |/ / | | | | | | | | this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
| * | | make user signatures a separate streamHubert Chathi2019-10-302-12/+25
| | | |
| * | | Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notifyHubert Chathi2019-10-3013-724/+889
| |\| |
| * | | make notification of signatures work with workersHubert Chathi2019-10-242-2/+16
| | | |
* | | | rename get_devices_by_remote to get_device_updates_by_remoteHubert Chathi2019-10-301-4/+4
| | | |
* | | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-3012-657/+822
|\ \ \ \ | | |/ / | |/| |
| * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-3012-84/+126
| |\ \ \ | | | | | | | | | | | | | | | erikj/split_out_persistence_store
| | * | | Port receipt and read markers to async/waitErik Johnston2019-10-291-4/+3
| | | |/ | | |/|
| | * | Merge pull request #6263 from matrix-org/erikj/caches_return_deferredsErik Johnston2019-10-291-1/+1
| | |\ \ | | | | | | | | | | Quick fix to ensure cache descriptors always return deferreds
| | | * | Quick fix to ensure cache descriptors always return deferredsErik Johnston2019-10-281-1/+1
| | | | |
| | * | | Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into developErik Johnston2019-10-291-4/+8
| | |\ \ \
| | | * | | Make room directory search case insensitiveErik Johnston2019-10-291-4/+8
| | | | | |
| | * | | | Add CI for synapse_port_db (#6140)Brendan Abolivier2019-10-281-3/+6
| | | |/ / | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds: * a test sqlite database * a configuration file for the sqlite database * a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`) as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that: 1. installs Synapse 2. updates the test sqlite database to the latest schema and runs background updates on it 3. creates an empty postgresql database 4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage) Step `2` is done via a new script located at `scripts-dev/update_database`. The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
| | * | | Convert EventContext to attrs (#6218)Richard van der Hoff2019-10-281-1/+6
| | | | | | | | | | | | | | | | | | | | * make EventContext use an attr
| | * | | Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keysHubert Chathi2019-10-252-0/+33
| | |\ \ \ | | | | | | | | | | | | delete keys when deleting backup versions
| | | * | | delete keys when deleting backupsHubert Chathi2019-10-242-0/+33
| | | | |/ | | | |/|
| | * | | Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into developErik Johnston2019-10-251-67/+67
| | |\ \ \ | | | | |/ | | | |/|
| | | * | Fix /keys/query API on workers.Erik Johnston2019-10-251-67/+67
| | | |/ | | | | | | | | | | | | | | | | The necessary getters were added only to the master store and not the worker stores.
| | * | Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_loggingErik Johnston2019-10-251-3/+1
| | |\ \ | | | | | | | | | | Reduce debug logging overhead
| | | * | Address codestyle concernsMichael Kaye2019-10-241-3/+1
| | | | |
| | | * | use %r to __repr__ objectsMichael Kaye2019-10-241-1/+1
| | | |/ | | | | | | | | | | | | This avoids calculating __repr__ unless we are going to log.
| | * / Make user/room stats log line less verbose.Michael Kaye2019-10-241-1/+1
| | |/
| * | Review commentsErik Johnston2019-10-303-10/+26
| | |
| * | Add DataStores and Storage classes.Erik Johnston2019-10-233-4/+34
| | |
| * | Move persist_events out from main data store.Erik Johnston2019-10-232-634/+711
| | | | | | | | | | | | | | | | | | This is in preparation for splitting out of state_groups_state from the main store into it own one, as persisting events depends on calculating state.
* | | apply changes as a result of PR reviewHubert Chathi2019-10-301-43/+36
| | |
* | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-24241-4341/+4597
|\ \ \ | | |/ | |/|
| * | Move schema delta files to the correct data store.Erik Johnston2019-10-242-0/+0
| |/ | | | | | | They were put in the global schema delta directory due to a bad merge.
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-221-11/+37
| |\ | | | | | | | | | erikj/refactor_stores
| * | Clean up prepare_database.py a bit and add commentsErik Johnston2019-10-221-7/+30
| | |
| * | Delete background_update table creation in main data_storeErik Johnston2019-10-221-21/+0
| | |
| * | Move README into synapse/storage/__init__.pyErik Johnston2019-10-222-13/+12
| | |
| * | Add a basic README to synapse.storageErik Johnston2019-10-221-0/+13
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-226-12/+337
| |\ \ | | | | | | | | | | | | erikj/refactor_stores
| * | | Fix schema management to work with multiple data stores.Erik Johnston2019-10-211-41/+108
| | | |
| * | | Move storage classes into a main "data store".Erik Johnston2019-10-21239-4292/+4466
| | | | | | | | | | | | | | | | | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* | | | move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with ↵Hubert Chathi2019-10-231-67/+67
| | | | | | | | | | | | | | | | workers
* | | | vendor-prefix the EDU name until MSC1756 is merged into the specHubert Chathi2019-10-221-1/+2
| | | |
* | | | update to work with newer code, and fix formattingHubert Chathi2019-10-221-1/+1
| | | |
* | | | make black happyHubert Chathi2019-10-221-22/+32
| | | |
* | | | don't crash if the user doesn't have cross-signing keysHubert Chathi2019-10-221-14/+25
| | | |
* | | | implement federation parts of cross-signingHubert Chathi2019-10-221-5/+51
| |_|/ |/| |
* | | Merge branch 'develop' into cross-signing_sig_uploadHubert Chathi2019-10-1838-908/+1949
|\ \ \ | | |/ | |/|
| * | Merge branch 'develop' into uhoreg/e2e_cross-signing_mergedHubert Chathi2019-10-1838-903/+1945
| |\|
| | * Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storageHubert Chathi2019-10-111-4/+14
| | |\ | | | | | | | | make storage layer in charge of interpreting the device key data
| | | * expand on commentHubert Chathi2019-10-111-1/+2
| | | |
| | | * make sure we actually return somethingHubert Chathi2019-10-101-2/+4
| | | |
| | | * make storage layer in charge of interpreting the device key dataHubert Chathi2019-10-101-2/+9
| | | |
| | * | Fix MAU reaping where reserved users are specified. (#6168)Neil Johnson2019-10-111-39/+62
| | | |
| | * | Merge pull request #6156 from matrix-org/erikj/postgres_anyErik Johnston2019-10-1014-109/+189
| | |\ \ | | | | | | | | | | Use Postgres ANY for selecting many values.
| | | * | Fix SQLite take 2Erik Johnston2019-10-101-1/+1
| | | | |
| | | * | Fix SQLiteErik Johnston2019-10-101-1/+1
| | | | |
| | | * | Remove add_in_list_sql_clauseErik Johnston2019-10-104-34/+25
| | | | |
| | | * | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_anyErik Johnston2019-10-101-3/+4
| | | |\ \
| | | * | | Replace IN usage with helper funcsErik Johnston2019-10-1011-97/+138
| | | | | |
| | | * | | Add helper funcs to use postgres ANYErik Johnston2019-10-101-8/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This means that we can write queries with `col = ANY(?)`, which helps postgres.
| | * | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-101-3/+4
| | |\ \ \ \ | | | | |/ / | | | |/| | | | | | | | erikj/disable_sql_bytes
| | | * | | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_innerErik Johnston2019-10-102-35/+61
| | | |\| |
| | | * | | Merge branch 'develop' of github.com:matrix-org/synapse into erikj/patch_innerErik Johnston2019-10-0923-706/+1180
| | | |\ \ \ | | | | | |/ | | | | |/|
| | | * | | Patch inlinecallbacks for log contextsErik Johnston2019-09-272-4/+5
| | | | | |
| | * | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-1016-725/+1030
| | |\ \ \ \ | | | | |_|/ | | | |/| | | | | | | | erikj/disable_sql_bytes
| | | * | | Fix races in room stats (and other) updates. (#6187)Richard van der Hoff2019-10-101-9/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Hopefully this will fix the occasional failures we were seeing in the room directory. The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8. We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
| | | * | | Rewrite the user_filter migration again (#6184)Richard van der Hoff2019-10-101-26/+32
| | | | |/ | | | |/| | | | | | | | | | | | | | | | you can't plausibly ALTER TABLE in sqlite, so we create the new table with the right schema to start with.
| | | * | Merge pull request #6185 from matrix-org/erikj/fix_censored_evnetsErik Johnston2019-10-092-4/+28
| | | |\ \ | | | | | | | | | | | | Fix inserting bytes as text in `censor_redactions`
| | | * | | Fix RegistrationStoreBrendan Abolivier2019-10-081-1/+1
| | | | | |
| | | * | | Cleanup client_ipsBrendan Abolivier2019-10-081-7/+0
| | | | | |
| | | * | | Don't end up with 4 classes in registrationBrendan Abolivier2019-10-081-52/+50
| | | | | |
| | | * | | LintBrendan Abolivier2019-10-073-3/+3
| | | | | |
| | | * | | Move roommember's bg updates to a dedicated storeBrendan Abolivier2019-10-071-108/+114
| | | | | |
| | | * | | Move user_directory's bg updates to a dedicated storeBrendan Abolivier2019-10-071-84/+94
| | | | | |
| | | * | | Move state's bg updates to a dedicated storeBrendan Abolivier2019-10-071-190/+204
| | | | | |
| | | * | | Move search's bg updates to a dedicated storeBrendan Abolivier2019-10-071-25/+31
| | | | | |
| | | * | | Move registration's bg updates to a dedicated storeBrendan Abolivier2019-10-071-95/+103
| | | | | |
| | | * | | Move media_repository's bg updates to a dedicated storeBrendan Abolivier2019-10-071-3/+9
| | | | | |
| | | * | | Move devices's bg updates to a dedicated storeBrendan Abolivier2019-10-071-22/+27
| | | | | |
| | | * | | Move deviceinbox's bg updates to a dedicated storeBrendan Abolivier2019-10-071-15/+22
| | | | | |
| | | * | | Move client_ips's bg updates to a dedicated storeBrendan Abolivier2019-10-071-94/+106
| | | | | |
| | | * | | add some logging to the rooms stats updates, to try to track down a flaky ↵Richard van der Hoff2019-10-071-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | test (#6167)
| | | * | | Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_indexBrendan Abolivier2019-10-071-2/+2
| | | |\ \ \ | | | | | | | | | | | | | | Fix unique_user_filter_index schema update
| | | | * | | Fix unique_user_filter_index schema updateBrendan Abolivier2019-10-071-2/+2
| | | | | | |
| | | * | | | Merge pull request #6159 from matrix-org/erikj/cache_membershipsErik Johnston2019-10-071-19/+45
| | | |\ \ \ \ | | | | |/ / / | | | |/| | | Cache room membership lookups in _get_joined_users_from_context
| | | | * | | Fix bug where we didn't pull out event IDErik Johnston2019-10-071-1/+1
| | | | | | |
| | | | * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-031-6/+15
| | | | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | erikj/cache_memberships
| | | | * | | | Use the right error....Erik Johnston2019-10-031-1/+1
| | | | | | | |
| | | | * | | | pep8Erik Johnston2019-10-031-1/+0
| | | | | | | |
| | | | * | | | Fixup names and commentsErik Johnston2019-10-031-6/+7
| | | | | | | |
| | | | * | | | cachedList descriptor doesn't like typingErik Johnston2019-10-031-2/+2
| | | | | | | |
| | | | * | | | Cache room membership lookups in _get_joined_users_from_contextErik Johnston2019-10-031-19/+45
| | | | | | | |
| | | * | | | | Update `user_filters` table to have a unique index, and non-null columns (#1172)Alexander Maznev2019-10-041-0/+46
| | | | |/ / / | | | |/| | |
| | | * | | | add some metrics on the federation sender (#6160)Richard van der Hoff2019-10-031-6/+15
| | | |/ / /
| | | * | | Merge pull request #6154 from matrix-org/erikj/fix_appservice_paginationErik Johnston2019-10-021-18/+18
| | | |\ \ \ | | | | | | | | | | | | | | Fix appservice room list pagination
| | | | * | | Fix appservice room list paginationErik Johnston2019-10-021-18/+18
| | | | | | |
| | | * | | | Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatableErik Johnston2019-10-021-0/+3
| | | |\ \ \ \ | | | | | | | | | | | | | | | | Fix not showing non-federatable rooms to remote room list queries
| | | | * | | | Fix not showing non-federatable rooms to remote room list queriesErik Johnston2019-10-021-0/+3
| | | | |/ / /
| | | * | | | Merge pull request #6148 from matrix-org/erikj/find_next_generatedErik Johnston2019-10-021-1/+3
| | | |\ \ \ \ | | | | |/ / / | | | |/| | | Bound find_next_generated_user_id DB query.
| | | | * | | Bound find_next_generated_user_id DB query.Erik Johnston2019-10-021-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can easily bound the set of user IDs we pull out of the DB, so lets do that.
| | | * | | | Fix public room list pagination.Erik Johnston2019-10-021-17/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We incorrectly used `room_id` as to bound the result set, even though we order by `joined_members, room_id`, leading to incorrect results after pagination.
| | | * | | | Land improved room list based on room stats (#6019)Erik Johnston2019-10-022-77/+167
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Use room_stats and room_state for room directory search
| | | * | | | Fix yields and copy instead of move push rules on room upgrade (#6144)Andrew Morgan2019-10-021-10/+6
| | | |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room. For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first. Also fix some missing yields that probably broke things quite a bit.
| | | * | | Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into developErik Johnston2019-10-025-18/+142
| | | |\ \ \
| | | * \ \ \ Merge branch 'erikj/fixup_devices_last_seen_query' of ↵Erik Johnston2019-10-013-7/+54
| | | |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | github.com:matrix-org/synapse into develop
| | | * | | | | Drop unused tables (#6115)Richard van der Hoff2019-09-301-0/+20
| | | | |_|_|/ | | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These tables are unused since #5893 (as amended by #6047), so we can now drop them. Fixes #6048.
| | * | | | | Do the update as a background indexErik Johnston2019-10-092-9/+51
| | | | | | |
| | * | | | | Disable bytes usage with postgresErik Johnston2019-10-083-3/+10
| | | |_|_|/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | More often than not passing bytes to `txn.execute` is a bug (where we meant to pass a string) that just happens to work if `BYTEA_OUTPUT` is set to `ESCAPE`. However, this is a bit of a footgun so we want to instead error when this happens, and force using `bytearray` if we actually want to use bytes.
| | * | | | Fix existing hex encoded json values in DBErik Johnston2019-10-081-0/+26
| | | | | |
| | * | | | Fix inserting bytes as textErik Johnston2019-10-081-4/+2
| | | |_|/ | | |/| |
| | * | | Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timingsErik Johnston2019-10-021-0/+18
| | |\ \ \ | | | | | | | | | | | | Fix errors storing large retry intervals.
| | | * | | Fix errors storing large retry intervals.Erik Johnston2019-10-021-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
| | * | | | Merge pull request #6145 from matrix-org/erikj/fix_censored_redactionsErik Johnston2019-10-021-0/+14
| | |\ \ \ \ | | | |/ / / | | |/| | | Fix fetching censored redactions from DB
| | | * | | Fix fetching censored redactions from DBErik Johnston2019-10-021-0/+14
| | | | |/ | | | |/| | | | | | | | | | | | | | | | | | | | | Fetching a censored redactions caused an exception due to the code expecting redactions to have a `redact` key, which redacted redactions don't have.
| | * | | Use `received_ts` to find uncensored redacted eventsErik Johnston2019-10-011-14/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Joining against `events` and ordering by `stream_ordering` is inefficient as it forced scanning the entirety of the redactions table. This isn't the case if we use `redactions.received_ts` column as we can then use an index.
| | * | | Add received_ts column to redactions.Erik Johnston2019-10-013-9/+92
| | | | | | | | | | | | | | | | | | | | | | | | | This will allow us to efficiently search for uncensored redactions in the DB before a given time.
| | * | | Don't repeatedly attempt to censor events we don't have.Erik Johnston2019-10-011-1/+17
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently we don't set `have_censored` column if we don't have the target event of a redaction, which means we repeatedly attempt to censor the same non-existant event. When we persist non-redacted events we unset the `have_censored` column for any redactions that target said event.
| | * / Fix `devices_last_seen` background update.Erik Johnston2019-09-303-7/+54
| | |/ | | | | | | | | | Fixes #6134.
| | * Fix dummy event insertion consent bug (#6053)Neil Johnson2019-09-261-3/+15
| | | | | | | | | Fixes #5905
| | * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-254-9/+94
| | |\ | | | | | | | | | | | | erikj/cleanup_user_ips_2
| | | * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-253-8/+93
| | | |\ | | | | | | | | | | | | | | | erikj/cleanup_user_ips
| | | | * Merge pull request #6092 from ↵Brendan Abolivier2019-09-242-6/+6
| | | | |\ | | | | | | | | | | | | | | | | | | | | | | | | matrix-org/babolivier/background_update_deactivated_return Fix the return value in the users_set_deactivated_flag background job
| | | | | * Incorporate reviewBrendan Abolivier2019-09-241-3/+3
| | | | | |
| | | | | * Fix loggingBrendan Abolivier2019-09-241-1/+1
| | | | | |
| | | | | * TypoBrendan Abolivier2019-09-231-1/+1
| | | | | |
| | | | | * Fix the return value in the users_set_deactivated_flag background jobBrendan Abolivier2019-09-231-4/+4
| | | | | |
| | | | * | Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_workRichard van der Hoff2019-09-241-1/+1
| | | | |\ \
| | | | | * | remove unused parameter to get_user_id_by_threepid (#6099)Richard van der Hoff2019-09-241-1/+1
| | | | | |/ | | | | | | | | | | | | Added in #5377, apparently in error
| | | | * | Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_workRichard van der Hoff2019-09-242-28/+45
| | | | |\|
| | | | | * Implement MSC2290 (#6043)Andrew Morgan2019-09-231-1/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away. This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints. Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
| | | | * | Merge branch 'develop' into rav/saml_mapping_workRichard van der Hoff2019-09-1928-552/+1649
| | | | |\ \
| | | | * | | Record mappings from saml users in an external tableRichard van der Hoff2019-09-132-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We want to assign unique mxids to saml users based on an incrementing suffix. For that to work, we need to record the allocated mxid in a separate table.
| | | * | | | Use if `is not None`Erik Johnston2019-09-251-1/+1
| | | | | | | | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * | | | | Review commentsErik Johnston2019-09-252-5/+2
| | | | | | |
| | * | | | | Prune rows in user_ips older than configured periodErik Johnston2019-09-241-8/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Defaults to pruning everything older than 28d.
| | * | | | | Add has_completed_background_updateErik Johnston2019-09-241-1/+24
| | |/ / / / | | | | | | | | | | | | | | | | | | This allows checking if a specific background update has completed.
| | * | | | Query devices table for last seen info.Erik Johnston2019-09-231-49/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a) simpler than querying user_ips directly and b) means we can purge older entries from user_ips without losing the required info. The storage functions now no longer return the access_token, since it was unused.
| | * | | | Add BG update to populate devices last seen infoErik Johnston2019-09-232-0/+55
| | | | | |
| | * | | | Add last seen info to devices table.Erik Johnston2019-09-232-0/+36
| | | |_|/ | | |/| | | | | | | | | | | | | | | | | This allows us to purge old user_ips entries without having to preserve the latest last seen info for active devices.
| | * | | Allow HS to send emails when adding an email to the HS (#6042)Andrew Morgan2019-09-201-7/+24
| | | | |
| | * | | Undo the deletion of some tables (#6047)Richard van der Hoff2019-09-191-20/+0
| | | |/ | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a partial revert of #5893. The problem is that if we drop these tables in the same release as removing the code that writes to them, it prevents users users from being able to roll back to a previous release. So let's leave the tables in place for now, and remember to drop them in a subsequent release. (Note that these tables haven't been *read* for *years*, so any missing rows resulting from a temporary upgrade to vNext won't cause a problem.)
| | * | Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-172-7/+41
| | | | | | | | | | | | | | | | Track the time that a server started failing at, for general analysis purposes.
| | * | Clean up some code in the retry logic (#6017)Richard van der Hoff2019-09-111-20/+0
| | | | | | | | | | | | | | | | * remove some unused code * make things which were constants into constants for efficiency and clarity
| | * | Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-usersJason Robinson2019-09-091-0/+37
| | |\ \ | | | | | | | | | | Only count real users when checking for auto-creation of auto-join room
| | | * | Fix code style, againJason Robinson2019-09-091-3/+1
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | | * | Simplify count_real_users SQL to only count user_type is null rowsJason Robinson2019-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | | * | Simplify is_real_user_txn check to trust user_type is null if real userJason Robinson2019-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | | * | Auto-fix a few code style issuesJason Robinson2019-09-091-4/+4
| | | | | | | | | | | | | | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | | * | Only count real users when checking for auto-creation of auto-join roomJason Robinson2019-09-091-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>
| | * | | Merge pull request #5934 from matrix-org/erikj/censor_redactionsErik Johnston2019-09-092-1/+119
| | |\ \ \ | | | |/ / | | |/| | Censor redactions in DB after a month
| | | * | Fix commentsErik Johnston2019-09-091-2/+3
| | | | |
| | | * | Use better names in SQLErik Johnston2019-09-091-5/+8
| | | | |
| | | * | Fixup commentErik Johnston2019-09-091-4/+12
| | | | |
| | | * | Don't start looping call unless enabledErik Johnston2019-09-091-1/+2
| | | | |
| | | * | Handle setting retention period to 0Erik Johnston2019-09-091-1/+1
| | | | |
| | | * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-09-0516-378/+959
| | | |\ \ | | | | | | | | | | | | | | | | | | erikj/censor_redactions
| | | * | | Make redaction retention period configurableErik Johnston2019-09-051-2/+4
| | | | | |
| | | * | | Censor redactions in DB after a monthErik Johnston2019-08-302-1/+104
| | | | | |
| * | | | | use the right function for when we're already in runInteractionHubert Chathi2019-10-181-4/+4
| | | | | |
| * | | | | Merge branch 'develop' into uhoreg/e2e_cross-signing_mergedHubert Chathi2019-09-074-81/+142
| |\| | | |
* | | | | | make changes based on PR feedbackHubert Chathi2019-09-241-11/+6
| | | | | |
* | | | | | Merge branch 'develop' into cross-signing_sig_uploadHubert Chathi2019-09-074-81/+142
|\ \ \ \ \ \ | | |/ / / / | |/| | | |
| * | | | | Servers-known-about statistic (#5981)Amber Brown2019-09-071-0/+59
| | | | | |
| * | | | | Merge pull request #5998 from matrix-org/erikj/fixup_federate_flagErik Johnston2019-09-061-1/+3
| |\ \ \ \ \ | | | | | | | | | | | | | | Correctly handle non-bool m.federate flag
| | * | | | | Correctly handle non-bool m.federate flagErik Johnston2019-09-061-1/+3
| | | |_|/ / | | |/| | |
| * | | | | Move delete_threepid_session into RegistrationWorkerStoreAndrew Morgan2019-09-061-25/+25
| | | | | |
| * | | | | Move get_threepid_validation_session into RegistrationWorkerStoreAndrew Morgan2019-09-061-54/+54
| |/ / / /
| * | | | Opentracing context cannot be noneJorik Schellekens2019-09-051-1/+1
| | | | |
* | | | | make black happyHubert Chathi2019-09-041-15/+9
| | | | |
* | | | | allow uploading signatures of master key signed by devicesHubert Chathi2019-09-041-1/+1
| | | | |
* | | | | implement device signature uploading/fetchingHubert Chathi2019-09-041-0/+38
| |/ / / |/| | |
* | | | Merge branch 'develop' into cross-signing_keysHubert Chathi2019-09-0416-377/+958
|\| | |
| * | | Fix and refactor room and user stats (#5971)Erik Johnston2019-09-045-345/+904
| | | | | | | | | | | | Previously the stats were not being correctly populated.
| * | | Remove double return statements (#5962)Andrew Morgan2019-09-033-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
| * | | Opentrace device lists (#5853)Jorik Schellekens2019-09-032-0/+26
| | | | | | | | | | | | Trace device list changes.
| * | | Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-3010-28/+28
| |/ / | | | | | | | | | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* | | Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keysHubert Chathi2019-08-2814-328/+653
|\ \ \
| * | | Merge branch 'develop' into uhoreg/e2e_cross-signing_mergedHubert Chathi2019-08-2814-307/+632
| |\| |
| | * | Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consentWill Hunt2019-08-281-0/+1
| | |\ \ | | | | | | | | | | Exempt support users from consent
| | | * | Ignore consent for support usersHalf-Shot2019-08-231-0/+1
| | | | |
| | * | | Fix a cache-invalidation bug for worker-based deployments (#5920)Richard van der Hoff2019-08-281-8/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some of the caches on worker processes were not being correctly invalidated when a room's state was changed in a way that did not affect the membership list of the room. We need to make sure we send out cache invalidations even when no memberships are changing.
| | * | | Allow schema deltas to be engine-specific (#5911)reivilibre2019-08-271-5/+19
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Allow schema deltas to be engine-specific Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net> * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net> * Code style (Black) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| | | * | | Code style (Black)Olivier Wilkinson (reivilibre)2019-08-271-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| | | * | | Allow schema deltas to be engine-specificOlivier Wilkinson (reivilibre)2019-08-271-5/+21
| | | |/ / | | | | | | | | | | | | | | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
| | * / / Add Admin API capability to set adminship of a user (#5878)reivilibre2019-08-271-0/+23
| | |/ / | | | | | | | | Admin API: Set adminship of a user
| | * | Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-222-6/+53
| | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * | Opentrace e2e keys (#5855)Jorik Schellekens2019-08-222-3/+49
| | | | | | | | | | | | Add opentracing tags and logs for e2e keys
| | * | Add missing index on users_in_public_rooms. (#5894)Richard van der Hoff2019-08-221-0/+17
| | | |
| | * | Servlet to purge old rooms (#5845)Richard van der Hoff2019-08-221-0/+137
| | | |
| | * | Drop some unused tables. (#5893)Richard van der Hoff2019-08-213-47/+22
| | | | | | | | | | | | These tables are never used, so we may as well drop them.
| | * | Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_indexHubert Chathi2019-08-162-1/+19
| | |\ \ | | | |/ | | |/| add the version field to the index for e2e_room_keys
| | | * update set_e2e_room_key to agree with fixed indexHubert Chathi2019-08-141-1/+1
| | | |
| | | * add the version field to the index for e2e_room_keysHubert Chathi2019-08-141-0/+18
| | | |
| | * | Merge pull request #5809 from matrix-org/erikj/handle_pusher_stopErik Johnston2019-08-131-8/+22
| | |\ \ | | | | | | | | | | Handle pusher being deleted during processing.
| | | * | Clarify docstringErik Johnston2019-08-131-1/+1
| | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | | * | Handle pusher being deleted during processing.Erik Johnston2019-08-011-8/+22
| | | | | | | | | | | | | | | | | | | | | | | | | Instead of throwing a StoreError lets break out of processing loop and mark the pusher as stopped.
| | * | | Merge pull request #5826 from matrix-org/erikj/reduce_event_pausesErik Johnston2019-08-131-128/+142
| | |\ \ \ | | | | | | | | | | | | Don't unnecessarily block notifying of new events.
| | | * | | Don't unnecessarily block notifying of new events.Erik Johnston2019-08-061-128/+142
| | | | |/ | | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When persisting events we calculate new stream orderings up front. Before we notify about an event all events with lower stream orderings must have finished being persisted. This PR moves the assignment of stream ordering till *after* calculated the new current state and split the batch of events into separate chunks for persistence. This means that if it takes a long time to calculate new current state then it will not block events in other rooms being notified about. This should help reduce some global pauses in the events stream which can last for tens of seconds (if not longer), caused by some particularly expensive state resolutions.
| | * | | Merge pull request #5788 from matrix-org/rav/metaredactionsRichard van der Hoff2019-08-121-101/+112
| | |\ \ \ | | | |/ / | | |/| | Fix handling of redactions of redactions
| | | * | Handle loops in redaction eventsRichard van der Hoff2019-07-301-60/+36
| | | | |
| | | * | recursively fetch redactionsRichard van der Hoff2019-07-301-32/+36
| | | | |
| | | * | split _get_events_from_db out of _enqueue_eventsRichard van der Hoff2019-07-291-32/+51
| | | | |
| | | * | Return dicts from _fetch_event_listRichard van der Hoff2019-07-291-15/+27
| | | | |
| * | | | Cross-signing [1/4] -- hidden devices (#5759)Hubert Chathi2019-08-123-10/+48
| |/ / / | | | | | | | | | | | | | | | | | | | | * allow devices to be marked as "hidden" This is a prerequisite for cross-signing, as it allows us to create other things that live within the device namespace, so they can be used for signatures.
| * | / Have ClientReaderSlavedStore inherit RegistrationStore (#5806)Andrew Morgan2019-08-011-21/+21
| | |/ | |/| | | | Fixes #5803
* | | use stream ID generator instead of timestampHubert Chathi2019-08-283-19/+20
| | |
* | | apply PR review suggestionsHubert Chathi2019-08-212-10/+11
| | |
* | | make changes from PR reviewHubert Chathi2019-08-012-41/+55
| | |
* | | Merge branch 'cross-signing_hidden' into cross-signing_keysHubert Chathi2019-08-0140-452/+566
|\ \ \
| * | | don't need to return the hidden column any moreHubert Chathi2019-08-011-1/+1
| | | |
| * | | really fix queries to work with PostgresHubert Chathi2019-07-311-19/+11
| | | | | | | | | | | | | | | | (by going back to not using SQL directly)
| * | | PostgreSQL, Y U no like?Hubert Chathi2019-07-311-3/+4
| | | |
| * | | remove extra SQL query paramHubert Chathi2019-07-311-1/+0
| | | |
| * | | Merge branch 'develop' into cross-signing_hiddenHubert Chathi2019-07-3040-432/+554
| |\| |
| | * | Don't recreate columnErik Johnston2019-07-302-1/+22
| | | |
| | * | Fix current_state_events membership background update.Erik Johnston2019-07-302-1/+1
| | |/ | | | | | | | | | | | | | | | Turns out not all rooms are in `rooms`, so lets fetch the room list from `current_state_events`. We move the delta file to force it to be run again.
| | * Merge pull request #5774 from matrix-org/erikj/fix_rejected_membershipErik Johnston2019-07-291-0/+6
| | |\ | | | | | | | | Fix room summary when rejected events are in state
| | | * Fix room summary when rejected events are in stateErik Johnston2019-07-261-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | Annoyingly, `current_state_events` table can include rejected events, in which case the membership column will be null. To work around this lets just always filter out null membership for now.
| | * | Merge pull request #5752 from matrix-org/erikj/forgotten_userErik Johnston2019-07-292-35/+123
| | |\ \ | | | | | | | | | | Remove some more joins on room_memberships
| | | * | Expand commentErik Johnston2019-07-291-0/+5
| | | | |
| | | * | Remove lost commentErik Johnston2019-07-291-7/+0
| | | |/
| | | * Remove join when calculating room summaries.Erik Johnston2019-07-241-13/+21
| | | |
| | | * Remove join on room_memberships when fetching rooms for user.Erik Johnston2019-07-241-22/+36
| | | |
| | | * Add function to get all forgotten rooms for userErik Johnston2019-07-242-0/+68
| | | | | | | | | | | | | | | | | | | | | | | | This will allow us to efficiently filter out rooms that have been forgotten in other queries without having to join against the `room_memberships` table.
| | * | Fix current_state bg update to work on old SQLiteErik Johnston2019-07-291-2/+2
| | | |
| | * | Merge branch 'master' into developRichard van der Hoff2019-07-261-0/+4
| | |\ \
| | | * | Fix DoS when there is a cycle in redaction eventsRichard van der Hoff2019-07-261-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | Make sure that synapse doesn't explode when a redaction redacts itself, or there is a larger cycle.
| | * | | Merge pull request #5767 from matrix-org/rav/redactions/cross_room_idRichard van der Hoff2019-07-251-0/+27
| | |\| | | | | | | | | | | | log when a redaction attempts to redact an event in a different room
| | | * | log when a redaction attempts to redact an event in a different roomRichard van der Hoff2019-07-251-0/+27
| | | | |
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-28 21:06:29 +0000