summary refs log tree commit diff
path: root/synapse/storage/stream.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move storage classes into a main "data store".Erik Johnston2019-10-211-948/+0
| | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-6/+6
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Replace returnValue with return (#5736)Amber Brown2019-07-231-23/+21
|
* Fixup docstringsErik Johnston2019-07-121-7/+5
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-07-051-1/+1
|\ | | | | | | erikj/admin_exfiltrate_data
| * Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
| |
* | Assume key existence. Update docstringsErik Johnston2019-07-051-6/+10
|/
* Run Black. (#5482)Amber Brown2019-06-201-18/+14
|
* Better wordsErik Johnston2019-06-061-1/+1
|
* Fix get_max_topological_token to never return NoneErik Johnston2019-05-211-1/+11
|
* Spelling and clarificationsErik Johnston2019-05-171-5/+9
|
* Update docstringErik Johnston2019-05-161-1/+12
|
* Make generating SQL bounds for pagination genericErik Johnston2019-05-151-61/+118
| | | | | This will allow us to reuse the same structure when we paginate e.g. relations
* Expose DataStore._get_events as get_events_as_listErik Johnston2019-05-141-7/+11
| | | | This is in preparation for reaction work which requires it.
* Run black on the rest of the storage module (#4996)Amber Brown2019-04-031-133/+151
|
* Add docstrings from matrix-org-hotfixesErik Johnston2019-03-061-0/+19
|
* Comments on get_all_new_events_streamRichard van der Hoff2018-10-121-1/+15
| | | | just some docstrings to clarify the behaviour here
* initial cut at a room summary API (#3574)Matthew Hodgson2018-08-161-2/+2
|
* Merge pull request #3612 from matrix-org/rav/store_heirarchyRichard van der Hoff2018-07-311-1/+1
|\ | | | | Make EventStore inherit from EventFederationStore
| * Make EventStore inherit from EventFederationStoreRichard van der Hoff2018-07-261-1/+1
| | | | | | | | | | | | | | | | (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.
* | make /context lazyload & filter aware (#3567)Matthew Hodgson2018-07-271-3/+11
|/ | | make /context lazyload & filter aware.
* run isortAmber Brown2018-07-091-9/+7
|
* Remove unused string formatting paramErik Johnston2018-05-221-1/+0
|
* Don't support limitless paginationErik Johnston2018-05-221-8/+6
| | | | | | | The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.
* Merge pull request #3199 from matrix-org/erikj/pagination_syncErik Johnston2018-05-091-39/+36
|\ | | | | Refactor sync APIs to reuse pagination API
| * Refactor sync APIs to reuse pagination APIErik Johnston2018-05-091-38/+35
| | | | | | | | | | | | The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.
| * Don't unnecessarily require token to be stream tokenErik Johnston2018-05-091-1/+1
| | | | | | | | | | This allows calling the `get_recent_event_ids_for_room` function in more situations.
* | Merge pull request #3198 from matrix-org/erikj/fixup_return_paginationErik Johnston2018-05-091-1/+15
|\ \ | | | | | | Refactor get_recent_events_for_room return type
| * | Fix up commentErik Johnston2018-05-091-1/+1
| | |
| * | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
| |\ \ | | |/ | |/| | | | erikj/fixup_return_pagination
| * | Refactor get_recent_events_for_room return typeErik Johnston2018-05-091-1/+15
| | | | | | | | | | | | | | | | | | There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-05-091-5/+6
|\ \ \ | | |/ | |/| | | | erikj/remove_membership_change
| * | Update commentsErik Johnston2018-05-091-5/+6
| |/
* / Remove unused code path from member change DB funcErik Johnston2018-05-091-24/+10
|/ | | | | The function is never called without a from_key, so we can remove all the handling for that scenario.
* Refactor pagination DB API to return concrete typeErik Johnston2018-05-091-28/+48
| | | | | This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.
* Remove unused from_token paramErik Johnston2018-05-091-7/+4
|
* Refactor recent events func to use pagination funcErik Johnston2018-05-091-48/+27
| | | | This also removes a cache that is unlikely to ever get hit.
* Fix up comments and make function privateErik Johnston2018-05-091-7/+7
|
* Reuse existing pagination code for context APIErik Johnston2018-05-081-75/+15
|
* Parse tokens before calling DB functionErik Johnston2018-05-081-14/+18
|
* Only fetch required fields from databaseErik Johnston2018-05-081-1/+2
|
* Split paginate_room_events storage functionErik Johnston2018-05-081-28/+72
|
* Merge branch 'develop' into py3-xrange-1Richard van der Hoff2018-04-301-2/+3
|\
| * Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-1/+1
| |\
| * | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-2/+3
| | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | Move more xrange to sixAdrian Tschira2018-04-281-1/+3
| |/ |/| | | | | | | | | plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Add missing consumeErrorsRichard van der Hoff2018-04-271-1/+1
|/ | | | | In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.
* Merge pull request #2946 from matrix-org/rav/timestamp_to_purgeRichard van der Hoff2018-03-061-0/+27
|\ | | | | Implement purge_history by timestamp
| * Provide a means to pass a timestamp to purge_historyRichard van der Hoff2018-03-051-0/+27
| |
* | Remove ability for AS users to call /events and /syncErik Johnston2018-03-051-76/+0
|/ | | | | | | This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144
* Document abstract class and method betterErik Johnston2018-03-011-8/+13
|
* Split out stream storeErik Johnston2018-03-011-157/+193
|
* Remove preserve_context_over_{fn, deferred}Richard van der Hoff2017-11-141-2/+2
| | | | | Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.
* Assume rooms likely haven't changedErik Johnston2017-03-131-0/+3
|
* Doc argsErik Johnston2017-02-011-0/+4
|
* CommentErik Johnston2017-02-011-0/+3
|
* Include newly joined users in /keys/changes APIErik Johnston2017-02-011-0/+7
|
* Add profile data to the room_membership table for joinsErik Johnston2016-12-081-0/+3
|
* Store federation stream positions in the databaseErik Johnston2016-11-211-0/+16
|
* Handle sending events and device messages over federationErik Johnston2016-11-171-0/+31
|
* Fix incredubly slow back pagination queryErik Johnston2016-10-241-6/+13
| | | | | | | If a client didn't specify a from token when paginating backwards synapse would attempt to query the (global) maximum topological token. This a) doesn't make much sense since they're room specific and b) there are no indices that lets postgres do this efficiently.
* Calculate the public room list from a stream_orderingErik Johnston2016-09-141-0/+3
|
* Preserve some logcontextsErik Johnston2016-08-241-3/+3
|
* CommentErik Johnston2016-07-201-0/+4
|
* Add support for filters in paginate_room_eventsErik Johnston2016-07-141-1/+51
|
* Merge branch 'erikj/shared_secret' into erikj/test2Erik Johnston2016-07-061-51/+93
|\
| * Fix typoMark Haines2016-07-051-1/+1
| |
| * Make get_events_around more efficient on sqlite3Mark Haines2016-07-051-13/+49
| |
| * Use true/false for boolean parameter inclusive to avoid potential for sqli, ↵Mark Haines2016-07-051-4/+6
| | | | | | | | and possibly make the code clearer
| * Use different SQL for postgres and sqlite3 for when using multicolumn indexesMark Haines2016-07-041-50/+50
| |
| * Use a query that postgresql optimises better for get_events_aroundMark Haines2016-07-041-12/+16
| |
* | Add ReadWriteLock for pagination and history pruneErik Johnston2016-07-051-2/+2
|/
* Remove event fetching from DB threadsErik Johnston2016-06-031-19/+15
|
* Remove some unused functions (#711)Mark Haines2016-04-081-90/+0
| | | | | | | | * Remove some unused functions * get_room_events_stream is only used in tests * is_exclusive_room might actually be something we want
* Use a stream id generator for backfilled idsMark Haines2016-04-011-1/+1
|
* Don't cache events in get_recent_events_for_roomErik Johnston2016-03-231-14/+19
|
* Add commentErik Johnston2016-03-091-0/+3
|
* Return the correct token formErik Johnston2016-03-091-1/+1
|
* Use topological orders for initial sync timelineErik Johnston2016-03-091-2/+2
|
* Remove unused param from get_max_tokenErik Johnston2016-02-181-1/+1
|
* Fix SYN-627, events are in incorrect room in /syncErik Johnston2016-02-111-1/+1
|
* Return events in correct order for /eventsErik Johnston2016-02-111-8/+11
|
* Fix up logcontextsErik Johnston2016-02-081-4/+5
|
* Add descriptionsErik Johnston2016-02-031-0/+1
|
* Fix flake8 warnings for new flake8Daniel Wagner-Hall2016-02-021-1/+1
|
* s/get_room_changes_for_user/get_membership_changes_for_user/Erik Johnston2016-02-021-2/+2
|
* Fetch events in a separate transaction.Erik Johnston2016-02-021-26/+29
| | | | | | | | | This has a couple of benefits: - It reduces the time of transactions, allowing other database requests to run. - Fetching events is given a dedicated database thread, and so can't starve other database requests.
* Order things correctlyErik Johnston2016-02-011-0/+4
|
* Switch over /events to use per room cachesErik Johnston2016-02-011-2/+2
|
* Merge pull request #543 from matrix-org/erikj/syncErik Johnston2016-02-011-0/+7
|\ | | | | Cache get_room_changes_for_user
| * Cache get_room_changes_for_userErik Johnston2016-01-291-0/+7
| |
* | Only use room_ids if in get_room_events_stream if is_guestErik Johnston2016-02-011-5/+0
|/
* Prefill stream change cachesErik Johnston2016-01-291-8/+0
|
* Change name and doc has_entity_changedErik Johnston2016-01-281-1/+1
|
* Cache tags and account dataErik Johnston2016-01-281-4/+4
|
* Ensure keys to RoomStreamChangeCache are intsErik Johnston2016-01-281-5/+6
|
* Invalidate caches properly. Remove unused argErik Johnston2016-01-281-1/+1
|
* Allow paginating backwards from stream tokenErik Johnston2016-01-281-2/+14
|
* Return correct type of tokenErik Johnston2016-01-281-3/+6
|
* PEP8Erik Johnston2016-01-271-3/+6
|
* Use the same path for incremental with gap or without gapErik Johnston2016-01-271-3/+3
|
* Add cache to room streamErik Johnston2016-01-271-0/+133
|
* Add a Homeserver.setup method.Erik Johnston2016-01-261-13/+0
| | | | | | This is for setting up dependencies that require work on startup. This is useful for the DataStore that wants to read a bunch from the database before initiliazing.
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Open up /events to anonymous users for room events onlyDaniel Wagner-Hall2015-11-051-8/+38
| | | | Squash-merge of PR #345 from daniel/anonymousevents
* Remove more unused parametersDaniel Wagner-Hall2015-11-021-2/+1
|
* DocsErik Johnston2015-10-281-0/+26
|
* Add room context apiErik Johnston2015-10-281-1/+110
|
* Allow users that have left a room to get the messages that happend in the ↵Mark Haines2015-09-091-1/+18
| | | | room before they left
* Remove undocumented and unimplemented 'feedback' parameter from the ↵Mark Haines2015-09-091-8/+2
| | | | Client-Server API
* Allow room initialSync for users that have left the room, returning a ↵Mark Haines2015-09-091-0/+15
| | | | snapshot of how the room was when they left it
* Move all the caches into their own package, synapse.util.cachesErik Johnston2015-08-111-1/+2
|
* Cache get_recent_events_for_roomErik Johnston2015-08-071-2/+2
|
* Use dictionary cache to do group -> state fetchingErik Johnston2015-08-051-2/+1
|
* Fix daedlockErik Johnston2015-05-151-0/+2
|
* Move fetching of events into their own transactionsErik Johnston2015-05-141-9/+10
|
* Fetch events from events_id in their own transactionsErik Johnston2015-05-131-10/+12
|
* Make get_current_token accept a direction parameter, which tells whether the ↵Erik Johnston2015-05-121-2/+18
| | | | source whether we want a token for going 'forwards' or 'backwards'
* Move storage.stream._StreamToken to types.RoomStreamTokenErik Johnston2015-05-111-85/+33
|
* Add more conditions on JOINs to make postgres go a little faster.Erik Johnston2015-04-301-3/+5
|
* Add support for postgres instead of mysql. Change sql accourdingly. blob + ↵Erik Johnston2015-04-141-8/+8
| | | | varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
* Remove unused importErik Johnston2015-04-091-1/+1
|
* Stream ordering and out of order insertions.Erik Johnston2015-04-091-20/+3
| | | | | | | Handle the fact that events can be persisted out of order, and so to get the "current max" stream token becomes non trivial - as we need to make sure that *all* stream tokens less than the current max have also successfully been persisted.
* Make work in both Maria and SQLite. Fix testsErik Johnston2015-04-011-6/+0
|
* Merge branch 'develop' of github.com:matrix-org/synapse into mysqlErik Johnston2015-03-201-1/+2
|\
| * Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangementErik Johnston2015-03-201-1/+2
| |\
| | * Add the tiniest of tiny one-element caches to get_room_events_max_id() as ↵Paul "LeoNerd" Evans2015-03-201-1/+2
| | | | | | | | | | | | it's read every time someone hits eventstream
| * | Rearrange storage modulesErik Johnston2015-03-201-0/+19
| |/
* | Rearrange storage modulesErik Johnston2015-03-201-0/+19
| |
* | Convert storage layer to be mysql compatibleErik Johnston2015-03-191-2/+2
|/
* Wrap all of get_app_service_rooms in a txn.Kegan Dougal2015-03-021-24/+22
|
* JOIN state_events rather than parsing unrecognized_keys to pull out member ↵Kegan Dougal2015-03-021-10/+4
| | | | state_keys
* PR tweaks: set earlier on and use 'as json' for compatKegan Dougal2015-03-021-4/+4
|
* PEP8Kegan Dougal2015-02-271-1/+0
|
* Check for membership invite events correctly.Kegan Dougal2015-02-261-2/+21
|
* Finish impl to get new events for AS. ASes should now be able to poll /eventsKegan Dougal2015-02-261-7/+55
|
* Comment unused variables.Kegan Dougal2015-02-251-2/+2
|
* Add stub functions and work out execution flow to implement AS event stream ↵Kegan Dougal2015-02-251-0/+21
| | | | polling.
* Fix token formattingMark Haines2015-01-301-2/+2
|
* Filter the recent events before applying the limit when doing an incremental ↵Mark Haines2015-01-301-6/+15
| | | | sync with a gap
* Merge branch 'client_v2_filter' into client_v2_syncMark Haines2015-01-291-3/+3
|\ | | | | | | | | Conflicts: synapse/rest/client/v2_alpha/__init__.py
| * SYN-252: Supply the stream and topological parts in the correct order to the ↵Mark Haines2015-01-291-3/+3
| | | | | | | | constructor
* | Use get_room_events_stream to get changes to the rooms if the number of ↵Mark Haines2015-01-291-0/+7
| | | | | | | | changes is small
* | Fix FormattingMark Haines2015-01-271-1/+0
| |
* | Start implementing incremental initial syncMark Haines2015-01-271-9/+32
|/
* get_room_events isn't called anywhereMark Haines2015-01-131-30/+0
|
* Fix stream token orderingMark Haines2015-01-131-72/+101
|
* SYN-178: Fix off by one.Kegan Dougal2015-01-121-2/+6
|
* Only fetch the columns we need.Erik Johnston2015-01-061-1/+1
|
* Merge pull request #28 from matrix-org/erikj-perfMark Haines2015-01-061-77/+68
|\ | | | | Database performance improvements.
| * PEP8Erik Johnston2015-01-061-1/+0
| |
| * Only fetch prev_content when a client is streaming/paginating. Use ↵Erik Johnston2015-01-061-77/+69
| | | | | | | | transactions for event streams.
* | Update copyright noticesMark Haines2015-01-061-1/+1
|/
* Don't return outliers when we get recent events for rooms.Erik Johnston2014-11-271-1/+1
|
* Fix pep8 codestyle warningsMark Haines2014-11-201-5/+6
|
* SYN-104: When going backwards the end token should be before the last eventMark Haines2014-11-181-0/+3
|
* Notify users about invites.Erik Johnston2014-11-101-2/+1
|
* Add transaction level logging and timing information. Add a _simple_delete ↵Erik Johnston2014-10-281-1/+4
| | | | method
* Add LIMIT to scalar subqueryErik Johnston2014-09-251-3/+6
|
* Rename deletions to redactionsErik Johnston2014-09-241-9/+9
|
* Add m.room.deletion. If an event is deleted it will be returned to clients ↵Erik Johnston2014-09-231-5/+25
| | | | 'pruned', i.e. all client specified keys will be removed.
* SYN-47: Fix bug where we still returned events for rooms we had left.Erik Johnston2014-09-191-1/+1
| | | | SYN-47 #resolve
* Add a .runInteraction() method on SQLBaseStore itself to wrap the .db_poolPaul "LeoNerd" Evans2014-09-121-1/+1
|
* When getting a state event also include the previous contentErik Johnston2014-09-061-8/+9
|
* fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org ↵Matthew Hodgson2014-09-031-1/+1
| | | | hasn't been incorporated in time for launch.
* Get the equalities right.Erik Johnston2014-08-291-2/+5
|
* Initial room event stream token must be s0, not s1, or everyone will miss ↵Paul "LeoNerd" Evans2014-08-281-1/+1
| | | | the very first room event
* Merge branch 'develop' into storage_transactionsMark Haines2014-08-271-2/+2
|\ | | | | | | | | | | Conflicts: synapse/handlers/room.py synapse/storage/stream.py
| * WIP: Completely change how event streaming and pagination work. This ↵Erik Johnston2014-08-261-2/+2
| | | | | | | | reflects the change in the underlying storage model.
* | Merge branch 'develop' into storage_transactionsMark Haines2014-08-261-7/+5
|\| | | | | | | | | | | | | Conflicts: synapse/api/auth.py synapse/handlers/room.py synapse/storage/__init__.py
| * Fix pyflakes errorsMark Haines2014-08-261-2/+0
| |
| * Order 'get_recent_events_for_room' correctly.Erik Johnston2014-08-261-1/+1
| |
| * Don't return state event outlier's when paginating.Erik Johnston2014-08-221-1/+2
| |
| * Change IM sync api to also return the current presence list.Erik Johnston2014-08-211-3/+2
| |
* | Move the event storage into a single transactionMark Haines2014-08-261-5/+8
|/
* Make event stream storage return all membership events about the user, ↵Erik Johnston2014-08-201-4/+6
| | | | regardless of if they were in the room or not.
* Only hit get_room_events_stream if we have a valid user_idErik Johnston2014-08-191-0/+3
|
* Fix token to correct formatErik Johnston2014-08-191-1/+1
|
* Remove debug logging from token parsing funcs.Erik Johnston2014-08-191-2/+0
|
* Add comment about what strorage.stream doesErik Johnston2014-08-191-0/+20
|
* Change the way pagination works to support out of order events.Erik Johnston2014-08-191-29/+157
|
* Fix typos in SQL and where we still had rowid's (which no longer exist)Erik Johnston2014-08-181-6/+6
|
* Make snapshot_all_rooms return results in the correct form, including start ↵Erik Johnston2014-08-181-4/+21
| | | | and end tokens.
* Add two different columns for ordering the events table, one which can be ↵Erik Johnston2014-08-181-8/+9
| | | | used for pagination and one which can be as tokens for notifying clients. Also add a 'processed' field which is currently always set to True
* PEP8 cleanupsErik Johnston2014-08-151-2/+0
|
* Comment.Erik Johnston2014-08-151-0/+1
|
* Fix bug when generating a key when get_room_events_stream returned zero rowsErik Johnston2014-08-151-3/+7
|
* Actually use MAX_STREAM_SIZE constant.Erik Johnston2014-08-151-1/+1
|
* Fix pagination to work with new db schemaErik Johnston2014-08-151-7/+24
|
* Correctly return new token when returning events. Serialize events correctly.Erik Johnston2014-08-151-5/+13
|
* Start chagning the events stream to work with the new DB schemaErik Johnston2014-08-151-0/+31
|
* Fix up typos and correct sql queriesErik Johnston2014-08-141-7/+4
|
* Update StreamStoreErik Johnston2014-08-141-245/+36
|
* add in whitespace after copyright statements to improve legibilityMatthew Hodgson2014-08-131-0/+1
|
* Reference Matrix Home Servermatrix.org2014-08-121-0/+282
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-02 21:31:19 +0000