summary refs log tree commit diff
path: root/synapse/streams (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix regression when bounding future tokens (#17391)Erik Johnston2024-07-021-4/+22
| | | | | | Fix bug added in #17386, where we accidentally used `room_key` for the receipts stream. See first commit. Reviewable commit-by-commit
* Fix sync waiting for an invalid token from the "future" (#17386)Erik Johnston2024-07-021-1/+63
| | | | | | | | | | | | | Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.
* Tidy up integer parsing (#17339)Denis Kasak2024-06-241-3/+0
| | | | | | | | | | The parse_integer function was previously made to reject negative values by default in https://github.com/element-hq/synapse/pull/16920, but the documentation stated otherwise. This fixes the documentation and also: - Removes explicit negative=False parameters from call sites. - Brings the negative default of parse_integer_from_args in alignment with parse_integer.
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-131-4/+5
|
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-233-0/+3
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Update license headersPatrick Cloke2023-11-213-30/+48
|
* Allow multiple workers to write to receipts stream. (#16432)Erik Johnston2023-10-251-2/+2
| | | Fixes #16417
* Some refactors around receipts stream (#16426)Erik Johnston2023-10-041-5/+10
|
* Use mypy 1.0 (#15052)David Robertson2023-02-161-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Add helper to parse an enum from query args & use it. (#14956)Patrick Cloke2023-02-011-9/+3
| | | | | | | | The `parse_enum` helper pulls an enum value from the query string (by delegating down to the parse_string helper with values generated from the enum). This is used to pull out "f" and "b" in most places and then we thread the resulting Direction enum throughout more code.
* Use an enum for direction. (#14927)Patrick Cloke2023-01-271-3/+8
| | | | For better type safety we use an enum instead of strings to configure direction (backwards or forwards).
* Fix a bug in the send_local_online_presence_to module API (#14880)Patrick Cloke2023-01-251-3/+3
| | | | | | | Destination was being used incorrectly (a single destination instead of a list of destinations was being passed). This also updates some of the types in the area to not use Collection[str], which is a footgun.
* Faster joins: omit partial rooms from eager syncs until the resync completes ↵David Robertson2023-01-231-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14870) * Allow `AbstractSet` in `StrCollection` Or else frozensets are excluded. This will be useful in an upcoming commit where I plan to change a function that accepts `List[str]` to accept `StrCollection` instead. * `rooms_to_exclude` -> `rooms_to_exclude_globally` I am about to make use of this exclusion mechanism to exclude rooms for a specific user and a specific sync. This rename helps to clarify the distinction between the global config and the rooms to exclude for a specific sync. * Better function names for internal sync methods * Track a list of excluded rooms on SyncResultBuilder I plan to feed a list of partially stated rooms for this sync to ignore * Exclude partial state rooms during eager sync using the mechanism established in the previous commit * Track un-partial-state stream in sync tokens So that we can work out which rooms have become fully-stated during a given sync period. * Fix mutation of `@cached` return value This was fouling up a complement test added alongside this PR. Excluding a room would mean the set of forgotten rooms in the cache would be extended. This means that room could be erroneously considered forgotten in the future. Introduced in #12310, Synapse 1.57.0. I don't think this had any user-visible side effects (until now). * SyncResultBuilder: track rooms to force as newly joined Similar plan as before. We've omitted rooms from certain sync responses; now we establish the mechanism to reintroduce them into future syncs. * Read new field, to present rooms as newly joined * Force un-partial-stated rooms to be newly-joined for eager incremental syncs only, provided they're still fully stated * Notify user stream listeners to wake up long polling syncs * Changelog * Typo fix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Unnecessary list cast Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Rephrase comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Another comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Fixup merge(?) * Poke notifier when receiving un-partial-stated msg over replication * Fixup merge whoops Thanks MV :) Co-authored-by: Mathieu Velen <mathieuv@matrix.org> Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Implement message forward pagination from start when no from is given, fixes ↵Benjamin Kampmann2022-11-241-0/+13
| | | | | #12383 (#14149) Fixes https://github.com/matrix-org/synapse/issues/12383
* Remove unused type-ignores (#14433)David Robertson2022-11-141-3/+6
| | | | | | | * Remove unused type-ignores Oversights in #14427 and #14429. * Changelog
* Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke2022-10-142-8/+6
| | | | | | | The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
* Remove support for the unstable dir flag on relations. (#14106)Patrick Cloke2022-10-071-2/+4
| | | | | | From MSC3715, this was unused by clients (and there was no way for clients to know it was supported). Matrix 1.4 defines the stable field.
* Instrument `/messages` for understandable traces in Jaeger (#13368)Eric Eastwood2022-08-031-0/+2
| | | | | | In Jaeger: - Before: huge list of uncategorized database calls - After: nice and collapsible into units of work
* Rework stream token to stop caring about groups. (#12897)Patrick Cloke2022-05-311-2/+2
|
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-2/+2
| | | | provided (#12370)
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Upgrade mypy to version 0.931 (#12030)Sean Quah2022-02-181-2/+4
| | | Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-5/+5
|
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-1/+1
|
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-212-19/+52
|
* Add a return type to parse_string. (#10438)Patrick Cloke2021-07-211-7/+9
| | | | And set the required attribute in a few places which will error if a parameter is not provided.
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-2/+2
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-143-3/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Make token serializing/deserializing async (#8427)Erik Johnston2020-09-301-4/+5
| | | The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
* Clean up types for PaginationConfig (#8250)Erik Johnston2020-09-081-40/+21
| | | This removes `SourcePaginationConfig` and `get_pagination_rows`. The reasoning behind this is that these generic classes/functions erased the types of the IDs it used (i.e. instead of passing around `StreamToken` it'd pass in e.g. `token.room_key`, which don't have uniform types).
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-3/+3
|
* Remove `ChainedIdGenerator`. (#8123)Erik Johnston2020-08-191-1/+1
| | | | | It's just a thin wrapper around two ID gens to make `get_current_token` and `get_next` return tuples. This can easily be replaced by calling the appropriate methods on the underlying ID gens directly.
* Convert streams to async. (#8014)Patrick Cloke2020-08-041-13/+9
|
* Fix some spelling mistakes / typos. (#7811)Patrick Cloke2020-07-092-3/+3
|
* Bump version of mypyErik Johnston2019-12-121-1/+3
|
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-3/+6
|
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Replace returnValue with return (#5736)Amber Brown2019-07-231-2/+2
|
* Fix /messages on workers when no from param specified.Erik Johnston2019-06-241-14/+18
| | | | | | | If no `from` param is specified we calculate and use the "current token" that inlcuded typing, presence, etc. These are unused during pagination and are not available on workers, so we simply don't calculate them.
* Run Black. (#5482)Amber Brown2019-06-202-46/+24
|
* Fix develop because I broke it :( (#3535)Amber Brown2018-07-141-0/+3
|
* Use parse_{int,str} and assert from http.servletKrombel2018-07-131-23/+6
| | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable
* run isortAmber Brown2018-07-092-7/+5
|
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-3/+3
| | | | what could possibly go wrong
* Fix bug in generating current tokenErik Johnston2017-07-201-0/+2
|
* Add groups to sync streamErik Johnston2017-07-201-0/+2
|
* Add basic implementation of local device list changesErik Johnston2017-01-251-0/+4
|
* Actually use the new functionErik Johnston2016-10-241-1/+1
|
* Fix incredubly slow back pagination queryErik Johnston2016-10-241-2/+28
| | | | | | | If a client didn't specify a from token when paginating backwards synapse would attempt to query the (global) maximum topological token. This a) doesn't make much sense since they're room specific and b) there are no indices that lets postgres do this efficiently.
* Add store-and-forward direct-to-device messagingMark Haines2016-08-251-0/+2
|
* Hook up the push rules to the notifierMark Haines2016-03-031-0/+4
|
* Clamp pagination limits to at most 1000Erik Johnston2016-01-141-2/+5
|
* copyrightsMatthew Hodgson2016-01-073-3/+3
|
* s/private_user_data/account_data/Mark Haines2015-11-181-4/+4
|
* Inform the client of new room tags using v1 /eventsMark Haines2015-10-291-0/+5
|
* Move NullSource out of synapse and into tests since it is only used by the testsMark Haines2015-09-221-16/+0
|
* synapse/streams/events.py:StreamSource was unusedMark Haines2015-09-221-12/+0
|
* Define __repr__ methods for StreamConfig and PaginationConfigMark Haines2015-09-221-3/+8
| | | | So that they can be used with "%r" log formats.
* Wire together receipts and the notifer/federationErik Johnston2015-07-071-1/+5
|
* Add receipts_key to StreamTokenErik Johnston2015-07-021-1/+2
|
* Make get_current_token accept a direction parameter, which tells whether the ↵Erik Johnston2015-05-121-3/+3
| | | | source whether we want a token for going 'forwards' or 'backwards'
* Update copyright noticesMark Haines2015-01-063-3/+3
|
* Allow GET /room/:room_id/messages without 'limit' parameter to default to 10Paul "LeoNerd" Evans2014-12-021-1/+5
|
* SYN-58: Allow passing explicit limit=0 to initialSync to request no messages ↵Paul "LeoNerd" Evans2014-11-201-6/+6
| | | | at all; missing still implies default 10
* Construct a source-specific 'SourcePaginationConfig' to pass into ↵Paul "LeoNerd" Evans2014-10-292-1/+24
| | | | get_pagination_rows; meaning each source doesn't have to care about its own name any more
* fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org ↵Matthew Hodgson2014-09-033-3/+3
| | | | hasn't been incorporated in time for launch.
* Renamed get_current_token_part to get_current_keyPaul "LeoNerd" Evans2014-08-291-6/+6
|
* Have EventSource's get_new_events_for_user() API work only on keys within ↵Paul "LeoNerd" Evans2014-08-291-3/+4
| | | | that source, not overall eventstream tokens
* Rename 'events_key' to 'room_key' so it matches the name of the event sourcePaul "LeoNerd" Evans2014-08-291-1/+1
|
* Define a new event stream data source for typing notifications (currently null)Paul "LeoNerd" Evans2014-08-291-7/+13
|
* Move the *EventSource classes into the handlers they relate to, so it's ↵Paul "LeoNerd" Evans2014-08-291-127/+4
| | | | easier to find the code
* Add logging to try and figure out what is going on with the presence stuffErik Johnston2014-08-281-1/+1
|
* Fix typo in NullSource.get_pagination_rows. Remove unused import.Erik Johnston2014-08-281-2/+1
|
* Define a NullSource useful for unit-testingPaul "LeoNerd" Evans2014-08-271-0/+16
|
* Index sources in a nicer fashion.Erik Johnston2014-08-271-11/+10
|
* Convert get_paginat_rows to use PaginationConfig. This allows people to ↵Erik Johnston2014-08-272-33/+54
| | | | supply directions.
* PEP8 tweaks.Erik Johnston2014-08-272-5/+3
|
* Implement presence event source. Change the way the notifier indexes listenersErik Johnston2014-08-271-25/+48
|
* Enable presence again. Fix up api to match old api.Erik Johnston2014-08-261-2/+2
|
* Fix exceptions so that the event stream works. Presence like events are ↵Erik Johnston2014-08-261-2/+4
| | | | turned off currently.
* WIP: Completely change how event streaming and pagination work. This ↵Erik Johnston2014-08-263-0/+235
reflects the change in the underlying storage model.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-09-20 18:49:48 +0000