summary refs log tree commit diff
path: root/synapse/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Convert Transaction and Edu object to attrs (#10542)Patrick Cloke2021-08-061-102/+0
| | | | | Instead of wrapping the JSON into an object, this creates concrete instances for Transaction and Edu. This allows for improved type hints and simplified code.
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Generics for `ObservableDeferred` (#10491)Richard van der Hoff2021-07-281-6/+8
| | | | | Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
* Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff2021-07-283-12/+21
| | | Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
* improve typing annotations in CachedCall (#10450)Richard van der Hoff2021-07-281-10/+17
| | | | | tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-193-8/+6
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1516-59/+57
|
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-052-35/+352
|
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-15/+20
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Add debug logging when we enter/exit Measure block (#10183)Erik Johnston2021-06-161-0/+5
| | | It can be helpful to know when trying to track down slow requests.
* Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-141-26/+73
| | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* When joining a remote room limit the number of events we concurrently check ↵Erik Johnston2021-06-081-5/+16
| | | | | signatures/hashes for (#10117) If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
* Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-22/+48
| | | Fixes #10068
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-274-35/+36
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-244-21/+19
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-244-47/+70
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-5/+3
|
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+153
|
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-6/+8
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Simplify exception handling in is_ascii. (#9985)Dan Callahan2021-05-141-3/+1
| | | | | | | | | | | | | | | We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>
* Use Python's secrets module instead of random (#9984)Dan Callahan2021-05-141-8/+11
| | | | | | | | | | | | | | | | Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-121-0/+20
|
* Merge branch 'master' into developBrendan Abolivier2021-05-111-14/+47
|\
| * Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-111-14/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* | Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-052-1/+78
| | | | | | | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* | Reduce memory footprint of caches (#9886)Erik Johnston2021-04-281-18/+59
|/
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+1
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-0/+30
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-222-4/+2
| | | This is no longer required, since we have dropped support for Python 3.5.
* Merge branch 'master' into developAndrew Morgan2021-04-201-5/+9
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-5/+9
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak2021-04-191-0/+32
| | | | | | | | Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1432-33/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-6/+8
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-32/+51
|
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-1/+3
|
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-292-44/+73
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-2/+2
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-242-2/+2
|
* Fixed code misc. quality issues (#9649)Ankit Dobhal2021-03-221-1/+1
| | | | | - Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
* Add logging to ObservableDeferred callbacks (#9523)Jonathan de Jong2021-03-091-8/+18
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-6/+4
|
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-041-0/+89
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-32/+2
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-2/+32
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1611-48/+45
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-0/+129
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Merge tag 'v1.27.0rc2' into developPatrick Cloke2021-02-111-3/+12
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
| * Backout changes for automatically calculating the public baseurl. (#9313)Patrick Cloke2021-02-111-3/+12
| | | | | | | | This breaks some people's configurations (if their Client-Server API is not accessed via port 443).
* | Type hints and validation improvements. (#9321)Patrick Cloke2021-02-081-15/+18
|/ | | | | * Adds type hints to the groups servlet and stringutils code. * Assert the maximum length of some input values for spec compliance.
* Merge branch 'social_login' into developRichard van der Hoff2021-02-011-0/+106
|\
| * Replace username picker with a template (#9275)Richard van der Hoff2021-02-011-0/+106
| | | | | | | | | | There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
* | Pass a dict, instead of None, to modules if a None config is specified in ↵Andrew Morgan2021-01-271-1/+2
|/ | | | | | | | | the homeserver config (#9229) If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead. This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](https://github.com/matrix-org/synapse/blob/5310808d3bebd17275355ecd474bc013e8c7462d/docs/sample_config.yaml#L1816-L1845) states) then `None` is not passed to the module. An empty dict is passed instead. This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
* Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-1/+1
|
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-0/+92
|
* Remote dependency on distutils (#9125)Richard van der Hoff2021-01-151-0/+19
| | | | | | | | | `distutils` is pretty much deprecated these days, and replaced with `setuptools`. It's also annoying because it's you can't `pip install` it, and it's hard to figure out which debian package we should depend on to make sure it's there. Since we only use it for a tiny function anyway, let's just vendor said function into our codebase.
* Fix event chain bg update. (#9118)Erik Johnston2021-01-141-1/+1
| | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-1/+52
|
* Fix "Starting metrics collection from sentinel context" errors (#9053)Erik Johnston2021-01-081-1/+2
|
* Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-1/+1
|
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-1/+9
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-112-8/+7
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Better formatting for config errors from modules (#8874)Richard van der Hoff2020-12-081-6/+58
| | | | | | | | | | The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError, and it will be magically turned into a legible error message. There's a few components to it: * Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it into an absolute path. * Generally improving the way ConfigErrors get printed. * Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
* Implement and use an @lru_cache decorator (#8595)Richard van der Hoff2020-10-301-47/+188
| | | We don't always need the full power of a DeferredCache.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-282-25/+21
|\ | | | | Fix serialisation errors when using third-party event rules.
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-282-25/+21
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
|/ | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Optimise CacheDescriptor (#8594) github/release-v1.21.3 release-v1.21.3Richard van der Hoff2020-10-211-5/+7
| | | don't bother constricting a CacheContext unless we need one.
* fix failure caseRichard van der Hoff2020-10-211-1/+4
|
* optimise DeferredCache.setRichard van der Hoff2020-10-211-3/+12
|
* Push some deferred wrangling down into DeferredCacheRichard van der Hoff2020-10-212-34/+55
|
* Fix 'LruCache' object has no attribute '_on_resize' (#8591)Richard van der Hoff2020-10-191-1/+9
| | | We need to make sure we are readu for the `set_cache_factor` callback.
* Add `DeferredCache.get_immediate` method (#8568)Richard van der Hoff2020-10-191-10/+25
| | | | | | | | | | | * Add `DeferredCache.get_immediate` method A bunch of things that are currently calling `DeferredCache.get` are only really interested in the result if it's completed. We can optimise and simplify this case. * Remove unused 'default' parameter to DeferredCache.get() * another get_immediate instance
* Replace DeferredCache with LruCache where possible (#8563)Richard van der Hoff2020-10-191-0/+3
| | | Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
* review commentsRichard van der Hoff2020-10-161-3/+8
|
* Apply suggestions from code reviewRichard van der Hoff2020-10-161-1/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* type annotations for LruCacheRichard van der Hoff2020-10-163-23/+77
|
* Make LruCache register its own metrics (#8561)Richard van der Hoff2020-10-164-54/+57
| | | | | rather than have everything that instantiates an LruCache manage metrics separately, have LruCache do it itself.
* Enable mypy for synapse.util.caches (#8547)Richard van der Hoff2020-10-151-1/+1
| | | This seemed to entail dragging in a type stub for SortedList.
* move DeferredCache into its own moduleRichard van der Hoff2020-10-142-282/+294
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-7/+12
|
* Add some more type annotations to CacheRichard van der Hoff2020-10-142-23/+61
|
* Add type hints to response cache. (#8507)Patrick Cloke2020-10-091-22/+28
|
* Clarify error message when plugin config parsers raise an error (#8492)Richard van der Hoff2020-10-081-1/+1
| | | | | | | | | This turns: Failed to parse config for 'myplugin': Exception('error message') into: Failed to parse config for 'myplugin': error message.
* Expose a `get_resource_usage` method in `Measure`Richard van der Hoff2020-09-291-10/+21
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-291-29/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Remove obsolete __future__ imports (#8337)Jonathan de Jong2020-09-171-2/+0
|
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Merge tag 'v1.20.0rc3' into developPatrick Cloke2020-09-111-1/+1
|\ | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc3 (2020-09-11) ============================== Bugfixes -------- - Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
| * Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+1
| |
* | Remove some unused distributor signals (#8216)Patrick Cloke2020-09-091-42/+8
| | | | | | | | | | Removes the `user_joined_room` and stops calling it since there are no observers. Also cleans-up some other unused signals and related code.
* | Add types to async_helpers (#8260)Patrick Cloke2020-09-081-50/+85
| |
* | Directly import json from the standard library. (#8259)Patrick Cloke2020-09-082-3/+4
|/ | | | | By importing from canonicaljson the simplejson module was still being used in some situations. After this change the std lib json is consistenty used throughout Synapse.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-0417-31/+31
|
* Fix typing for `@cached` wrapped functions (#8240)Erik Johnston2020-09-031-14/+28
| | | This requires adding a mypy plugin to fiddle with the type signatures a bit.
* Convert ReadWriteLock to async/await. (#8202)Patrick Cloke2020-08-281-8/+8
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+12
|
* Remove the unused inlineCallbacks code-paths in the caching code (#8119)Patrick Cloke2020-08-191-46/+8
|
* Remove : from allowed client_secret chars (#8101)Andrew Morgan2020-08-181-3/+1
| | | | | | | Closes: https://github.com/matrix-org/synapse/issues/6766 Equivalent Sydent PR: https://github.com/matrix-org/sydent/pull/309 I believe it's now time to remove the extra allowed `:` from `client_secret` parameters.
* Fix typing for notifier (#8064)Erik Johnston2020-08-121-3/+6
|
* Remove unnecessary maybeDeferred calls (#8044)Patrick Cloke2020-08-071-1/+1
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-072-2/+9
|
* Convert some util functions to async (#8035)Patrick Cloke2020-08-062-28/+27
|
* Stop the parent process flushing the logs on exit (#8012)Richard van der Hoff2020-08-051-2/+8
| | | This solves the problem that the first few lines are logged twice on matrix.org. Hopefully the comments explain it.
* re-implement daemonize (#8011)Richard van der Hoff2020-08-041-0/+131
| | | | | | | | | | | | | This has long been something I've wanted to do. Basically the `Daemonize` code is both too flexible and not flexible enough, in that it offers a bunch of features that we don't use (changing UID, closing FDs in the child, logging to syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err to a file instead of /dev/null; having the parent not exit until the child is running). As a first step, I've lifted the Daemonize code and removed the bits we don't use. This should be a non-functional change. Fixing everything else will come later.
* Fix deprecation warning: import ABC from collections.abc (#7892)Karthikeyan Singaravelan2020-07-201-1/+1
|
* Convert device handler to async/await (#7871)Patrick Cloke2020-07-171-2/+26
|
* Fix some spelling mistakes / typos. (#7811)Patrick Cloke2020-07-096-7/+7
|
* Fix inconsistent handling of upper and lower cases of email addresses. (#7021)Dirk Klimpel2020-07-031-0/+23
| | | fixes #7016
* Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708)Patrick Cloke2020-06-161-1/+1
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-165-12/+4
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-153-10/+4
|
* Performance improvements and refactor of Ratelimiter (#7595)Andrew Morgan2020-06-051-1/+1
| | | | | | | | | | While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both: * Rather undocumented, and * causing a *lot* of config checks This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. Best to be reviewed commit-by-commit.
* Speed up processing of federation stream RDATA rows.Erik Johnston2020-05-271-0/+12
| | | | | | Instead of storing and sending an ACK for every single row we send synchronously, we instead do it asynchronously while batching up updates.
* Don't apply cache factor to event cache. (#7578)Erik Johnston2020-05-271-0/+4
| | | | This is already correctly done when we instansiate the cache, but wasn't when it got reloaded (which always happens at least once on startup).
* Fix stacktrace mangling in `patch_inline_callbacks` (#7554)Richard van der Hoff2020-05-221-2/+7
| | | `Failure()` is more cunning than `Failure(e)`.
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-152-27/+8
|
* remove to_asciiRichard van der Hoff2020-05-151-19/+1
| | | | this is a no-op on python 3.
* Remove `exception_to_unicode`Richard van der Hoff2020-05-151-36/+0
| | | | this is a no-op on python 3.
* Strictly enforce canonicaljson requirements in a new room version (#7381)Patrick Cloke2020-05-141-1/+1
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-117-92/+206
| | | | variables (#6391)
* Speed up fetching device lists changes in sync.Erik Johnston2020-05-051-4/+15
| | | | | Currently we copy `users_who_share_room` needlessly about three times, which is expensive when the set is large (which it can easily be).
* Extend StreamChangeCache to support multiple entities per stream ID (#7303)Richard van der Hoff2020-04-221-46/+71
| | | | | | | | | | | | | | | | | | | First some background: StreamChangeCache is used to keep track of what "entities" have changed since a given stream ID. So for example, we might use it to keep track of when the last to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2]. Now, it turns out that StreamChangeCache didn't support more than one thing being changed at a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send to-device messages to more than one user at a time. As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped ok with having multiple things changing on the same stream ID, and it seems we never actually use the methods which don't work on the stream change caches where we allow multiple changes at the same stream ID. But that feels horribly fragile, hence: let's update StreamChangeCache to properly support this, and add some typing and some more tests while we're at it. [1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301 [2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
* On catchup, process each row with its own stream id (#7286)Richard van der Hoff2020-04-201-0/+3
| | | | | | Other parts of the code (such as the StreamChangeCache) assume that there will not be multiple changes with the same stream id. This code was introduced in #7024, and I hope this fixes #7206.
* Rewrite prune_old_outbound_device_pokes for efficiency (#7159)Richard van der Hoff2020-03-301-1/+20
| | | | make sure we clear out all but one update for the user
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-242-20/+20
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Fix stacktraces when using ObservableDeferred and async/await (#6836)Erik Johnston2020-02-031-0/+4
|
* Validate client_secret parameter (#6767)Andrew Morgan2020-01-241-0/+17
|
* Log saml assertions rather than the whole responseRichard van der Hoff2020-01-161-0/+13
| | | | | | ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.
* move batch_iter to a separate moduleRichard van der Hoff2020-01-162-17/+35
|
* Handle `config` not being set for synapse plugin modulesRichard van der Hoff2020-01-121-1/+1
| | | | | Some modules don't need any config, so having to define a `config` property just to keep the loader happy is a bit annoying.
* Persist auth/state events at backwards extremities when we fetch them (#6526)Richard van der Hoff2019-12-161-2/+2
| | | The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
* look up cross-signing keys from the DB in bulk (#6486)Hubert Chathi2019-12-121-1/+1
|
* Remove SnapshotCache in favour of ResponseCacheErik Johnston2019-12-091-94/+0
|
* Fix inaccurate per-block metrics (#6491)Richard van der Hoff2019-12-091-42/+18
| | | | | `Measure` incorrectly assumed that it was the only thing being done by the parent `LoggingContext`. For instance, during a "renew group attestations" operation, hundreds of `outbound_request` calls could take place in parallel, all using the same `LoggingContext`. This would mean that any resources used during *any* of those calls would be reported against *all* of them, producing wildly inaccurate results. Instead, we now give each `Measure` block its own `LoggingContext` (using the parent `LoggingContext` mechanism to ensure that the log lines look correct and that the metrics are ultimately propogated to the top level for reporting against requests/backgrond tasks).
* Port SyncHandler to async/awaitErik Johnston2019-12-051-6/+17
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Fix LruCache callback deduplication (#6213)V024602019-11-071-11/+37
|
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-314-6/+6
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Clarify docstringErik Johnston2019-10-301-0/+4
|
* Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-5/+2
| | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
* Handle FileNotFound error in checking git repository version (#6284)Andrew Morgan2019-10-301-4/+6
|
* Make concurrently_execute work with async/awaitErik Johnston2019-10-291-4/+3
|
* Update docstringErik Johnston2019-10-291-3/+2
|
* Quick fix to ensure cache descriptors always return deferredsErik Johnston2019-10-281-2/+2
|
* Add maybe_awaitable and fix __init__ bugsErik Johnston2019-10-111-0/+29
|
* Fixup commentsErik Johnston2019-10-101-3/+3
| | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix py3.5Erik Johnston2019-10-101-1/+1
|
* Fix py3.5Erik Johnston2019-10-101-1/+1
|
* sortErik Johnston2019-10-101-3/+1
|
* Appease mypyErik Johnston2019-10-101-13/+18
|
* Add comentsErik Johnston2019-10-101-5/+25
|
* Log correct contextErik Johnston2019-10-101-6/+6
|
* Test for sentinel commitErik Johnston2019-10-101-2/+19
|
* Move patch_inline_callbacks into synapse/Erik Johnston2019-10-101-0/+179
|
* add some metrics on the federation sender (#6160)Richard van der Hoff2019-10-031-2/+4
|
* Fix up some typechecking (#6150)Amber Brown2019-10-025-8/+33
| | | | | | * type checking fixes * changelog
* Fix errors storing large retry intervals.Erik Johnston2019-10-021-1/+1
| | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
* Merge branch 'develop' into rav/fix_attribute_mappingRichard van der Hoff2019-09-193-21/+68
|\
| * Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-1/+15
| | | | | | | | Track the time that a server started failing at, for general analysis purposes.
| * Remove the cap on federation retry interval. (#6026)Richard van der Hoff2019-09-121-2/+2
| | | | | | | | | | | | Essentially the intention here is to end up blacklisting servers which never respond to federation requests. Fixes https://github.com/matrix-org/synapse/issues/5113.
| * Fix bug in calculating the federation retry backoff period (#6025)Richard van der Hoff2019-09-121-2/+3
| | | | | | | | This was intended to introduce an element of jitter; instead it gave you a 30/60 chance of resetting to zero.
| * Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)Andrew Morgan2019-09-111-0/+33
| | | | | | | | | | | | | | This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted. Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140). Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
| * Clean up some code in the retry logic (#6017)Richard van der Hoff2019-09-111-16/+13
| | | | | | | | * remove some unused code * make things which were constants into constants for efficiency and clarity
| * Revert "Use the v2 lookup API for 3PID invites (#5897)" (#5937)Andrew Morgan2019-08-301-33/+0
| | | | | | | | | | This reverts commit 71fc04069a5770a204c3514e0237d7374df257a8. This broke 3PID invites as #5892 was required for it to work correctly.
| * Use the v2 lookup API for 3PID invites (#5897)Andrew Morgan2019-08-281-0/+33
| | | | | | | | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/5861 Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites. Sytest PR: https://github.com/matrix-org/sytest/pull/679
| * Retry well-known lookup before expiry.Erik Johnston2019-08-131-3/+5
| | | | | | | | | | | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* | Fix a bug with saml attribute maps.Richard van der Hoff2019-09-191-1/+19
|/ | | | | | | | | | | | | Fixes a bug where the default attribute maps were prioritised over user-specified ones, resulting in incorrect mappings. The problem is that if you call SPConfig.load() multiple times, it adds new attribute mappers to a list. So by calling it with the default config first, and then the user-specified config, we would always get the default mappers before the user-specified mappers. To solve this, let's merge the config dicts first, and then pass them to SPConfig.
* Add kwargs and docBrendan Abolivier2019-07-291-2/+4
|
* Add ability to pass arguments to looping callsBrendan Abolivier2019-07-291-2/+2
|
* Fix some error cases in the caching layer. (#5749)Richard van der Hoff2019-07-251-32/+42
| | | | | | | There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.
* Add a prometheus metric for active cache lookups. (#5750)Richard van der Hoff2019-07-242-2/+33
| | | | | | * Add a prometheus metric for active cache lookups. * changelog
* Replace returnValue with return (#5736)Amber Brown2019-07-236-15/+13
|
* Cache get_version_string.Erik Johnston2019-07-221-2/+21
| | | | | | | | | The version of a module isn't going to change over the lifetime of the process (assuming no funky hot reloading is going on, which it isn't), so let's just cache the result to avoid spawning lots of git subprocesses. Fixes #5672.
* Fixes to the federation rate limiter (#5621)Richard van der Hoff2019-07-051-8/+8
| | | | | | | - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request
* Improve the backwards compatibility re-exports of synapse.logging.context ↵Amber Brown2019-07-053-5/+61
| | | | | | | | (#5617) * Improve the backwards compatibility re-exports of synapse.logging.context. * reexport logformatter too
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-0412-962/+26
|
* Fix 'utime went backwards' errors on daemonization. (#5609)Richard van der Hoff2019-07-031-4/+13
| | | | | | | | * Fix 'utime went backwards' errors on daemonization. Fixes #5608 * remove spurious debug
* Fix a number of "Starting txn from sentinel context" warnings (#5605)Richard van der Hoff2019-07-031-1/+7
| | | | Fixes #5602, #5603
* Fix media repo breaking (#5593)Amber Brown2019-07-021-2/+7
|
* Prevent multiple upgrades on the same room at once (#5051)Andrew Morgan2019-06-251-1/+1
| | | | | | | Closes #4583 Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time. Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.
* Avoid raising exceptions in metricsRichard van der Hoff2019-06-241-8/+14
| | | | | Sentry will catch the errors if they happen, so that should be good enough, and woun't make things explode if we hit the error condition.
* Merge branch 'develop' into rav/cleanup_metricsRichard van der Hoff2019-06-2427-314/+317
|\
| * Run Black. (#5482)Amber Brown2019-06-2027-314/+317
| |
* | Sanity-checking for metrics updatesRichard van der Hoff2019-06-191-7/+33
|/ | | | Check that our clocks go forward.
* Call RetryLimiter correctly (#5340)Richard van der Hoff2019-06-041-1/+6
| | | Fixes a regression introduced in #5335.
* Avoid rapidly backing-off a server if we ignore the retry intervalRichard van der Hoff2019-06-031-23/+37
|
* Improve logging for logcontext leaks. (#5288)Richard van der Hoff2019-05-291-9/+13
|
* Make all the rate limiting options more consistent (#5181)Amber Brown2019-05-151-32/+15
|
* Merge pull request #5183 from matrix-org/erikj/async_serialize_eventErik Johnston2019-05-151-0/+19
|\ | | | | Allow client event serialization to be async
| * Update docstring with correct return typeErik Johnston2019-05-151-1/+1
| | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Allow client event serialization to be asyncErik Johnston2019-05-141-0/+19
| |
* | comment about user_joined_roomRichard van der Hoff2019-05-141-0/+1
|/
* Merge branch 'master' into developRichard van der Hoff2019-05-031-2/+7
|\
| * Use SystemRandom for token generationRichard van der Hoff2019-05-031-2/+7
| |
* | Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
| |
* | Fix disappearing exceptions in manhole. (#5035)Richard van der Hoff2019-04-101-2/+57
|/ | | Avoid sending syntax errors from the manhole to sentry.
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-0/+161
|
* Merge pull request #4486 from xperimental/workaround-4216Richard van der Hoff2019-01-301-1/+4
|\ | | | | Implement workaround for login error.
| * Implement workaround for login error.Robert Jacob2019-01-301-1/+4
| | | | | | | | Signed-off-by: Robert Jacob <xperimental@solidproject.de>
* | Make linearizer more quiet (#4507)Amber Brown2019-01-291-5/+5
|/
* Fix incorrect logcontexts after a Deferred was cancelled (#4407)Richard van der Hoff2019-01-171-1/+3
|
* Fix UnicodeDecodeError when postgres is not configured in english (#4253)Richard van der Hoff2018-12-041-1/+38
| | | | This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-253-54/+76
|\ | | | | | | erikj/alias_disallow_list
| * Correctly account for cpu usage by background threads (#4074)Richard van der Hoff2018-10-231-51/+69
| | | | | | | | | | | | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064
| * Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase ↵Amber Brown2018-10-201-1/+3
| | | | | | | | on py3) (#4068)
| * Fix manhole on py3 (pt 2) (#4067)Amber Brown2018-10-191-0/+2
| |
| * make a bytestringAmber Brown2018-10-191-2/+2
| |
* | Anchor returned regex to start and end of stringErik Johnston2018-10-191-2/+6
| |
* | Add config option to control alias creationErik Johnston2018-10-191-0/+21
|/
* Remove unnecessary extra function call layerErik Johnston2018-10-081-16/+13
|
* Use errback pattern and catch async failuresErik Johnston2018-10-081-14/+29
|
* Log looping call exceptionsErik Johnston2018-10-051-1/+18
| | | | | | | | If a looping call function errors, then it kills the loop entirely. Currently it throws away the exception logs, so we should make it actually log them. Fixes #3929
* Correctly match 'dict.pop' apiErik Johnston2018-10-011-3/+11
|
* Don't update eviction metrics on explicit removalErik Johnston2018-10-011-5/+0
|
* Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cacheRichard van der Hoff2018-09-281-4/+37
|\
| * Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-4/+37
| | | | | | | | | | | | | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
* | Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cacheRichard van der Hoff2018-09-261-10/+17
|\|
| * Log which cache is throwing exceptionsRichard van der Hoff2018-09-261-10/+17
| |
| * Fix ExpiringCache.__len__ to be accurateErik Johnston2018-09-261-10/+7
| | | | | | | | | | | | | | | | | | | | | | | | It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)
* | Fix ExpiringCache.__len__ to be accurateErik Johnston2018-09-211-12/+9
| | | | | | | | | | | | It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.
* | Add a five minute cache to get_destination_retry_timingsErik Johnston2018-09-211-0/+13
| | | | | | | | Hopefully helps with #3931
* | Make ExpiringCache slightly more performantErik Johnston2018-09-211-1/+5
|/
* Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* Improve the logging when handling a federation transaction (#3904)Richard van der Hoff2018-09-191-1/+1
| | | | | | | | | | Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.
* Replace custom DeferredTimeoutError with defer.TimeoutErrorErik Johnston2018-09-191-9/+3
|
* Run canceller first to allow it to generate correct errorErik Johnston2018-09-191-2/+5
|
* Update to use new timeout function everywhere.Erik Johnston2018-09-191-54/+19
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* Fix timeout functionErik Johnston2018-09-151-1/+2
| | | | | Turns out deferred.cancel sometimes throws, so we do that last to ensure that we always do resolve the new deferred.
* Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+51
| | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
* Add in flight real time metrics for Measure blocksErik Johnston2018-09-141-0/+22
|
* Change the manhole SSH key to have more bitsErik Johnston2018-09-111-13/+31
| | | | | Newer versions of openssh client refuse to connect to the old key due to its length.
* Fix exceptions when a connection is closed before we read the headersRichard van der Hoff2018-08-201-1/+3
| | | | | This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.
* Robustness fix for logcontext filterRichard van der Hoff2018-08-201-1/+7
| | | | | Make the logcontext filter not explode if it somehow ends up with a logcontext of None, since that infinite-loops the whole logging system.
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-203-8/+29
|
* Merge branch 'rav/fix_linearizer_cancellation' into developRichard van der Hoff2018-08-101-43/+68
|\
| * Fix linearizer cancellation on twisted < 18.7Richard van der Hoff2018-08-101-43/+68
| | | | | | | | | | | | Turns out that cancellation of inlineDeferreds didn't really work properly until Twisted 18.7. This commit refactors Linearizer.queue to avoid inlineCallbacks.
* | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-105-4/+4
|/
* Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-3/+3
|
* fix invalidationRichard van der Hoff2018-07-271-1/+1
|
* Rewrite cache list decoratorRichard van der Hoff2018-07-271-67/+64
| | | | | Because it was complicated and annoyed me. I suspect this will be more efficient too.
* Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-261-1/+1
| | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* Test and fix support for cancellation in LinearizerRichard van der Hoff2018-07-201-6/+22
|
* Combine Limiter and LinearizerRichard van der Hoff2018-07-201-89/+10
| | | | | Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.
* Improvements to the LimiterRichard van der Hoff2018-07-201-13/+20
| | | | | * give them names, to improve logging * use a deque rather than a list for efficiency
* Add a sleep to the Limiter to fix stack overflows.Richard van der Hoff2018-07-201-3/+20
| | | | Fixes #3570