summary refs log tree commit diff
path: root/synapse/logging (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-221-0/+1
|
* Use mypy 1.0 (#15052)David Robertson2023-02-161-7/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Tag federation request spans with the worker name (#15042)David Robertson2023-02-091-1/+9
| | | | | | | * Systematically include worker name as process info * Changelog * don't bother with inner setdefault
* Tag /send_join responses to detect faster joins (#14950)David Robertson2023-01-311-0/+5
| | | | | | | | | * Tag /send_join responses to detect faster joins * Changelog * Define a proper SynapseTag * isort
* Improve logging and opentracing for to-device message handling (#14598)Richard van der Hoff2022-12-061-2/+9
| | | | | | | A batch of changes intended to make it easier to trace to-device messages through the system. The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.) I've also generally improved the data we send to opentracing for these messages.
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-162-22/+21
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Update mypy and mypy-zope, attempt 3 (#13993)David Robertson2022-09-302-12/+12
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Revert "Update mypy and mypy-zope (#13925)"David Robertson2022-09-302-12/+12
| | | | This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
* Update mypy and mypy-zope (#13925)David Robertson2022-09-302-12/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Tag trace with instance name (#13761)Eric Eastwood2022-09-091-2/+4
| | | | | | | | We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load. As suggested by @clokep and @reivilibre in, - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
* Strip number suffix from instance name to consolidate services that traces ↵Eric Eastwood2022-09-091-1/+12
| | | | | | | | | | | | are spread over (#13729) The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985 Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through. With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇 Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.
* Instrument the federation/backfill part of `/messages` (#13489)Eric Eastwood2022-08-161-3/+16
| | | | | | | | | Instrument the federation/backfill part of `/messages` so it's easier to follow what's going on in Jaeger when viewing a trace. Split out from https://github.com/matrix-org/synapse/pull/13440 Follow-up from https://github.com/matrix-org/synapse/pull/13368 Part of https://github.com/matrix-org/synapse/issues/13356
* Revert "Update locked versions of mypy and mypy-zope (#13521)"David Robertson2022-08-152-12/+12
| | | | | | | | This reverts commit f383b9b3eceaa082d5ae690550fe41460b711779. Other PRs were seeing mypy failures that looked to be related to mypy-zope. Confusingly, we didn't see this on #13521. Revert this for now and investigate later.
* Update locked versions of mypy and mypy-zope (#13521)David Robertson2022-08-152-12/+12
|
* Allow use of both `@trace` and `@tag_args` stacked on the same function (#13453)Eric Eastwood2022-08-091-56/+102
| | | | | | | | | | | | | ```py @trace @tag_args async def get_oldest_event_ids_with_depth_in_room(...) ... ``` Before this PR, you would see a warning in the logs and the span was not exported: ``` 2022-08-03 19:11:59,383 - synapse.logging.opentracing - 835 - ERROR - GET-0 - @trace may not have wrapped EventFederationWorkerStore.get_oldest_event_ids_with_depth_in_room correctly! The function is not async but returned a coroutine. ```
* Fix `@tag_args` being off-by-one (ahead) (#13452)Eric Eastwood2022-08-041-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix @tag_args being off-by-one (ahead) Example: ``` argspec.args=[ 'self', 'room_id' ] args=( <synapse.storage.databases.main.DataStore object at 0x10d0b8d00>, '!HBehERstyQBxyJDLfR:my.synapse.server' ) ``` --- The previous logic was also flawed and we can end up in a situation like this: ``` argspec.args=['self', 'dest', 'room_id', 'limit', 'extremities'] args=(<synapse.federation.federation_client.FederationClient object at 0x7f1651c18160>, 'hs1', '!jAEHKIubyIfuLOdfpY:hs1') ``` From this source: ```py async def backfill( self, dest: str, room_id: str, limit: int, extremities: Collection[str] ) -> Optional[List[EventBase]]: ``` And this usage: ```py events = await self._federation_client.backfill( dest, room_id, limit=limit, extremities=extremities ) ``` which would previously cause this error: ``` synapse_main | 2022-08-04 06:13:12,051 - synapse.handlers.federation - 424 - ERROR - GET-5 - Failed to backfill from hs1 because tuple index out of range synapse_main | Traceback (most recent call last): synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation.py", line 392, in try_backfill synapse_main | await self._federation_event_handler.backfill( synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper synapse_main | return await func(*args, **kwargs) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 593, in backfill synapse_main | events = await self._federation_client.backfill( synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper synapse_main | return await func(*args, **kwargs) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 827, in _wrapper synapse_main | with wrapping_logic(func, *args, **kwargs): synapse_main | File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__ synapse_main | return next(self.gen) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 922, in _wrapping_logic synapse_main | set_attribute("ARG_" + arg, str(args[i + 1])) # type: ignore[index] synapse_main | IndexError: tuple index out of range ```
* Additional fixes for opentracing type hints. (#13362)Patrick Cloke2022-07-251-2/+2
|
* Add missing types to opentracing. (#13345)Patrick Cloke2022-07-211-9/+35
| | | After this change `synapse.logging` is fully typed.
* Add type annotations to `trace` decorator. (#13328)Patrick Cloke2022-07-191-22/+28
| | | | Functions that are decorated with `trace` are now properly typed and the type hints for them are fixed.
* More type hints for `synapse.logging` (#13103)Patrick Cloke2022-06-302-42/+54
| | | | Completes type hints for synapse.logging.scopecontextmanager and (partially) for synapse.logging.opentracing.
* Easy type hints in synapse.logging.opentracing (#12894)David Robertson2022-05-271-49/+65
|
* Another batch of type annotations (#12726)David Robertson2022-05-134-21/+45
|
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-041-1/+1
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-271-9/+17
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Add opentracing spans to calls to external cache (#12380)Erik Johnston2022-04-071-0/+3
|
* Use `ParamSpec` in type hints for `synapse.logging.context` (#12150)Sean Quah2022-03-081-20/+24
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Remove support for the legacy structured logging configuration. (#12008)Patrick Cloke2022-02-171-163/+0
|
* Improve opentracing for federation requests (#11870)Richard van der Hoff2022-02-031-0/+12
| | | | | | | | | The idea here is to set the parent span for incoming federation requests to the *outgoing* span on the other end. That means that you can see (most of) the full end-to-end flow when you have a process that includes federation requests. However, in order not to lose information, we still want a link to the `incoming-federation-request` span from the servlet, so we have to create another span to do exactly that.
* Fixes for opentracing scopes (#11869)Richard van der Hoff2022-02-022-35/+70
| | | | | | | | | | | | | | | | | | | | `start_active_span` was inconsistent as to whether it would activate the span immediately, or wait for `scope.__enter__` to happen (it depended on whether the current logcontext already had an associated scope). The inconsistency was rather confusing if you were hoping to set up a couple of separate spans before activating either. Looking at the other implementations of opentracing `ScopeManager`s, the intention is that it *should* be activated immediately, as the name implies. Indeed, the idea is that you don't have to use the scope as a contextmanager at all - you can just call `.close` on the result. Hence, our cleanup has to happen in `.close` rather than `.__exit__`. So, the main change here is to ensure that `start_active_span` does activate the span, and that `scope.close()` does close the scope. We also add some tests, which requires a `tracer` param so that we don't have to rely on the global variable in unit tests.
* Add missing `auto_attribs=True` to the `_WrappedRustReporter` class (#11768)Andrew Morgan2022-01-191-1/+1
|
* Remove `log_function` and its uses (#11761)Richard van der Hoff2022-01-181-76/+0
| | | | | | | I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-133-16/+16
|
* Update to the current version of Black and run it on Synapse codebase (#11596)Shay2021-12-231-1/+0
| | | | | | | | | * update black version * run updated version of black on code * newsfragment * enumerate python versions
* Fix mypy error with opentracing.tags. (#11622)Patrick Cloke2021-12-231-0/+1
|
* Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff2021-12-201-46/+22
| | | | | | | | | | | | | | | | | | | | | | * remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
* Add opentracing types (#11603)Shay2021-12-202-10/+16
|
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-46/+103
|
* Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-101-2/+2
|
* Type hints for the remaining two files in `synapse.http`. (#11164)David Robertson2021-10-281-2/+2
| | | | | | | | | | | | | | | | | | | * Teach MyPy that the sentinel context is False This means that if `ctx: LoggingContextOrSentinel` then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really neat find! * Annotate RequestMetrics - Raise errors for sentry if we use the sentinel context - Ensure we don't raise an error and carry on, but not recording stats - Include stack trace in the error case to lower Sean's blood pressure * Make mypy pass for synapse.http.request_metrics * Make synapse.http.connectproxyclient pass mypy Co-authored-by: reivilibre <oliverw@matrix.org>
* Annotate `log_function` decorator (#10943)reivilibre2021-10-271-2/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix opentracing and Prometheus metrics for replication requests (#10996)Sean Quah2021-10-121-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | This commit fixes two bugs to do with decorators not instrumenting `ReplicationEndpoint`'s `send_request` correctly. There are two decorators on `send_request`: Prometheus' `Gauge.track_inprogress()` and Synapse's `opentracing.trace`. `Gauge.track_inprogress()` does not have any support for async functions when used as a decorator. Since async functions behave like regular functions that return coroutines, only the creation of the coroutine was covered by the metric and none of the actual body of `send_request`. `Gauge.track_inprogress()` returns a regular, non-async function wrapping `send_request`, which is the source of the next bug. The `opentracing.trace` decorator would normally handle async functions correctly, but since the wrapped `send_request` is a non-async function, the decorator ends up suffering from the same issue as `Gauge.track_inprogress()`: the opentracing span only measures the creation of the coroutine and none of the actual function body. Using `Gauge.track_inprogress()` as a context manager instead of a decorator resolves both bugs.
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-8/+8
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Include exception in json logging (#11028)Nick Barrett2021-10-081-0/+6
|
* Log stack traces when a missing opentracing span is detected (#10983)Sean Quah2021-10-051-0/+1
| | | | Make it easier to track down where opentracing spans are going missing by including stack traces in the logs.
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-3/+3
|
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-1/+1
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Use the BaseReporter super-class for _WrappedRustReporter. (#10799)Patrick Cloke2021-09-131-1/+11
| | | | This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op for versions before that.
* Fix PeriodicallyFlushingMemoryHandler inhibiting application shutdown (#10517)reivilibre2021-08-031-0/+1
|
* Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)reivilibre2021-07-271-0/+88
| | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Fix a handful of type annotations. (#10446)Richard van der Hoff2021-07-221-2/+2
| | | | | | | | | | * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-154-21/+21
|
* Expose opentracing trace id in response headers (#10199)Richard van der Hoff2021-06-181-0/+21
| | | Fixes: #9480
* Improve comments in the structured logging code. (#10188)Patrick Cloke2021-06-161-4/+5
|
* Make opentracing trace into event persistence (#10134)Richard van der Hoff2021-06-161-2/+55
| | | | | | | | | | | | | | * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans
* Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-83/+19
| | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* opentracing: use a consistent name for background processes (#10135)Richard van der Hoff2021-06-071-0/+1
| | | | ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.
* Enable Prometheus metrics for the jaeger client library (#10112)Richard van der Hoff2021-06-041-0/+3
|
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-031-0/+6
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Fix opentracing inject to use the SpanContext, not the Span (#10074)Brad Murray2021-06-011-5/+5
| | | Signed-off-by: Brad Murray brad@beeper.com
* Set opentracing priority before setting other tags (#10092)Richard van der Hoff2021-05-281-4/+17
| | | ... because tags on spans which aren't being sampled get thrown away.
* Add debug logging for issue #9533 (#9959)Richard van der Hoff2021-05-111-1/+6
| | | | | Hopefully this will help us track down where to-device messages are getting lost/delayed.
* Merge branch 'master' into developAndrew Morgan2021-04-221-3/+12
|\
| * Make LoggingContext's name optional (#9857)Richard van der Hoff2021-04-211-3/+12
| | | | | | | | Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
* | Merge branch 'master' into developAndrew Morgan2021-04-201-10/+4
|\|
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-10/+4
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-2/+2
| |
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-149-9/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-1/+2
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Record more information into structured logs. (#9654)Patrick Cloke2021-04-081-11/+59
| | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
* Improve tracing for to device messages (#9686)Erik Johnston2021-04-011-0/+8
|
* Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2021-03-291-2/+2
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-1/+1
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-1/+1
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-8/+15
|
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-2/+4
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-031-2/+4
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-165-13/+23
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add type hints to E2E handler. (#9232)Patrick Cloke2021-01-281-1/+1
| | | This finishes adding type hints to the `synapse.handlers` module.
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-18/+32
|
* Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-19/+5
|
* Gracefully handle a pending logging connection during shutdown. (#8685)Patrick Cloke2020-10-291-9/+18
|
* Support generating structured logs in addition to standard logs. (#8607)Patrick Cloke2020-10-295-453/+218
| | | | | | | This modifies the configuration of structured logging to be usable from the standard Python logging configuration. This also separates the formatting of logs from the transport allowing JSON logs to files or standard logs to sockets.
* Start fewer opentracing spans (#8640)Erik Johnston2020-10-261-5/+5
| | | | | | | #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.
* Separate the TCP and terse JSON formatting code. (#8587)Patrick Cloke2020-10-212-186/+238
| | | | This should (theoretically) allow for using the TCP code with a different output type and make it easier to use the JSON code with files / console.
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-9/+1
|
* Create a mechanism for marking tests "logcontext clean" (#8399)Richard van der Hoff2020-09-281-20/+23
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-4/+4
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-3/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix typos in comments.Patrick Cloke2020-09-141-1/+1
|
* Use the right constructor for log records (#8278)Richard van der Hoff2020-09-081-3/+3
| | | | | | Update `log_function` to use the right factory to create log records, to make sure that they have `request` attributes. Fixes: #8267.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-044-9/+9
|
* Use the JSON encoder without whitespace in more places. (#8124)Patrick Cloke2020-08-201-3/+2
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+5
|
* Fix error reporting when using `opentracing.trace` (#7961)Erik Johnston2020-07-272-12/+3
|
* Clean up PreserveLoggingContext (#7877)Richard van der Hoff2020-07-221-16/+13
| | | This had some dead code and some just plain wrong docstrings.
* Remove unused code from synapse.logging.utils. (#7897)Karthikeyan Singaravelan2020-07-201-126/+0
|
* Fix the trace function for async functions. (#7872)Patrick Cloke2020-07-171-23/+40
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* isort 5 compatibility (#7786)Will Hunt2020-07-051-2/+2
| | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-031-37/+31
|
* Add support for using rust-python-jaeger-reporter (#7697)Erik Johnston2020-06-171-3/+36
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-2/+1
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-8/+2
|
* Improve per-block CPU and DB usage metrics (#7426)Patrick Cloke2020-05-061-12/+26
|
* Add backwards compatibility codepath to LoggingContext. (#7408)Patrick Cloke2020-05-051-0/+41
|
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-291-13/+10
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Reduce the number of calls to `resource.getrusage` (#7183)Richard van der Hoff2020-04-031-39/+63
| | | Let's just call `getrusage` once on each logcontext change, rather than twice.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-0/+5
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-243-126/+125
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Hopefully mypy is happy nowBrendan Abolivier2020-03-101-1/+2
|
* Attempt at appeasing the gods of mypyBrendan Abolivier2020-03-101-1/+1
|
* Add type hints to logging/context.py (#6309)Neil Pilgrim2020-03-071-47/+74
| | | | | | * Add type hints to logging/context.py Signed-off-by: neiljp (Neil Pilgrim) <github@kepier.clara.net>
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* comment for run_in_backgroundRichard van der Hoff2020-01-121-0/+3
|
* Bump version of mypyErik Johnston2019-12-122-1/+4
|
* Update commentErik Johnston2019-12-101-2/+4
|
* Fix make_deferred_yieldable to work with coroutinesErik Johnston2019-12-101-1/+8
|
* Add benchmarks for structured logging performance (#6266)Amber Brown2019-12-031-0/+1
|
* Improve the performance of structured logging (#6322)Amber Brown2019-11-262-30/+90
|
* A couple more instancesAndrew Morgan2019-11-121-1/+1
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-1/+1
| | | * update version of black and also fix the mypy config being overridden
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-1/+1
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-4/+10
|
* Fix up some typechecking (#6150)Amber Brown2019-10-022-8/+21
| | | | | | * type checking fixes * changelog
* Fix logcontext spam on non-Linux platforms (#6059)J. Ryan Stinnett2019-09-181-2/+11
| | | | | | | This checks whether the current platform supports thread resource usage tracking before logging a warning to avoid log spam. Fixes https://github.com/matrix-org/synapse/issues/6055
* Fix for structured logging tests stomping on logs (#6023)Amber Brown2019-09-133-9/+11
|
* Don't assume there is a 'self' arg in @trace decoratorErik Johnston2019-09-061-5/+5
|
* Gracefully handle log context slips and missing opentracing import errors. ↵Jorik Schellekens2019-09-051-24/+58
| | | | (#5988)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-051-1/+1
|
* Fix opentracing contexts missing from outbound replication requests (#5982)Jorik Schellekens2019-09-051-4/+16
|
* Opentrace device lists (#5853)Jorik Schellekens2019-09-031-53/+17
| | | Trace device list changes.
* Implement a structured logging output system. (#5680)Amber Brown2019-08-283-1/+665
|
* Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-221-0/+26
| | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Opentracing across workers (#5771)Jorik Schellekens2019-08-221-65/+79
| | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* Opentracing doc update (#5776)Jorik Schellekens2019-08-201-27/+40
| | | Update opentracing docs to use the unified 'trace' method
* Make Jaeger fully configurable (#5694)Jorik Schellekens2019-07-231-4/+7
| | | | | | * Allow Jaeger to be configured * Update sample config
* Replace returnValue with return (#5736)Amber Brown2019-07-231-3/+3
|
* Opentracing Utils (#5722)Jorik Schellekens2019-07-232-101/+356
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add decerators for tracing functions * Use the new clean contexts * Context and edu utils * Move opentracing setters * Move whitelisting * Sectioning comments * Better args wrapper * Docstrings Co-Authored-By: Erik Johnston <erik@matrix.org> * Remove unused methods. * Don't use global * One tracing decorator to rule them all.
* Merge branch 'release-v1.2.0' into developJorik Schellekens2019-07-221-0/+125
|\
| * Opentracing Documentation (#5703)Jorik Schellekens2019-07-221-0/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Opentracing survival guide * Update decorator names in doc * Doc cleanup These are all alterations as a result of comments in #5703, it includes mostly typos and clarifications. The most interesting changes are: - Split developer and user docs into two sections - Add a high level description of OpenTracing * newsfile * Move contributer specific info to docstring. * Sample config. * Trailing whitespace. * Update 5703.misc * Apply suggestions from code review Mostly just rewording parts of the docs for clarity. Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Don't accept opentracing data from clients. (#5715)Jorik Schellekens2019-07-191-2/+1
|/ | | | | | * Don't accept opentracing data from clients. * newsfile
* Clean up opentracing configuration options (#5712)Richard van der Hoff2019-07-182-26/+20
| | | | | | | | | | | | | | | | | Clean up config settings and dead code. This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular: * There should be a blank line after `## Section ##' headings * There should be a blank line between each config setting * There should be a `#`-only line between a comment and the setting it describes * We don't really do the `# #` style commenting-out of whole sections if we can help it * rename `tracer_enabled` to `enabled` While we're here, do more config parsing upfront, which makes it easier to use later on. Also removes redundant code from LogContextScopeManager. Also changes the changelog fragment to a `feature` - it's exciting!
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-113-2/+508
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-044-0/+940