summary refs log tree commit diff
path: root/synapse/logging/opentracing.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-161-2/+2
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Update mypy and mypy-zope, attempt 3 (#13993)David Robertson2022-09-301-2/+2
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Revert "Update mypy and mypy-zope (#13925)"David Robertson2022-09-301-2/+2
| | | | This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
* Update mypy and mypy-zope (#13925)David Robertson2022-09-301-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Tag trace with instance name (#13761)Eric Eastwood2022-09-091-2/+4
| | | | | | | | We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load. As suggested by @clokep and @reivilibre in, - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578
* Strip number suffix from instance name to consolidate services that traces ↵Eric Eastwood2022-09-091-1/+12
| | | | | | | | | | | | are spread over (#13729) The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985 Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through. With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇 Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.
* Instrument the federation/backfill part of `/messages` (#13489)Eric Eastwood2022-08-161-3/+16
| | | | | | | | | Instrument the federation/backfill part of `/messages` so it's easier to follow what's going on in Jaeger when viewing a trace. Split out from https://github.com/matrix-org/synapse/pull/13440 Follow-up from https://github.com/matrix-org/synapse/pull/13368 Part of https://github.com/matrix-org/synapse/issues/13356
* Revert "Update locked versions of mypy and mypy-zope (#13521)"David Robertson2022-08-151-2/+2
| | | | | | | | This reverts commit f383b9b3eceaa082d5ae690550fe41460b711779. Other PRs were seeing mypy failures that looked to be related to mypy-zope. Confusingly, we didn't see this on #13521. Revert this for now and investigate later.
* Update locked versions of mypy and mypy-zope (#13521)David Robertson2022-08-151-2/+2
|
* Allow use of both `@trace` and `@tag_args` stacked on the same function (#13453)Eric Eastwood2022-08-091-56/+102
| | | | | | | | | | | | | ```py @trace @tag_args async def get_oldest_event_ids_with_depth_in_room(...) ... ``` Before this PR, you would see a warning in the logs and the span was not exported: ``` 2022-08-03 19:11:59,383 - synapse.logging.opentracing - 835 - ERROR - GET-0 - @trace may not have wrapped EventFederationWorkerStore.get_oldest_event_ids_with_depth_in_room correctly! The function is not async but returned a coroutine. ```
* Fix `@tag_args` being off-by-one (ahead) (#13452)Eric Eastwood2022-08-041-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix @tag_args being off-by-one (ahead) Example: ``` argspec.args=[ 'self', 'room_id' ] args=( <synapse.storage.databases.main.DataStore object at 0x10d0b8d00>, '!HBehERstyQBxyJDLfR:my.synapse.server' ) ``` --- The previous logic was also flawed and we can end up in a situation like this: ``` argspec.args=['self', 'dest', 'room_id', 'limit', 'extremities'] args=(<synapse.federation.federation_client.FederationClient object at 0x7f1651c18160>, 'hs1', '!jAEHKIubyIfuLOdfpY:hs1') ``` From this source: ```py async def backfill( self, dest: str, room_id: str, limit: int, extremities: Collection[str] ) -> Optional[List[EventBase]]: ``` And this usage: ```py events = await self._federation_client.backfill( dest, room_id, limit=limit, extremities=extremities ) ``` which would previously cause this error: ``` synapse_main | 2022-08-04 06:13:12,051 - synapse.handlers.federation - 424 - ERROR - GET-5 - Failed to backfill from hs1 because tuple index out of range synapse_main | Traceback (most recent call last): synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation.py", line 392, in try_backfill synapse_main | await self._federation_event_handler.backfill( synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper synapse_main | return await func(*args, **kwargs) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 593, in backfill synapse_main | events = await self._federation_client.backfill( synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 828, in _wrapper synapse_main | return await func(*args, **kwargs) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 827, in _wrapper synapse_main | with wrapping_logic(func, *args, **kwargs): synapse_main | File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__ synapse_main | return next(self.gen) synapse_main | File "/usr/local/lib/python3.9/site-packages/synapse/logging/tracing.py", line 922, in _wrapping_logic synapse_main | set_attribute("ARG_" + arg, str(args[i + 1])) # type: ignore[index] synapse_main | IndexError: tuple index out of range ```
* Additional fixes for opentracing type hints. (#13362)Patrick Cloke2022-07-251-2/+2
|
* Add missing types to opentracing. (#13345)Patrick Cloke2022-07-211-9/+35
| | | After this change `synapse.logging` is fully typed.
* Add type annotations to `trace` decorator. (#13328)Patrick Cloke2022-07-191-22/+28
| | | | Functions that are decorated with `trace` are now properly typed and the type hints for them are fixed.
* More type hints for `synapse.logging` (#13103)Patrick Cloke2022-06-301-26/+35
| | | | Completes type hints for synapse.logging.scopecontextmanager and (partially) for synapse.logging.opentracing.
* Easy type hints in synapse.logging.opentracing (#12894)David Robertson2022-05-271-49/+65
|
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-041-1/+1
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Add opentracing spans to calls to external cache (#12380)Erik Johnston2022-04-071-0/+3
|
* Improve opentracing for federation requests (#11870)Richard van der Hoff2022-02-031-0/+12
| | | | | | | | | The idea here is to set the parent span for incoming federation requests to the *outgoing* span on the other end. That means that you can see (most of) the full end-to-end flow when you have a process that includes federation requests. However, in order not to lose information, we still want a link to the `incoming-federation-request` span from the servlet, so we have to create another span to do exactly that.
* Fixes for opentracing scopes (#11869)Richard van der Hoff2022-02-021-6/+23
| | | | | | | | | | | | | | | | | | | | `start_active_span` was inconsistent as to whether it would activate the span immediately, or wait for `scope.__enter__` to happen (it depended on whether the current logcontext already had an associated scope). The inconsistency was rather confusing if you were hoping to set up a couple of separate spans before activating either. Looking at the other implementations of opentracing `ScopeManager`s, the intention is that it *should* be activated immediately, as the name implies. Indeed, the idea is that you don't have to use the scope as a contextmanager at all - you can just call `.close` on the result. Hence, our cleanup has to happen in `.close` rather than `.__exit__`. So, the main change here is to ensure that `start_active_span` does activate the span, and that `scope.close()` does close the scope. We also add some tests, which requires a `tracer` param so that we don't have to rely on the global variable in unit tests.
* Add missing `auto_attribs=True` to the `_WrappedRustReporter` class (#11768)Andrew Morgan2022-01-191-1/+1
|
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-1/+1
|
* Fix mypy error with opentracing.tags. (#11622)Patrick Cloke2021-12-231-0/+1
|
* Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff2021-12-201-46/+22
| | | | | | | | | | | | | | | | | | | | | | * remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
* Add opentracing types (#11603)Shay2021-12-201-9/+15
|
* Fix opentracing and Prometheus metrics for replication requests (#10996)Sean Quah2021-10-121-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | This commit fixes two bugs to do with decorators not instrumenting `ReplicationEndpoint`'s `send_request` correctly. There are two decorators on `send_request`: Prometheus' `Gauge.track_inprogress()` and Synapse's `opentracing.trace`. `Gauge.track_inprogress()` does not have any support for async functions when used as a decorator. Since async functions behave like regular functions that return coroutines, only the creation of the coroutine was covered by the metric and none of the actual body of `send_request`. `Gauge.track_inprogress()` returns a regular, non-async function wrapping `send_request`, which is the source of the next bug. The `opentracing.trace` decorator would normally handle async functions correctly, but since the wrapped `send_request` is a non-async function, the decorator ends up suffering from the same issue as `Gauge.track_inprogress()`: the opentracing span only measures the creation of the coroutine and none of the actual function body. Using `Gauge.track_inprogress()` as a context manager instead of a decorator resolves both bugs.
* Log stack traces when a missing opentracing span is detected (#10983)Sean Quah2021-10-051-0/+1
| | | | Make it easier to track down where opentracing spans are going missing by including stack traces in the logs.
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-3/+3
|
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-1/+1
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Use the BaseReporter super-class for _WrappedRustReporter. (#10799)Patrick Cloke2021-09-131-1/+11
| | | | This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op for versions before that.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-5/+5
|
* Expose opentracing trace id in response headers (#10199)Richard van der Hoff2021-06-181-0/+21
| | | Fixes: #9480
* Make opentracing trace into event persistence (#10134)Richard van der Hoff2021-06-161-2/+55
| | | | | | | | | | | | | | * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans
* Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-83/+19
| | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* opentracing: use a consistent name for background processes (#10135)Richard van der Hoff2021-06-071-0/+1
| | | | ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.
* Enable Prometheus metrics for the jaeger client library (#10112)Richard van der Hoff2021-06-041-0/+3
|
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-031-0/+6
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Fix opentracing inject to use the SpanContext, not the Span (#10074)Brad Murray2021-06-011-5/+5
| | | Signed-off-by: Brad Murray brad@beeper.com
* Set opentracing priority before setting other tags (#10092)Richard van der Hoff2021-05-281-4/+17
| | | ... because tags on spans which aren't being sampled get thrown away.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-1/+2
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Improve tracing for to device messages (#9686)Erik Johnston2021-04-011-0/+8
|
* Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2021-03-291-2/+2
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-1/+1
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-5/+3
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add type hints to E2E handler. (#9232)Patrick Cloke2021-01-281-1/+1
| | | This finishes adding type hints to the `synapse.handlers` module.
* Start fewer opentracing spans (#8640)Erik Johnston2020-10-261-5/+5
| | | | | | | #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.
* Fix typos in comments.Patrick Cloke2020-09-141-1/+1
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-1/+1
|
* Use the JSON encoder without whitespace in more places. (#8124)Patrick Cloke2020-08-201-3/+2
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+5
|
* Fix error reporting when using `opentracing.trace` (#7961)Erik Johnston2020-07-271-12/+1
|
* Fix the trace function for async functions. (#7872)Patrick Cloke2020-07-171-23/+40
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* isort 5 compatibility (#7786)Will Hunt2020-07-051-2/+2
| | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-031-37/+31
|
* Add support for using rust-python-jaeger-reporter (#7697)Erik Johnston2020-06-171-3/+36
|
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-291-13/+10
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-4/+10
|
* Fix up some typechecking (#6150)Amber Brown2019-10-021-4/+5
| | | | | | * type checking fixes * changelog
* Fix for structured logging tests stomping on logs (#6023)Amber Brown2019-09-131-2/+2
|
* Don't assume there is a 'self' arg in @trace decoratorErik Johnston2019-09-061-5/+5
|
* Gracefully handle log context slips and missing opentracing import errors. ↵Jorik Schellekens2019-09-051-24/+58
| | | | (#5988)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-051-1/+1
|
* Fix opentracing contexts missing from outbound replication requests (#5982)Jorik Schellekens2019-09-051-4/+16
|
* Opentrace device lists (#5853)Jorik Schellekens2019-09-031-53/+17
| | | Trace device list changes.
* Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-221-0/+26
| | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Opentracing across workers (#5771)Jorik Schellekens2019-08-221-65/+79
| | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* Opentracing doc update (#5776)Jorik Schellekens2019-08-201-27/+40
| | | Update opentracing docs to use the unified 'trace' method
* Make Jaeger fully configurable (#5694)Jorik Schellekens2019-07-231-4/+7
| | | | | | * Allow Jaeger to be configured * Update sample config
* Replace returnValue with return (#5736)Amber Brown2019-07-231-3/+3
|
* Opentracing Utils (#5722)Jorik Schellekens2019-07-231-100/+355
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add decerators for tracing functions * Use the new clean contexts * Context and edu utils * Move opentracing setters * Move whitelisting * Sectioning comments * Better args wrapper * Docstrings Co-Authored-By: Erik Johnston <erik@matrix.org> * Remove unused methods. * Don't use global * One tracing decorator to rule them all.
* Merge branch 'release-v1.2.0' into developJorik Schellekens2019-07-221-0/+125
|\
| * Opentracing Documentation (#5703)Jorik Schellekens2019-07-221-0/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Opentracing survival guide * Update decorator names in doc * Doc cleanup These are all alterations as a result of comments in #5703, it includes mostly typos and clarifications. The most interesting changes are: - Split developer and user docs into two sections - Add a high level description of OpenTracing * newsfile * Move contributer specific info to docstring. * Sample config. * Trailing whitespace. * Update 5703.misc * Apply suggestions from code review Mostly just rewording parts of the docs for clarity. Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Don't accept opentracing data from clients. (#5715)Jorik Schellekens2019-07-191-2/+1
|/ | | | | | * Don't accept opentracing data from clients. * newsfile
* Clean up opentracing configuration options (#5712)Richard van der Hoff2019-07-181-23/+19
| | | | | | | | | | | | | | | | | Clean up config settings and dead code. This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular: * There should be a blank line after `## Section ##' headings * There should be a blank line between each config setting * There should be a `#`-only line between a comment and the setting it describes * We don't really do the `# #` style commenting-out of whole sections if we can help it * rename `tracer_enabled` to `enabled` While we're here, do more config parsing upfront, which makes it easier to use later on. Also removes redundant code from LogContextScopeManager. Also changes the changelog fragment to a `feature` - it's exciting!
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-111-0/+362
* Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-29 00:34:29 +0000