| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
| |
No point recreating these for each call, since they are frozen
|
| | |
|
| | |
|
| |
|
|
|
|
| |
* Include 'prev_content' field in AS events
Signed-off-by: Vaishnav Nair <nairvaishnav007@icloud.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
| |
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.
Co-authored-by: David Robertson <davidr@element.io>
|
| |
|
|
|
|
|
|
|
|
|
| |
This is some odds and ends found during the review of #11791
and while continuing to work in this code:
* Return attrs classes instead of dictionaries from some methods
to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
to not include the bundle at all, as the docstring states).
|
| |
|
|
|
|
|
|
|
| |
* Don't print HTTPStatus.* in "Processed..." logs
Fixes #11812. See also #7118 and
https://github.com/matrix-org/synapse/pull/7188#r401719326 in
particular.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| | |
|
| |
|
|
|
|
|
| |
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).
This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.
More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Similar to #11817.
In `_create_power_level_validator` we
- retrieve `validator`. This is a class implementing the
`jsonschema.protocols.Validator` interface. In other words,
`validator: Type[jsonschema.protocols.Validator]`.
- we then create an second validator class by modifying the original
`validator`. We return that class, which is also of type
`Type[jsonschema.protocols.Validator]`.
So the original annotation was incorrect: it claimed we were returning
an instance of jsonSchema.Draft7Validator, not the class (or a subclass)
itself. (Strictly speaking this is incorrect, because `POWER_LEVELS_SCHEMA`
isn't pinned to a particular version of JSON Schema. But there are other
complications with the type stubs if you try to fix this; I felt like
the change herein was a decent compromise that better expresses intent).
(I suspect/hope the typeshed project would welcome an effort to improve
the jsonschema stubs. Let's see if I get some spare time.)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#11816)
* add check that gc.freeze is available before calling
* newsfragment
* lint
* Update comment
Co-authored-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* CI: run Complement on the VM, not inside Docker
This requires https://github.com/matrix-org/complement/pull/289
We now run Complement on the VM instead of inside a Docker container.
This is to allow Complement to bind to any high-numbered port when it
starts up its own federation servers. We want to do this to allow for
more concurrency when running complement tests. Previously, Complement
only ever bound to `:8448` when running its own federation server. This
prevented multiple federation tests running at the same time as they would
fight each other on the port. This did however allow Complement to run
in Docker, as the host could just port forward `:8448` to allow homeserver
containers to communicate to Complement. Now that we are using random
ports however, we cannot use Docker to run Complement. This ends up
being a good thing because:
- Running Complement tests locally is closer to how they run in CI.
- Allows the `CI` env var to be removed in Complement.
- Slightly speeds up runs as we don't need to pull down the Complement
image prior to running tests. This assumes GHA caches actions sensibly.
* Changelog
* Full stop
* Update .github/workflows/tests.yml
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Review comments
* Update .github/workflows/tests.yml
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
|
| |
|
| |
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
|
|
|
| |
* Make functions in python deltas optional
It's annoying to always have to write stubs for these.
* Documentation for delta files
* changelog
|
| |
|
|
|
|
|
| |
* Docs: add missing PR submission process how-tos
The documentation says that in order to submit a pull request you have to run the linter and links to [Run the linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters). IMO "Run the linters" should explain that development dependencies are a pre-requisite.
I also included `pip install wheel` which I had to run inside my virtual environment on ubuntu before I `pip install -e ".[all,dev]"` would succeed.
|
| |
|
| |
* Fix get federation status of destination if no error occured
|
| | |
|
| |
|
|
|
| |
PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our
arm64 docker images now build in a sensible amount of time, so we can skip the
amd64-only build.
|
| |
|
|
| |
This prevents the URL preview code from reading
a stream forever.
|
| |
|
|
|
| |
Images which are data URLs will no longer break URL
previews and will properly be "downloaded" and
thumbnailed.
|
| |
|
|
|
| |
upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* remove reference in comments to python3.6
* upgrade tox python env in script
* bump python version in example for completeness
* upgrade python version requirement in setup doc
* upgrade necessary python version in __init__.py
* upgrade python version in setup.py
* newsfragment
* drops refs to bionic and replace with focal
* bump refs to postgres 9.6 to 10
* fix hanging ci
* try installing tzdata first
* revert change made in b979f336
* ignore new random mypy error while debugging other error
* fix lint error for temporary workaround
* revert change to install list
* try passing env var
* export debian frontend var?
* move line and add comment
* bump pillow dependency
* bump lxml depenency
* install libjpeg-dev for pillow
* bump automat version to one compatible with py3.8
* add libwebp for pillow
* bump twisted trunk python version
* change suffix of newsfragment
* remove redundant python 3.7 checks
* lint
|
| |
|
|
| |
The PR was cherrypicked into v1.51.0rc1.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.51.0rc1 (2022-01-21)
==============================
Features
--------
- Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](https://github.com/matrix-org/synapse/issues/11561), [\#11749](https://github.com/matrix-org/synapse/issues/11749), [\#11757](https://github.com/matrix-org/synapse/issues/11757))
- Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11577](https://github.com/matrix-org/synapse/issues/11577))
- Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](https://github.com/matrix-org/synapse/issues/11672))
- Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](https://github.com/matrix-org/synapse/issues/11675), [\#11770](https://github.com/matrix-org/synapse/issues/11770))
Bugfixes
--------
- Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events
received over federation. ([\#11530](https://github.com/matrix-org/synapse/issues/11530))
- Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](https://github.com/matrix-org/synapse/issues/11587))
- Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](https://github.com/matrix-org/synapse/issues/11593))
- Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612), [\#11659](https://github.com/matrix-org/synapse/issues/11659), [\#11791](https://github.com/matrix-org/synapse/issues/11791))
- Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](https://github.com/matrix-org/synapse/issues/11667))
- Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](https://github.com/matrix-org/synapse/issues/11669))
- Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](https://github.com/matrix-org/synapse/issues/11695))
- Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](https://github.com/matrix-org/synapse/issues/11710), [\#11745](https://github.com/matrix-org/synapse/issues/11745))
- Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](https://github.com/matrix-org/synapse/issues/11737))
- Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](https://github.com/matrix-org/synapse/issues/11775))
- Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](https://github.com/matrix-org/synapse/issues/11786))
Improved Documentation
----------------------
- Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](https://github.com/matrix-org/synapse/issues/11686))
- Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](https://github.com/matrix-org/synapse/issues/11715))
- Document that the minimum supported PostgreSQL version is now 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725))
- Fix typo in demo docs: differnt. ([\#11735](https://github.com/matrix-org/synapse/issues/11735))
- Update room spec URL in config files. ([\#11739](https://github.com/matrix-org/synapse/issues/11739))
- Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](https://github.com/matrix-org/synapse/issues/11740))
- Update documentation for configuring login with Facebook. ([\#11755](https://github.com/matrix-org/synapse/issues/11755))
- Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](https://github.com/matrix-org/synapse/issues/11781))
Deprecations and Removals
-------------------------
- Remove the unstable `/send_relation` endpoint. ([\#11682](https://github.com/matrix-org/synapse/issues/11682))
- Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724))
- Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](https://github.com/matrix-org/synapse/issues/11576))
- Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](https://github.com/matrix-org/synapse/issues/11774), [\#11783](https://github.com/matrix-org/synapse/issues/11783))
Internal Changes
----------------
- Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](https://github.com/matrix-org/synapse/issues/11685))
- Use buildkit's cache feature to speed up docker builds. ([\#11691](https://github.com/matrix-org/synapse/issues/11691))
- Use `auto_attribs` and native type hints for attrs classes. ([\#11692](https://github.com/matrix-org/synapse/issues/11692), [\#11768](https://github.com/matrix-org/synapse/issues/11768))
- Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](https://github.com/matrix-org/synapse/issues/11693))
- Remove fallback code for Python 2. ([\#11699](https://github.com/matrix-org/synapse/issues/11699))
- Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic. ([\#11701](https://github.com/matrix-org/synapse/issues/11701))
- Add the option to write SQLite test dbs to disk when running tests. ([\#11702](https://github.com/matrix-org/synapse/issues/11702))
- Improve Complement test output for Gitub Actions. ([\#11707](https://github.com/matrix-org/synapse/issues/11707))
- Fix docstring on `add_account_data_for_user`. ([\#11716](https://github.com/matrix-org/synapse/issues/11716))
- Complement environment variable name change and update `.gitignore`. ([\#11718](https://github.com/matrix-org/synapse/issues/11718))
- Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](https://github.com/matrix-org/synapse/issues/11723))
- Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724), [\#11771](https://github.com/matrix-org/synapse/issues/11771))
- Minor efficiency improvements when inserting many values into the database. ([\#11742](https://github.com/matrix-org/synapse/issues/11742))
- Invite PR authors to give themselves credit in the changelog. ([\#11744](https://github.com/matrix-org/synapse/issues/11744))
- Add optional debugging to investigate [issue 8631](https://github.com/matrix-org/synapse/issues/8631). ([\#11760](https://github.com/matrix-org/synapse/issues/11760))
- Remove `log_function` utility function and its uses. ([\#11761](https://github.com/matrix-org/synapse/issues/11761))
- Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](https://github.com/matrix-org/synapse/issues/11765))
- Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](https://github.com/matrix-org/synapse/issues/11766))
- Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](https://github.com/matrix-org/synapse/issues/11776))
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
... and start populating them for new events
|
| | | |
|
| | |
| |
| | |
This is a follow-up to #10565.
|
| | |
| |
| |
| | |
Preparation for dropping this table altogether. Part of #6574.
|
| | |
| |
| |
| | |
This is more efficient, since we only have to look up `state_key` in the event
dict once, rather than three (!) times.
|
| | |
| |
| |
| | |
whose key is prefixed with the user ID (#11788)
|
| |/ |
|
| | |
|
| |
|
|
|
| |
To not change the behaviour during the deprecation period.
Follow-up to #11774.
|
| |
|
|
| |
simultaneously (#11765)
|
| |
|
|
| |
(#11770)
|
| |
|
| |
#11595 dropped support for python 3.6, but forgot to update this doc.
|
| |
|
|
|
|
|
| |
`web_client_location`. (#11774)
This changes the behaviour of the root endpoint to redirect
directly to the configuration of `web_client_location` if it is
given an HTTP(S) URL.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Debug for #8631.
I'm having a hard time tracking down what's going wrong in that issue.
In the reported example, I could see server A sending federation traffic
to server B and all was well. Yet B reports out-of-sync device updates
from A.
I couldn't see what was _in_ the events being sent from A to B. So I
have added some crude logging to track
- when we have updates to send to a remote HS
- the edus we actually accumulate to send
- when a federation transaction includes a device list update edu
- when such an EDU is received
This is a bit of a sledgehammer.
|
| |
|
|
|
| |
Updates complement.sh to read the ref from an environment
variable (defaulting to master) when downloading a complement
bundle for testing.
|
| |
|
|
| |
`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.
|
| |
|
|
| |
A couple of surprises for me here, so thought I'd document them
|
| | |
|
| | |
|
| |
|
|
|
|
| |
By scraping Open Graph information from the HTML even
when an autodiscovery endpoint is found. The results are
then combined to capture as much information as possible
from the page.
|
| |
|
|
|
|
| |
Per updates to MSC3440.
This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
|
| |
|
|
|
|
|
| |
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.
Rather, it tends to clutter up debug logs with useless information.
|
| |
|
|
| |
filter appservice users. (#11675)
|
| |
|
|
|
|
|
|
|
| |
* Warn against using Let's Encrypt certs for encrypted TURN
This helps to avoid client-side issues:
* https://github.com/vector-im/element-android/issues/1533
* https://github.com/vector-im/element-ios/issues/2712
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
|
| |
|
| |
Fixes #11741
|
| |
|
| |
... and a minor thinko fix in the sample config.
|
| |
|
|
|
|
|
|
|
|
|
| |
The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only*
measures the time taken to execute timed calls and callbacks from threads. That neglects everything that
happens off the back of I/O, which is obviously quite a lot for us.
To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is
the only place it should wait for something to happen - the rest of the loop *should* be quick.
I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and
it's a nuisance to populate.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)
* Document different order direction of numerical fields
"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.
Signed-off-by: Daniël Sonck <daniel@sonck.nl>
|
| |
|
|
| |
dictionary (#11576)
|
| |
|
|
|
|
|
| |
* Fix sample_config.yaml in regards track_puppeted_user_ips
Closes #11741
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |
|
|
| |
This should be (slightly) more efficient and it is simpler
to have a single method for inserting multiple values.
|
| |
|
|
|
| |
documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500
Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
|
| |
|
| |
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
|
| |
* change spec url in config files
* Create 11739.txt
* .txt -> .doc
|
| | |
|
| | |
|
| |
|
|
|
| |
Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection.
Part 1 of a quest to simplify the reactor monkey-patching.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently when puppeting another user, the user doing the puppeting is
tracked for client IPs and MAU (if configured).
When tracking MAU is important, it becomes necessary to be possible to
also track the client IPs and MAU of puppeted users. As an example a
client that manages user creation and creation of tokens via the Synapse
admin API, passing those tokens for the client to use.
This PR adds optional configuration to enable tracking of puppeted users
into monthly active users. The default behaviour stays the same.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |
|
| |
Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
|
| |
|
| |
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Deal with mypy errors w/ type-hinted pynacl 1.5.0
Fixes #11644.
I really don't like that we're monkey patching pynacl SignedKey
instances with alg and version objects. But I'm too scared to make the
changes necessary right now.
(Ideally I would replace `signedjson.types.SingingKey` with a runtime class which
wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
|
| |
|
|
|
| |
* Prettier complement logs
* Changelog
|
| |
|
|
|
|
|
| |
By returning all of the m.space.child state of the space, not just
the first 50. The number of rooms returned is still capped at 50.
For the federation API this implies that the requesting server will
need to individually query for any other rooms it is not joined to.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Optionally use an on-disk sqlite db in tests
When debugging a test it is sometimes useful to inspect the state of the
DB. This is not easy when the db is in-memory: one cannot attach the
sqlite CLI to another process's DB.
With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use
`_trial_temp/test.db` as our sqlite database. One can then use
`sqlite3 _trial_temp/test.db` and query to your heart's content.
The DB is destroyed and recreated between different test cases.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
| |
(#11701)
|
| |\ |
|
| | | |
|
| | |
| |
| |
| | |
`glob_to_regex` in `synapse.util`, where it moved from. (#11696)
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.
Overall this should cause no change in behavior, but is prep work
for other improvements.
|
| | |
| |
| |
| | |
This was removed from MSC2674 before that was approved
and is not used by any known clients.
|
| | |
| |
| |
| | |
On Python 2, indexing a byte-string gives back a byte-string,
while on Python 3 it gives back the ASCII equivalent as an int.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
federation (#11530)
* add some tests to verify we are stripping unauthorized fields out of unsigned
* add function to strip unauthorized fields from the unsigned object of event
* newsfragment
* update newsfragment number
* add check to on_send_membership_event
* refactor tests
* fix lint error
* slightly refactor tests and add some comments
* slight refactor
* refactor tests
* fix import error
* slight refactor
* remove unsigned filtration code from synapse/handlers/federation_event.py
* lint
* move unsigned filtering code to event base
* refactor tests
* update newsfragment
* requested changes
* remove unused retun values
|
| | |
| |
| |
| |
| | |
error to be returned, not `M_UNKNOWN` (#11672)
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
| |
| |
| | |
as per
https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
|
| | |
| |
| |
| |
| |
| |
| | |
Fixes minor discrepancies between the /hierarchy endpoint described
in MSC2946 and the implementation.
Note that the changes impact the stable and unstable /hierarchy and
unstable /spaces endpoints for both client and federation APIs.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
* newsfragment
* fix newsfragment number
* update changelog
* remove extra space
|
| |/
|
| |
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* `_auth_and_persist_outliers`: mark persisted events as outliers
Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.
Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.
* `process_remote_join`: stop flagging as outlier
The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.
* `send_join`: remove `outlier=True`
The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.
* `get_event_auth`: remove `outlier=True`
stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.
* `_get_remote_auth_chain_for_event`: remove `outlier=True`
we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.
* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter
This param is now never set to True, so we can remove it.
* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param
This is no longer set anywhere, so we can remove it.
* `get_pdu`: remove unused `outlier` parameter
... and chase it down into `get_pdu_from_destination_raw`.
* `event_from_pdu_json`: remove redundant `outlier` param
This is never set to `True`, so can be removed.
* changelog
* update docstring
|
| |
|
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
|
| |
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
|
| |
|
|
|
| |
#11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
|
| |
|
|
| |
before the names of methods, as appropriate. (#11681)
|
| | |
|
| |
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
|
| |
|
| |
this should not be a case-insensitive match.
|
| |
|
|
|
|
|
|
|
|
|
| |
* Fix AssertionErrors after purging events
If you purged a bunch of events from your database, and then restarted synapse
without receiving more events, then you would get a bunch of AssertionErrors on
restart.
This fixes the situation by rewinding the stream processors.
* `check-newsfragment`: ignore deleted newsfiles
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Events returned by `backfill` should not be flagged as outliers.
Fixes:
```
AssertionError: null
File "synapse/handlers/federation.py", line 313, in try_backfill
dom, room_id, limit=100, extremities=extremities
File "synapse/handlers/federation_event.py", line 517, in backfill
await self._process_pulled_events(dest, events, backfilled=True)
File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
await self._process_pulled_event(origin, ev, backfilled=backfilled)
File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
assert not event.internal_metadata.is_outlier()
```
See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992
Fixes #8894.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Push `get_room_{min,max_stream_ordering}` into StreamStore
Both implementations of this are identical, so we may as well push it down and
get rid of the abstract base class nonsense.
* Remove redundant `StreamStore` class
This is empty now
* Remove redundant `get_current_events_token`
This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid
of it.
* newsfile
|
| |
|
|
| |
"Unknown room" can mean a multitude of things here. To help with debugging, add
some more words to the exception text.
|
| |
|
|
| |
Invites and knocks will now include the topic in the stripped state
send to clients before joining the room.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* update Trove classifiers to remove py36
* stop building bionic
* update dh-virtualenv
* newsfragment
* fix newsfragment
* update version refs
* another try at correct tag
* Update changelog
|
| |
|
| |
To improve type hints throughout the code.
|
| | |
|
| |
|
|
|
| |
exclusion list. (#11657)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
| |
By using cast and making ignores more specific.
|
| |
|
|
| |
Both of those APIs return state events, which will not have bundled
aggregations added anyway.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* update black version
* run updated version of black on code
* newsfragment
* enumerate python versions
|
| | |
|
| |
|
|
| |
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
|
| |
|
|
|
|
|
|
|
|
|
| |
* remove python 3.6 and postgres 9.6 from github workflow
* remove python 3.6 env from tox
* newsfragment
* correct postgres version
* add py310 to tox env list
|
| |\ |
|
| | |
| |
| |
| | |
`tests.server.setup_test_homeserver`. (#11503)
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Wrap `auth.get_user_by_req` in an opentracing span
give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.
This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.
* Emit opentracing span for encoding json responses
This can be a significant time sink.
* Rename all sync spans with a prefix
* Write an opentracing span for encoding sync response
* opentracing span to group generate_room_entries
* opentracing spans within sync.encode_response
* changelog
* Use the `trace` decorator instead of context managers
|
| | |
| |
| |
| |
| | |
And make bundling aggregations opt-in, instead of opt-out to avoid
having APIs to include extraneous data (and being much heavier than
necessary).
|
| | |
| |
| | |
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* remove `start_active_span_from_request`
Instead, pull out a separate function, `span_context_from_request`, to extract
the parent span, which we can then pass into `start_active_span` as
normal. This seems to be clearer all round.
* Remove redundant tags from `incoming-federation-request`
These are all wrapped up inside a parent span generated in AsyncResource, so
there's no point duplicating all the tags that are set there.
* Leave request spans open until the request completes
It may take some time for the response to be encoded into JSON, and that JSON
to be streamed back to the client, and really we want that inside the top-level
span, so let's hand responsibility for closure to the SynapseRequest.
* opentracing logs for HTTP request events
* changelog
|
| |\ \ |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Disable aggregation bundling on `/sync` responses
A partial revert of #11478. This turns out to have had a significant CPU impact
on initial-sync handling. For now, let's disable it, until we find a more
efficient way of achieving this.
* Fix tests.
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
|
| | |
| |
| |
| | |
Instead of returning 404 errors with HTML bodies when an unknown
prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
|
| | |
| |
| | |
Instead of the backported version.
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
|
| | | |
|
| | |
| |
| |
| | |
Instead of Iterable since the generators are not allowed due
to the potential for their re-use.
|
| | | |
|
| | |
| |
| |
| | |
masquerade as specific devices. (#11538)
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
`COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a
`COUNT(NULL)` is also 0.
|
| | | |
|
| | |
| |
| |
| |
| | |
A couple of safety-checks to hopefully stop people doing what I just did, and create a storage
function which only works the first time it is called (and not when it is re-run due to a database
concurrency error or similar).
|
| | |
| |
| |
| |
| |
| |
| | |
* Splits the logic for parsing HTML from the resource handling code.
* Fix a circular import in the oEmbed code (which uses the HTML parsing code).
* Renames some of the HTML parsing methods to:
* Make it clear which methods are "internal" to the module.
* Clarify what the methods do.
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| | |
| |
| |
| |
| | |
The event still needs to have `auth_events` defined to be valid.
Split out from https://github.com/matrix-org/synapse/pull/11114
|
| | |
| |
| |
| |
| |
| |
| |
| | |
batch (MSC2716) (#11487)
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
We did some work on making sure the `state_groups` were shared in
https://github.com/matrix-org/synapse/pull/10975
|
| | |
| |
| | |
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
|
| | |
| |
| |
| |
| |
| | |
Create a new dict helper method `simple_insert_many_values_txn`, which takes
raw row values, rather than {key=>value} dicts. This saves us a bunch of dict
munging, and makes it easier to use generators rather than creating
intermediate lists and dicts.
|
| | |
| |
| | |
this field is never read, so we may as well stop populating it.
|
| | |
| |
| |
| | |
This should fix pushers with an email in non-canonical form is used as
the pushkey.
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| | |
And expand some type hints in the receipts storage module.
|
| | | |
|
| | |
| |
| | |
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
| |
| |
| | |
`tests.rest.client.test_auth`. (#11520)
|
| | |
| |
| |
| | |
Somehow I'd managed to get my database in a pickle with stream ids. These
changes were useful to debug.
|
| | | |
|
| |/
|
|
|
|
| |
(#11331)
Pull the DataStore from the HomeServer instance, which
always exists.
|
| | |
|
| |
|
|
|
|
| |
`matrix-python-common` (#11505) (#11527)
This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
|
| |
|
|
|
|
|
|
| |
* Move sync_token up to the top
* Pull out _get_ignored_users
* Try to signpost the body of `_generate_sync_entry_for_rooms`
* Pull out _calculate_user_changes
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
| |
Fixes #11517
|
| |
|
|
|
|
| |
After #10847, `looping_background_call` would print an error in the logs
every time a non-async function was called. Since the error would be
caught and ignored immediately, there were no other side effects.
|
| |
|
|
|
| |
If, for some reason, we don't have the create event, we should still be able to
purge a room.
|
| | |
|
| |
|
|
| |
(#11522)
|
| |
|
|
| |
into the Matrix specification. (#11435)
|
| |
|
| |
As a step towards allowing back-channel logout for OIDC.
|
| | |
|
| |
|
|
|
|
|
|
| |
Due to updates to MSC2675 this includes a few fixes:
* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
|
| | |
|
| |
|
|
|
|
|
| |
whether we have fetched an events' `prev_events` (#11469)
Spawning from https://github.com/matrix-org/synapse/pull/9445#discussion_r758958181
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
| |
to refreshable access tokens. (#11445)
|
| | |
|
| | |
|
| |
|
|
| |
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
|
| |
We're going to add a `state_key` column to the `events` table, so we need to
add some disambiguation to queries which use it.
|
| |
|
|
|
| |
This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* move wiki pages to synapse/docs and add a few titles where necessary
* update SUMMARY.md with added pages
* add changelog
* move incorrectly located newsfragment
* update changelog number
* snake case added files and update summary.md accordingly
* update issue/pr links
* update relative links to docs
* update changelog to indicate that we moved wiki pages to the docs and state reasoning
* requested changes to admin_faq.md
* requested changes to database_maintenance_tools.md
* requested changes to understanding_synapse_through_graphana_graphs.md
* add changelog
* fix leftover merge errata
* fix unwanted changes from merge
* use two spaces between entries
* outdent code blocks
|
| |
|
| |
This skips a few methods which are difficult to type.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
closest event to a given timestamp (#9445)
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030
Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
```
Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
"event_id": ...
"origin_server_ts": ...
}
```
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* move wiki pages to synapse/docs and add a few titles where necessary
* update SUMMARY.md with added pages
* add changelog
* move incorrectly located newsfragment
* update changelog number
* snake case added files and update summary.md accordingly
* update issue/pr links
* update relative links to docs
* update changelog to indicate that we moved wiki pages to the docs and state reasoning
* revert unintentional change to CHANGES.md
* add link
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Update CHANGES.md
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
This adds some misc. type hints to helper methods used
in the `synapse.config` module.
|
| |
|
| |
As specified for Matrix v1.1.
|
| |
|
|
|
| |
* Add note to postgres doc about hugepages
* Newsfragment
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add check to catch syanpse master process starting when workers are configured
* add test to verify that starting master process with worker config raises error
* newsfragment
* specify config.worker.worker_app in check
* update test
* report specific config option that triggered the error
Co-authored-by: reivilibre <oliverw@matrix.org>
* clarify error message
Co-authored-by: reivilibre <oliverw@matrix.org>
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
| |
Per updates to MSC2675 which now states that bundled
aggregations should be included from the `/relations`
endpoint.
|
| |
|
|
|
|
|
|
|
|
|
| |
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.
Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
|
| | |
|
| |
|
|
|
| |
The tests helpers automatically convert dictionaries to
JSON payloads, no need to do it manually for each
test.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
If you're trying to shut down Synapse, it's rather handy if it *actually* shuts
down before you move on.
|
| | |
| |
| | |
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
(`_persist_events_and_state_updates`) (#11417)
Part of https://github.com/matrix-org/synapse/issues/11300
Call stack:
- `_persist_events_and_state_updates` (added `use_negative_stream_ordering`)
- `_persist_events_txn`
- `_update_room_depths_txn` (added `update_room_forward_stream_ordering`)
- `_update_metadata_tables_txn`
- `_store_room_members_txn` (added `inhibit_local_membership_updates`)
Using keyword-only arguments (`*`) to reduce the mistakes from `backfilled` being left as a positional argument somewhere and being interpreted wrong by our new arguments.
|
| | |
| |
| |
| |
| |
| | |
This also makes additional updates where the implementation
had drifted from the approved MSC.
Unstable endpoints will be removed at a later data.
|
| | |
| |
| | |
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| | |
| |
| |
| |
| | |
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
|
| | | |
|
| | |
| |
| |
| | |
tokens. (#11428)
|
| | |
| |
| |
| |
| | |
If we tried to request multiple keys for the same server, we would end up
dropping some of those requests.
|
| | |
| |
| |
| | |
Signed-off-by: Tulir Asokan <tulir@beeper.com>
|
| | | |
|
| | | |
|
| | |
| |
| | |
This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since e81fa9264873369653171157514ff68226491fff, Synapse depends on
the use_float flag which has been introduced in ijson 3.1 and
is not available in 3.0. This is known to cause runtime errors
with send_join.
Signed-off-by: Daniel Molkentin <danimo@infra.run>
Co-authored-by: Daniel Molkentin <danimo@infra.run>
|
| | |
| |
| |
| | |
accept the `refresh_tokens` parameter in the request body rather than in the URL parameters. (#11430)
|
| | |
| |
| |
| | |
Also refactor the stream ID trackers/generators a bit and try to
document them better.
|
| | |
| |
| |
| | |
refresh tokens are in use. (#11425)
|
| | |
| |
| |
| |
| | |
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The previous fix for the ongoing event fetches counter
(8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and
incorrect.
When the database is unreachable, `_do_fetch` never gets run and so
`_event_fetch_ongoing` is never decremented.
The previous fix also moved the `_event_fetch_ongoing` decrement outside
of the `_event_fetch_lock` which allowed race conditions to corrupt the
counter.
|
| |/
|
|
|
|
|
|
|
|
| |
This change makes mypy complain if the constants are ever reassigned,
and, more usefully, makes mypy type them as `Literal`s instead of `str`s,
allowing code of the following form to pass mypy:
```py
def do_something(membership: Literal["join", "leave"], ...): ...
do_something(Membership.JOIN, ...)
```
|
| |
|
|
|
| |
(#11421)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
| |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
| |
`refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. (#11388)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag
* changelog entry
* update changelog
* Delete 11394.removal
Duplicate, wrong number
* add no-op background update and change newfragment so it will be consolidated with associated work
* remove unused code
* Remove code associated with deprecated flag from legacy docker dynamic config file
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| | |
|
| |
|
|
|
| |
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
|
| |
|
|
|
| |
Instead of only known relation types. This also reworks the background
update for thread relations to crawl events and search for any relation
type, not just threaded relations.
|
| |
|
|
|
|
| |
Instead of having admins poke into the database directly.
Can currently run jobs to populate stats and to populate
the user directory.
|
| |
|
|
|
|
|
|
| |
If `room_list_publication_rules` was configured with a rule with a
non-wildcard alias and a room was created with an alias then an
internal server error would have been thrown.
This fixes the error and properly applies the publication rules
during room creation.
|
| | |
|
| |
|
| |
As the tests are currently failing and not run in CI.
|
| | |
|
| |
|
|
| |
I'd find it helpful to have a docker image corresponding to current develop,
without having to build my own.
|
