summary refs log tree commit diff
path: root/synapse/rest (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-01-25Add admin API to reset connection timeouts for remote server (#11639)Dirk Klimpel2-3/+47
* Fix get federation status of destination if no error occured
2022-01-25Ignore the jsonschema type. (#11817)Patrick Cloke2-1/+4
2022-01-24Skip the initial amd64-only Docker build (#11810)Richard van der Hoff2-12/+3
PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our arm64 docker images now build in a sensible amount of time, so we can skip the amd64-only build.
2022-01-24Apply a timeout to reading the body when fetching a file. (#11784)Patrick Cloke2-3/+13
This prevents the URL preview code from reading a stream forever.
2022-01-24Support rendering previews with data: URLs in them (#11767)Patrick Cloke6-74/+299
Images which are data URLs will no longer break URL previews and will properly be "downloaded" and thumbnailed.
2022-01-241.50.2 v1.50.2Andrew Morgan4-2/+16
2022-01-24Remove account data (including client config, push rules and ignored users) ↵reivilibre5-3/+299
upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24Fix logic for dropping old events in fed queue (#11806)Andrew Morgan3-7/+27
Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-01-241.51.0rc2 v1.51.0rc2Andrew Morgan4-2/+16
2022-01-24Fix logic for dropping old events in fed queue (#11806)Andrew Morgan3-7/+29
Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-01-21Drop support for and remove references to EOL Python 3.6 (#11683)Shay13-28/+25
* remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
2022-01-21Remove obsolete newsfileOlivier Wilkinson (reivilibre)1-1/+0
The PR was cherrypicked into v1.51.0rc1.
2022-01-21Edit the changelog according to feedback v1.51.0rc1Olivier Wilkinson (reivilibre)1-3/+3
2022-01-21Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff5-5/+42
... and start populating them for new events
2022-01-21Edit the changelog for grammar and clarityOlivier Wilkinson (reivilibre)1-16/+15
2022-01-211.51.0rc1Olivier Wilkinson (reivilibre)59-58/+83
2022-01-21Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke7-54/+85
2022-01-21Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke7-54/+85
2022-01-21Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff5-3/+22
This is a follow-up to #10565.
2022-01-21Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff5-33/+31
Preparation for dropping this table altogether. Part of #6574.
2022-01-21Add `FrozenEvent.get_state_key` and use it in a couple of places (#11793)Richard van der Hoff3-5/+11
This is more efficient, since we only have to look up `state_key` in the event dict once, rather than three (!) times.
2022-01-21Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre6-9/+10
whose key is prefixed with the user ID (#11788)
2022-01-21Make `get_account_data_for_room_and_type` a tree cache (#11789)reivilibre2-1/+2
2022-01-20Correctly await on_logged_out callbacks (#11786)Brendan Abolivier3-2/+29
2022-01-20Fix redirecting to the webclient for non-HTTP(S) web_client_location. (#11783)Patrick Cloke2-1/+9
To not change the behaviour during the deprecation period. Follow-up to #11774.
2022-01-20Add a regression test for using both webclient and client resources ↵Andrew Morgan2-0/+109
simultaneously (#11765)
2022-01-20Partially revert #11675; prevent attempting to create pushers on workers ↵Andrew Morgan2-7/+6
(#11770)
2022-01-20installation.md: drop python 3.6 support (#11781)Richard van der Hoff2-1/+2
#11595 dropped support for python 3.6, but forgot to update this doc.
2022-01-20Add deprecation warnings for `webclient` listener and non-HTTP(S) ↵Patrick Cloke5-28/+40
`web_client_location`. (#11774) This changes the behaviour of the root endpoint to redirect directly to the configuration of `web_client_location` if it is given an HTTP(S) URL.
2022-01-20Debug for device lists updates (#11760)David Robertson4-0/+46
Debug for #8631. I'm having a hard time tracking down what's going wrong in that issue. In the reported example, I could see server A sending federation traffic to server B and all was well. Yet B reports out-of-sync device updates from A. I couldn't see what was _in_ the events being sent from A to B. So I have added some crude logging to track - when we have updates to send to a remote HS - the edus we actually accumulate to send - when a federation transaction includes a device list update edu - when such an EDU is received This is a bit of a sledgehammer.
2022-01-20Allow overriding the complement ref. (#11766)Nicolas Werner3-7/+10
Updates complement.sh to read the ref from an environment variable (defaulting to master) when downloading a complement bundle for testing.
2022-01-20Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah4-12/+102
`FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
2022-01-19Comments and typing for `_update_outliers_txn` (#11776)Richard van der Hoff2-12/+24
A couple of surprises for me here, so thought I'd document them
2022-01-19Fix mypy for platforms without epoll support. (#11771)Patrick Cloke2-2/+3
2022-01-19Add missing `auto_attribs=True` to the `_WrappedRustReporter` class (#11768)Andrew Morgan2-1/+2
2022-01-18Fix preview of imgur and Tenor URLs. (#11669)Philippe Daouadi4-14/+39
By scraping Open Graph information from the HTML even when an autodiscovery endpoint is found. The results are then combined to capture as much information as possible from the page.
2022-01-18Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke9-18/+85
Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
2022-01-18Wording fixes to 1.50.0/1 changelog entries v1.50.1Andrew Morgan1-3/+3
2022-01-18Reword 1.50.0 warning a bit in the changelogAndrew Morgan1-1/+3
2022-01-181.50.1Andrew Morgan4-2/+20
2022-01-18Partially revert #11602 to prevent webclient overriding client resource (#11764)Andrew Morgan2-2/+8
2022-01-18Remove `log_function` and its uses (#11761)Richard van der Hoff13-155/+1
I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
2022-01-18Move python/postgres deprecation notice to the top of 1.50 changelog v1.50.0Andrew Morgan1-2/+2
2022-01-181.50.0Andrew Morgan3-4/+15
2022-01-17Add a flag to the `synapse_review_recent_signups` script to ignore and ↵lukasdenk3-8/+23
filter appservice users. (#11675)
2022-01-17Warn against using Let's Encrypt certs for encrypted TURN (#11686)AndrewFerr2-0/+17
* Warn against using Let's Encrypt certs for encrypted TURN This helps to avoid client-side issues: * https://github.com/vector-im/element-android/issues/1533 * https://github.com/vector-im/element-ios/issues/2712 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2022-01-17Fix up docs for `track_puppeted_user_ips` (again) (#11757)Richard van der Hoff5-10/+27
Fixes #11741
2022-01-17Update documentation for configuring facebook login (#11755)Richard van der Hoff4-18/+26
... and a minor thinko fix in the sample config.
2022-01-17Improve `reactor_tick_time` metric (#11724)Richard van der Hoff5-109/+86
The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* measures the time taken to execute timed calls and callbacks from threads. That neglects everything that happens off the back of I/O, which is obviously quite a lot for us. To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is the only place it should wait for something to happen - the rest of the loop *should* be quick. I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and it's a nuisance to populate.
2022-01-17Make pagination of rooms in admin api stable (#11737)Daniel Sonck3-28/+38
Always add state.room_id after the configurable ORDER BY. Otherwise, for any sort, certain pages can contain results from other pages. (Especially when sorting by creator, since there may be many rooms by the same creator) * Document different order direction of numerical fields "joined_members", "joined_local_members", "version" and "state_events" are ordered in descending direction by default (dir=f). Added a note in tests to explain the differences in ordering. Signed-off-by: Daniël Sonck <daniel@sonck.nl>
2022-01-14Remove the 'password_hash' from the Users Admin API endpoint response ↵Andrew Morgan5-43/+86
dictionary (#11576)
2022-01-14Fix sample_config.yaml in regards track_puppeted_user_ips (#11749)Jason Robinson3-10/+13
* Fix sample_config.yaml in regards track_puppeted_user_ips Closes #11741 Signed-off-by: Jason Robinson <jasonr@matrix.org>
2022-01-14Tweak the changelog summary section v1.50.0rc2Olivier Wilkinson (reivilibre)1-4/+3
2022-01-141.50.0rc2Olivier Wilkinson (reivilibre)7-5/+33
2022-01-13Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke19-298/+263
This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
2022-01-13 Fix missing app variable in mail subject (#11745)David Robertson3-2/+4
documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500 Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
2022-01-13Invite PR submitters to credit themselves (#11744)David Robertson2-0/+2
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-13Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre3-17/+190
fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
2022-01-13update room spec url in config files (#11739)qwertyforce3-2/+3
* change spec url in config files * Create 11739.txt * .txt -> .doc
2022-01-13Mention python3-venv and libpq-dev dependencies in contribution guide (#11740)Andy Balaam2-1/+4
2022-01-13Include bundled aggregations in the sync response cache. (#11659)Patrick Cloke4-19/+19
2022-01-13Simplify GC prometheus metrics (#11723)Richard van der Hoff4-160/+209
Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection. Part 1 of a quest to simplify the reactor monkey-patching.
2022-01-13Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke40-307/+300
2022-01-13Fix typo in demo docs: differnt (#11735)Andy Balaam2-1/+2
2022-01-12Allow tracking puppeted users for MAU (#11561)Jason Robinson5-0/+63
Currently when puppeting another user, the user doing the puppeting is tracked for client IPs and MAU (if configured). When tracking MAU is important, it becomes necessary to be possible to also track the client IPs and MAU of puppeted users. As an example a client that manages user creation and creation of tokens via the Synapse admin API, passing those tokens for the client to use. This PR adds optional configuration to enable tracking of puppeted users into monthly active users. The default behaviour stays the same. Signed-off-by: Jason Robinson <jasonr@matrix.org>
2022-01-12Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre3-2/+57
be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-12Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)David Robertson2-1/+8
* Deal with mypy errors w/ type-hinted pynacl 1.5.0 Fixes #11644. I really don't like that we're monkey patching pynacl SignedKey instances with alg and version objects. But I'm too scared to make the changes necessary right now. (Ideally I would replace `signedjson.types.SingingKey` with a runtime class which wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
2022-01-12Fix documentation of supported PostgreSQL version (#11725)haslersn2-1/+2
Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
2022-01-12Fix documentation of supported PostgreSQL version (#11725)haslersn2-1/+2
Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
2022-01-12Use buildkit's cache feature to speed up docker builds (#11691)Richard van der Hoff4-5/+28
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-10Complement environment variable name change and update .gitignore. (#11718)Michael Kaye3-1/+6
2022-01-10Document the `SYNAPSE_TEST_PERSIST_SQLITE_DB` unit test env var (#11715)Andrew Morgan2-0/+22
2022-01-10Fix docstring on `add_account_data_for_user`. (#11716)reivilibre3-2/+3
2022-01-10Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)David Robertson2-1/+8
* Deal with mypy errors w/ type-hinted pynacl 1.5.0 Fixes #11644. I really don't like that we're monkey patching pynacl SignedKey instances with alg and version objects. But I'm too scared to make the changes necessary right now. (Ideally I would replace `signedjson.types.SingingKey` with a runtime class which wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
2022-01-10Prettier complement logs (#11707)kegsay2-1/+3
* Prettier complement logs * Changelog
2022-01-07Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke3-8/+55
By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
2022-01-07Optionally use an on-disk sqlite db in tests (#11702)David Robertson3-1/+23
* Optionally use an on-disk sqlite db in tests When debugging a test it is sometimes useful to inspect the state of the DB. This is not easy when the db is in-memory: one cannot attach the sqlite CLI to another process's DB. With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use `_trial_temp/test.db` as our sqlite database. One can then use `sqlite3 _trial_temp/test.db` and query to your heart's content. The DB is destroyed and recreated between different test cases. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-07Test that bans win a join against a race when computing `/sync` response ↵David Robertson3-3/+105
(#11701)
2022-01-07Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke17-156/+249
This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
2022-01-07Move changelog entry for drop of support to 'Deprecations and Removals'Olivier Wilkinson (reivilibre)1-1/+1
2022-01-06Remove the /send_relation endpoint. (#11682)Patrick Cloke3-133/+19
This was removed from MSC2674 before that was approved and is not used by any known clients.
2022-01-06Remove a Python 2-ism and improve type hints. (#11699)Patrick Cloke2-12/+8
On Python 2, indexing a byte-string gives back a byte-string, while on Python 3 it gives back the ASCII equivalent as an int.
2022-01-06Re-run Towncrier to add in the changelog entry for the Mjolnir workaround v1.50.0rc1Olivier Wilkinson (reivilibre)2-1/+1
2022-01-06Strip unauthorized fields from `unsigned` object in events received over ↵Shay3-0/+99
federation (#11530) * add some tests to verify we are stripping unauthorized fields out of unsigned * add function to strip unauthorized fields from the unsigned object of event * newsfragment * update newsfragment number * add check to on_send_membership_event * refactor tests * fix lint error * slightly refactor tests and add some comments * slight refactor * refactor tests * fix import error * slight refactor * remove unsigned filtration code from synapse/handlers/federation_event.py * lint * move unsigned filtering code to event base * refactor tests * update newsfragment * requested changes * remove unused retun values
2022-01-06Work around Mjolnir compatibility issue by adding an import for ↵reivilibre2-0/+8
`glob_to_regex` in `synapse.util`, where it moved from. (#11696)
2022-01-06Make room creations denied by `user_may_create_room` cause an `M_FORBIDDEN` ↵lukasdenk2-2/+7
error to be returned, not `M_UNKNOWN` (#11672) Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-06Remove debug logging for #4422 (#11693)David Robertson2-37/+2
as per https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
2022-01-05Fix space hierarchy endpoint to match MSC2946 (#11667)Travis Ralston2-2/+5
Fixes minor discrepancies between the /hierarchy endpoint described in MSC2946 and the implementation. Note that the changes impact the stable and unstable /hierarchy and unstable /spaces endpoints for both client and federation APIs.
2022-01-05Fix get federation status of destination if no error occured (#11593)Dirk Klimpel4-25/+88
2022-01-05Tweak changelog for #11677Olivier Wilkinson (reivilibre)1-1/+1
2022-01-05Run `pyupgrade --py37-plus --keep-percent-format` on Synapse (#11685)Shay3-2/+2
* newsfragment * fix newsfragment number * update changelog * remove extra space
2022-01-05Re-run towncrier.Patrick Cloke2-1/+1
2022-01-05Include `io.element.thread` capability for MSC3440. (#11690)Patrick Cloke2-0/+4
2022-01-05Mention drop of support in changelogOlivier Wilkinson (reivilibre)1-0/+3
2022-01-05Add support removal notice to the upgrade notesOlivier Wilkinson (reivilibre)1-0/+11
2022-01-05Tweak changelogOlivier Wilkinson (reivilibre)1-9/+8
2022-01-05Cache empty responses from `/user/devices` (#11587)David Robertson5-5/+114
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-051.50.0rc1Olivier Wilkinson (reivilibre)75-73/+92
2022-01-05Refactor the way we set `outlier` (#11634)Richard van der Hoff5-47/+16
* `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
2022-01-05Fix link from generated configuration file to documentation (#11678)Philipp Matthias Schäfer3-2/+3
Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: reivilibre <oliverw@matrix.org>
2022-01-05Fix SimpleHttpClient not sending Accept header in `get_json` (#11677)Fr3shTea2-1/+2
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05Add admin API to get users' account data (#11664)Dirk Klimpel5-0/+198
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre9-123/+14
#11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05Clarify SSO mapping provider documentation by writing `def` or `async def` ↵reivilibre2-12/+13
before the names of methods, as appropriate. (#11681)
2022-01-05Improve Docker docs for use with Postgres (#11640)Callum Macdonald3-0/+6
2022-01-05Correct Synapse install command for FreeBSD. (#11267)Donny Johnson2-1/+2
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-04update ngnix reverse-proxy example (#11680)Richard van der Hoff2-1/+2
this should not be a case-insensitive match.
2022-01-04Fix AssertionErrors after purging events (#11642)Richard van der Hoff6-6/+30
* Fix AssertionErrors after purging events If you purged a bunch of events from your database, and then restarted synapse without receiving more events, then you would get a bunch of AssertionErrors on restart. This fixes the situation by rewinding the stream processors. * `check-newsfragment`: ignore deleted newsfiles
2022-01-04`FederationClient.backfill`: stop flagging events as outliers (#11632)Richard van der Hoff3-2/+5
Events returned by `backfill` should not be flagged as outliers. Fixes: ``` AssertionError: null File "synapse/handlers/federation.py", line 313, in try_backfill dom, room_id, limit=100, extremities=extremities File "synapse/handlers/federation_event.py", line 517, in backfill await self._process_pulled_events(dest, events, backfilled=True) File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events await self._process_pulled_event(origin, ev, backfilled=backfilled) File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event assert not event.internal_metadata.is_outlier() ``` See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992 Fixes #8894.
2022-01-04Remove redundant `get_current_events_token` (#11643)Richard van der Hoff7-36/+20
* Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
2022-01-04Better error messages from `get_create_event_for_room` (#11638)Richard van der Hoff2-1/+6
"Unknown room" can mean a multitude of things here. To help with debugging, add some more words to the exception text.
2022-01-04Include the topic event in the prejoin state, per MSC3173. (#11666)Patrick Cloke4-0/+13
Invites and knocks will now include the topic in the stripped state send to clients before joining the room.
2022-01-03Drop Bionic from Debian builds (#11633)Shay4-7/+5
* update Trove classifiers to remove py36 * stop building bionic * update dh-virtualenv * newsfragment * fix newsfragment * update version refs * another try at correct tag * Update changelog
2021-12-30Convert all namedtuples to attrs. (#11665)Patrick Cloke22-206/+231
To improve type hints throughout the code.
2021-12-30Add type hints to `synapse/storage/databases/main/events_bg_updates.py` (#11654)Dirk Klimpel3-30/+44
2021-12-29Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre3-3/+3
exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-29Improve type hints in storage classes. (#11652)Dirk Klimpel10-34/+44
By using cast and making ignores more specific.
2021-12-29Do not attempt to bundled aggregations for /members and /state. (#11623)Patrick Cloke2-3/+2
Both of those APIs return state events, which will not have bundled aggregations added anyway.
2021-12-29Add type hints to `synapse/storage/databases/main/stats.py` (#11653)Dirk Klimpel3-42/+57
2021-12-23Update to the current version of Black and run it on Synapse codebase (#11596)Shay5-4/+3
* update black version * run updated version of black on code * newsfragment * enumerate python versions
2021-12-23Fix mypy error with opentracing.tags. (#11622)Patrick Cloke2-0/+2
2021-12-23Add details for how to set up TURN behind NAT (#11553)AndrewFerr2-15/+53
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-12-21Drop EOL python 3.6 from CI (#11595)Shay3-7/+8
* remove python 3.6 and postgres 9.6 from github workflow * remove python 3.6 env from tox * newsfragment * correct postgres version * add py310 to tox env list
2021-12-21update changelog v1.49.2Richard van der Hoff1-1/+1
postgres 10 _+_
2021-12-211.49.2Richard van der Hoff4-3/+19
2021-12-21Pin to frozendict<2.1.2 (#11625)Richard van der Hoff2-1/+3
... to work around breakage on buster (https://github.com/Marco-Sulla/python-frozendict/issues/41)
2021-12-21Refactor `tests.util.setup_test_homeserver` and ↵reivilibre5-185/+195
`tests.server.setup_test_homeserver`. (#11503)
2021-12-21Add type hints to event_push_actions. (#11594)Patrick Cloke11-154/+225
2021-12-21Add notes about dropping support for Python 3.6 and Postgres 9.6. v1.49.1Richard van der Hoff2-2/+6
2021-12-21typopoRichard van der Hoff1-1/+1
2021-12-21more wordsRichard van der Hoff1-0/+2
2021-12-21Various opentracing enhancements (#11619)Richard van der Hoff5-21/+65
* Wrap `auth.get_user_by_req` in an opentracing span give `get_user_by_req` its own opentracing span, since it can result in a non-trivial number of sub-spans which it is useful to group together. This requires a bit of reorganisation because it also sets some tags (and may force tracing) on the servlet span. * Emit opentracing span for encoding json responses This can be a significant time sink. * Rename all sync spans with a prefix * Write an opentracing span for encoding sync response * opentracing span to group generate_room_entries * opentracing spans within sync.encode_response * changelog * Use the `trace` decorator instead of context managers
2021-12-211.49.1Richard van der Hoff4-2/+16
2021-12-20Do not bundle aggregations for APIs which shouldn't include them. (#11592)Patrick Cloke9-27/+31
And make bundling aggregations opt-in, instead of opt-out to avoid having APIs to include extraneous data (and being much heavier than necessary).
2021-12-20Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff5-48/+149
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff4-73/+65
* remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
2021-12-20Disable aggregation bundling on `/sync` responses (#11583)Richard van der Hoff3-6/+15
* Disable aggregation bundling on `/sync` responses A partial revert of #11478. This turns out to have had a significant CPU impact on initial-sync handling. For now, let's disable it, until we find a more efficient way of achieving this. * Fix tests. Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-12-20Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke3-10/+6
Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
2021-12-20Use mock from standard library (#11588)V024603-12/+10
Instead of the backported version.
2021-12-20Add opentracing types (#11603)Shay5-13/+18
2021-12-16Add type hints to `synapse/tests/rest/admin` (#11590)Dirk Klimpel7-57/+74
2021-12-16Add type hints to `synapse/storage/databases/main/transactions.py` (#11589)Dirk Klimpel3-25/+29
2021-12-16Add MSC2716 and MSC3030 to `/versions` -> `unstable_features` (#11582)Eric Eastwood3-1/+6
As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
2021-12-15Add type hints to `synapse/storage/databases/main/room.py` (#11575)Sean Quah5-77/+108
2021-12-15Require Collections as the parameters for simple_* methods. (#11580)Patrick Cloke3-40/+12
Instead of Iterable since the generators are not allowed due to the potential for their re-use.
2021-12-15Convert EventStreamResult to attrs. (#11574)Patrick Cloke3-9/+24
2021-12-15Add experimental support for MSC3202: allowing application services to ↵reivilibre5-14/+162
masquerade as specific devices. (#11538)
2021-12-14Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah6-79/+188
2021-12-14Add missing type hints to `synapse.logging.context` (#11556)Sean Quah13-122/+215
2021-12-14Add missing type hints to `synapse.appservice` (#11360)Patrick Cloke7-93/+148
2021-12-14Revert "Revert accidental fast-forward merge from v1.49.0rc1"Olivier Wilkinson (reivilibre)165-2708/+7720
This reverts commit 158d73ebdd61eef33831ae5f6990acf07244fc55.
2021-12-14Restructure changelog v1.49.0Olivier Wilkinson (reivilibre)1-2/+9
2021-12-141.49.0Olivier Wilkinson (reivilibre)3-1/+18
2021-12-14Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah9-21/+20
`COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
2021-12-14Add missing type hints to synapse.http. (#11571)Patrick Cloke8-51/+76
2021-12-13checks for generators in database functions (#11564)Richard van der Hoff3-7/+49
A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).
2021-12-13Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke6-401/+432
* Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
2021-12-13Type hint the constructors of the data store classes (#11555)Sean Quah35-87/+351
2021-12-13Add type hints to `synapse/storage/databases/main/end_to_end_keys.py` (#11551)Sean Quah4-69/+150
2021-12-13Add type hints to `synapse/storage/databases/main/account_data.py` (#11546)Sean Quah4-33/+87
2021-12-13Make `get_device` return None if the device doesn't exist rather than ↵reivilibre6-13/+20
raising an exception. (#11565) Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-10Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood4-7/+124
The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
2021-12-10Test to ensure we share the same `state_group` across the whole historical ↵Eric Eastwood2-0/+181
batch (MSC2716) (#11487) Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 We did some work on making sure the `state_groups` were shared in https://github.com/matrix-org/synapse/pull/10975
2021-12-10Adjust _get_rooms_changed comments (#11550)David Robertson2-22/+30
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
2021-12-10skip some dict munging in event persistence (#11560)Richard van der Hoff3-60/+114
Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.
2021-12-10Stop populating `state_events.prev_state` (#11558)Richard van der Hoff3-5/+5
this field is never read, so we may as well stop populating it.
2021-12-10Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke5-4/+11
This should fix pushers with an email in non-canonical form is used as the pushkey.
2021-12-09Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke6-17/+161
2021-12-09Additional type hints for the config module, part 2. (#11480)Patrick Cloke5-19/+28
2021-12-09Use HTTPStatus constants in place of literals in `synapse.http` (#11543)Dirk Klimpel4-19/+47
2021-12-09Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi4-6/+32
2021-12-09Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel6-12/+13
2021-12-09Allow guests to send state events (#11378)Robert Long2-1/+2
2021-12-08Add a constant for receipt types (m.read). (#11531)Patrick Cloke9-45/+87
And expand some type hints in the receipts storage module.
2021-12-08Clean up `synapse.rest.admin` (#11535)Dirk Klimpel15-165/+96
2021-12-08Document the usage of refresh tokens. (#11427)reivilibre3-0/+141
Co-authored-by: David Robertson <davidr@element.io>
2021-12-08Use HTTPStatus constants in place of literals in ↵reivilibre2-47/+88
`tests.rest.client.test_auth`. (#11520)
2021-12-08Improve log messages for stream ids (#11536)Richard van der Hoff3-4/+7
Somehow I'd managed to get my database in a pickle with stream ids. These changes were useful to debug.
2021-12-08Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi4-5/+13
2021-12-07Fix case in `wait_for_background_updates` where `self.store` does not exist ↵Andrew Morgan2-7/+5
(#11331) Pull the DataStore from the HomeServer instance, which always exists.
2021-12-07Revert accidental fast-forward merge from v1.49.0rc1Sean Quah165-7720/+2708
Revert "Sort internal changes in changelog" Revert "Update CHANGES.md" Revert "1.49.0rc1" Revert "Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505) (#11527)" Revert "Refactors in `_generate_sync_entry_for_rooms` (#11515)" Revert "Correctly register shutdown handler for presence workers (#11518)" Revert "Fix `ModuleApi.looping_background_call` for non-async functions (#11524)" Revert "Fix 'delete room' admin api to work on incomplete rooms (#11523)" Revert "Correctly ignore invites from ignored users (#11511)" Revert "Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522)" Revert "Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435)" Revert "Save the OIDC session ID (sid) with the device on login (#11482)" Revert "Add admin API to get some information about federation status (#11407)" Revert "Include bundled aggregations in /sync and related fixes (#11478)" Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)" Revert "Update backward extremity docs to make it clear that it does not indicate whether we have fetched an events' `prev_events` (#11469)" Revert "Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445)" Revert "Add type hints to `synapse/tests/rest/admin` (#11501)" Revert "Revert accidental commits to develop." Revert "Newsfile" Revert "Give `tests.server.setup_test_homeserver` (nominally!) the same behaviour" Revert "Move `tests.utils.setup_test_homeserver` to `tests.server`" Revert "Convert one of the `setup_test_homeserver`s to `make_test_homeserver_synchronous`" Revert "Disambiguate queries on `state_key` (#11497)" Revert "Comments on the /sync tentacles (#11494)" Revert "Clean up tests.storage.test_appservice (#11492)" Revert "Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)" Revert "Clean up `tests.test_visibility` to remove legacy code. (#11495)" Revert "Minor cleanup on recently ported doc pages (#11466)" Revert "Add most of the missing type hints to `synapse.federation`. (#11483)" Revert "Avoid waiting for zombie processes in `synctl stop` (#11490)" Revert "Fix media repository failing when media store path contains symlinks (#11446)" Revert "Add type annotations to `tests.storage.test_appservice`. (#11488)" Revert "`scripts-dev/sign_json`: support for signing events (#11486)" Revert "Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)" Revert "Port wiki pages to documentation website (#11402)" Revert "Add a license header and comment. (#11479)" Revert "Clean-up get_version_string (#11468)" Revert "Link background update controller docs to summary (#11475)" Revert "Additional type hints for config module. (#11465)" Revert "Register the login redirect endpoint for v3. (#11451)" Revert "Update openid.md" Revert "Remove mention of OIDC certification from Dex (#11470)" Revert "Add a note about huge pages to our Postgres doc (#11467)" Revert "Don't start Synapse master process if `worker_app` is set (#11416)" Revert "Expose worker & homeserver as entrypoints in `setup.py` (#11449)" Revert "Bundle relations of relations into the `/relations` result. (#11284)" Revert "Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)" Revert "Eliminate a few `Any`s in `LruCache` type hints (#11453)" Revert "Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)" Revert "Merge branch 'master' into develop" This reverts commit 26b5d2320f62b5eb6262c7614fbdfc364a4dfc02. This reverts commit bce4220f387bf5448387f0ed7d14ed1e41e40747. This reverts commit 966b5d0fa0893c3b628c942dfc232e285417f46d. This reverts commit 088d748f2cb51f03f3bcacc0fb3af1e0f9607737. This reverts commit 14d593f72d10b4d8cb67e3288bb3131ee30ccf59. This reverts commit 2a3ec6facf79f6aae011d9fb6f9ed5e43c7b6bec. This reverts commit eccc49d7554d1fab001e1fefb0fda8ffb254b630. This reverts commit b1ecd19c5d19815b69e425d80f442bf2877cab76. This reverts commit 9c55dedc8c4484e6269451a8c3c10b3e314aeb4a. This reverts commit 2d42e586a8c54be1a83643148358b1651c1ca666. This reverts commit 2f053f3f82ca174cc1c858c75afffae51af8ce0d. This reverts commit a15a893df8428395df7cb95b729431575001c38a. This reverts commit 8b4b153c9e86c04c7db8c74fde4b6a04becbc461. This reverts commit 494ebd7347ba52d702802fba4c3bb13e7bfbc2cf. This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a. This reverts commit 4eb77965cd016181d2111f37d93526e9bb0434f0. This reverts commit 637df95de63196033a6da4a6e286e1d58ea517b6. This reverts commit e5f426cd54609e7f05f8241d845e6e36c5f10d9a. This reverts commit 8cd68b8102eeab1b525712097c1b2e9679c11896. This reverts commit 6cae125e20865c52d770b24278bb7ab8fde5bc0d. This reverts commit 7be88fbf48156b36b6daefb228e1258e7d48cae4. This reverts commit b3fd99b74a3f6f42a9afd1b19ee4c60e38e8e91a. This reverts commit f7ec6e7d9e0dc360d9fb41f3a1afd7bdba1475c7. This reverts commit 5640992d176a499204a0756b1677c9b1575b0a49. This reverts commit d26808dd854006bd26a2366c675428ce0737238c. This reverts commit f91624a5950e14ba9007eed9bfa1c828676d4745. This reverts commit 16d39a5490ce74c901c7a8dbb990c6e83c379207. This reverts commit 8a4c2969874c0b7d72003f2523883eba8a348e83. This reverts commit 49e1356ee3d5d72929c91f778b3a231726c1413c. This reverts commit d2279f471ba8f44d9f578e62b286897a338d8aa1. This reverts commit b50e39df578adc3f86c5efa16bee9035cfdab61b. This reverts commit 858d80bf0f9f656a03992794874081b806e49222. This reverts commit 435f04480728c5d982e1a63c1b2777784bf9cd26. This reverts commit f61462e1be36a51dbf571076afa8e1930cb182f4. This reverts commit a6f1a3abecf8e8fd3e1bff439a06b853df18f194. This reverts commit 84dc50e160a2ec6590813374b5a1e58b97f7a18d. This reverts commit ed635d32853ee0a3e5ec1078679b27e7844a4ac7. This reverts commit 7b62791e001d6a4f8897ed48b3232d7f8fe6aa48. This reverts commit 153194c7717d8016b0eb974c81b1baee7dc1917d. This reverts commit f44d729d4ccae61bc0cdd5774acb3233eb5f7c13. This reverts commit a265fbd397ae72b2d3ea4c9310591ff1d0f3e05c. This reverts commit b9fef1a7cdfcc128fa589a32160e6aa7ed8964d7. This reverts commit b0eb64ff7bf6bde42046e091f8bdea9b7aab5f04. This reverts commit f1795463bf503a6fca909d77f598f641f9349f56. This reverts commit 70cbb1a5e311f609b624e3fae1a1712db639c51e. This reverts commit 42bf0204635213e2c75188b19ee66dc7e7d8a35e. This reverts commit 379f2650cf875f50c59524147ec0e33cfd5ef60c. This reverts commit 7ff22d6da41cd5ca80db95c18b409aea38e49fcd. This reverts commit 5a0b652d36ae4b6d423498c1f2c82c97a49c6f75. This reverts commit 432a174bc192740ac7a0a755009f6099b8363ad9. This reverts commit b14f8a1baf6f500997ae4c1d6a6d72094ce14270, reversing changes made to e713855dca17a7605bae99ea8d71bc7f8657e4b8.
2021-12-07Sort internal changes in changelog v1.49.0rc1Sean Quah1-16/+16
2021-12-07Update CHANGES.mdSean Quah1-30/+26
2021-12-071.49.0rc1Sean Quah65-63/+87
2021-12-07Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah8-12/+124
`matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
2021-12-07Refactors in `_generate_sync_entry_for_rooms` (#11515)David Robertson3-46/+79
* Move sync_token up to the top * Pull out _get_ignored_users * Try to signpost the body of `_generate_sync_entry_for_rooms` * Pull out _calculate_user_changes Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-07Correctly register shutdown handler for presence workers (#11518)David Robertson2-1/+2
Fixes #11517
2021-12-07Fix `ModuleApi.looping_background_call` for non-async functions (#11524)Sean Quah2-3/+3
After #10847, `looping_background_call` would print an error in the logs every time a non-async function was called. Since the error would be caught and ignored immediately, there were no other side effects.
2021-12-07Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff5-37/+33
If, for some reason, we don't have the create event, we should still be able to purge a room.
2021-12-07Correctly ignore invites from ignored users (#11511)David Robertson2-5/+7
2021-12-07Fix the test breakage introduced by #11435 as a result of concurrent PRs ↵reivilibre2-1/+2
(#11522)
2021-12-06Stabilise support for MSC2918 refresh tokens as they have now been merged ↵reivilibre6-44/+115
into the Matrix specification. (#11435)
2021-12-06Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech15-65/+370
As a step towards allowing back-channel logout for OIDC.
2021-12-06Add admin API to get some information about federation status (#11407)Dirk Klimpel7-0/+783
2021-12-06Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke10-101/+169
Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregatio