summary refs log tree commit diff
path: root/scripts-dev/complement.sh (follow)
Commit message (Collapse)AuthorAgeFilesLines
* complement: enable dirty runs (#16520)kegsay2023-10-271-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * complement: enable dirty runs * Add changelog * Set a low connpool limit when running in Complement Dirty runs can cause many containers to be running concurrently, which seems to easily exhaust resources on the host. The increased speedup from dirty runs also seems to use more db connections on workers, which are misconfigured currently to have `SUM(workers * cp_max) > max_connections`, causing ``` FATAL: sorry, too many clients already ``` which results in tests failing. * Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake * Debug logging * Only run flakey tests * Only adjust connection pool limits in worker mode * Move cp vars to somewhere where they get executed in CI * Move cp values back to where they actually work * Debug logging * Try p=1 to see if this makes worker mode happier * Remove debug logging
* Update complement.sh to match new public API shape (#16466)kegsay2023-10-121-2/+2
| | | | | | | | | | | | | * Update complement.sh to match new public API shape Sister PR to https://github.com/matrix-org/complement/pull/666 Context: https://github.com/matrix-org/complement/issues/654#issuecomment-1746613495 * Changelog * Pedantry * Run complement plz
* Support room version 11 (#15912)Patrick Cloke2023-07-181-1/+1
| | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* Unix Sockets for HTTP Replication (#15708)Jason Little2023-07-111-0/+4
| | | | | | | | | Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at. --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-4/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Log when events are (unexpectedly) filtered out of responses in tests (#14213)Eric Eastwood2023-06-011-0/+4
| | | | | See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492 This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
* Allow running Complement integration tests via podman (#15543)Andrew Morgan2023-05-051-12/+26
|
* Hack to rebuild the complement editable image (#15184)David Robertson2023-03-071-1/+11
| | | | | * Hack to rebuild the complement editable image * Changelog
* Allow enabling the asyncio reactor in complement (#14858)realtyem2023-02-011-0/+5
| | | Signed-off-by: Jason Little realtyem@gmail.com
* Enable Faster Remote Room Joins against worker-mode Synapse. (#14752)reivilibre2023-01-221-7/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Enable Complement tests for Faster Remote Room Joins on worker-mode * (dangerous) Add an override to allow Complement to use FRRJ under workers * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Fix race where we didn't send out replication notification * MORE HACKS * Fix get_un_partial_stated_rooms_token to take instance_name * Fix bad merge * Remove warning * Correctly advance un_partial_stated_room_stream * Fix merge * Add another notify_replication * Fixups * Create a separate ReplicationNotifier * Fix test * Fix portdb * Create a separate ReplicationNotifier * Fix test * Fix portdb * Fix presence test * Newsfile * Apply suggestions from code review * Update changelog.d/14752.misc Co-authored-by: Erik Johnston <erik@matrix.org> * lint Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Implement MSC3930: polls push rules (#14787)Andrew Morgan2023-01-191-1/+1
|
* Implement MSC3890: Remotely silence local notifications (#14775)Andrew Morgan2023-01-131-1/+1
|
* Add experimental support for MSC3391: deleting account data (#14714)Andrew Morgan2023-01-011-1/+1
|
* Add `--editable` flag to `complement.sh` which uses an editable install of ↵reivilibre2022-12-071-18/+82
| | | | | Synapse for faster turn-around times whilst developing iteratively. (#14548) Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* Move MSC3030 `/timestamp_to_event` endpoint to stable v1 location (#14471)Eric Eastwood2022-11-281-3/+3
| | | | | | | | Fix https://github.com/matrix-org/synapse/issues/14390 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` Complement test changes: https://github.com/matrix-org/complement/pull/559
* Allow override for requesting specific worker types for Complement on ↵realtyem2022-11-081-0/+3
| | | | | | | | | | | | | | | | | | | | | | | command line. (#14324) * Expose getting SYNAPSE_WORKER_TYPES from external, allowing override of workers requested. * Add WORKER_TYPES variable option to complement.sh script that passes requested workers into start_for_complement.sh entrypoint. * Update docs to reflect this new ability. * Changelog * Don't rely on soft wrapping to format long strings Good idea dklimpel. Thanks for catching that. Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> * Small nits just noticed in docs. * Fixup new line in docs. Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
* Enable testing MSC3874 in complement. (#14339)Patrick Cloke2022-11-011-1/+1
|
* Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke2022-10-121-2/+5
| | | Co-authored-by: Eric Eastwood <erice@element.io>
* Enable Complement CI tests in the 'latest deps' test run. (#13213)reivilibre2022-08-011-0/+1
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add --build-only option to complement.sh to prevent actually running ↵reivilibre2022-07-071-3/+18
| | | | Complement. (#13158)
* Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` ↵reivilibre2022-07-051-0/+12
| | | | environment when using `complement.sh`. (#13152)
* Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079)reivilibre2022-07-011-1/+6
|
* complement.sh: Permit skipping docker build (#13143)Richard van der Hoff2022-07-011-18/+54
| | | Add a `-f` argument to `complement.sh` to skip the docker build
* Collapse Docker build commands in Complement CI runs to make the logs easier ↵reivilibre2022-06-151-0/+15
| | | | to read. (#13058)
* Up complement time outs (#13048)Erik Johnston2022-06-141-3/+3
|
* Fix example of running complement.sh. (#12990)Patrick Cloke2022-06-091-1/+1
|
* Merge the Complement testing Docker images into a single, multi-purpose ↵reivilibre2022-06-081-10/+23
| | | | | image. (#12881) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fixes to MSC3787 implementation (#12858)David Robertson2022-05-241-1/+1
|
* Disable 'faster room join' Complement tests when testing against Synapse ↵reivilibre2022-05-231-1/+8
| | | | with workers. (#12842)
* Tweaks to workers-under-complement (#12637)Richard van der Hoff2022-05-091-2/+11
| | | | | | | * Bump the HS startup timeout * Log prefixes for more processes * Bump the overall timeout
* Await un-partial-stating after a partial-state join (#12399)Richard van der Hoff2022-04-211-1/+1
| | | | | | When we join a room via the faster-joins mechanism, we end up with "partial state" at some points on the event DAG. Many parts of the codebase need to wait for the full state to load. So, we implement a mechanism to keep track of which events have partial state, and wait for them to be fully-populated.
* Fix up healthcheck generation for workers docker image (#12405)Richard van der Hoff2022-04-111-1/+1
| | | This wasn't quite generating the right thing.
* Move complement setup stuff into the Synapse repo (#12404)Richard van der Hoff2022-04-111-12/+5
| | | | | | Fixes matrix-org/complement#330 (or it will, once we remove the old files). It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
* Remove references to "msc2403" (#12165)Andrew Morgan2022-03-301-1/+1
|
* Enable MSC2716 Complement tests in Synapse (#12145)Eric Eastwood2022-03-031-1/+1
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Enable MSC3030 Complement tests in Synapse (#12144)Eric Eastwood2022-03-031-1/+1
| | | | | The Complement tests for MSC3030 are now merged, https://github.com/matrix-org/complement/pull/178 Synapse implmentation: https://github.com/matrix-org/synapse/pull/9445
* Rename default branch of complement.sh to main (#12063)Nicolas Werner2022-02-231-2/+2
| | | | | | | The complement.sh script relies on the name of the ref matching the name of the unpacked folder. The branch redirect from renaming the default branch breaks that assumption. Signed-off-by: Nicolas Werner <n.werner@famedly.com>
* Allow overriding the complement ref. (#11766)Nicolas Werner2022-01-201-6/+8
| | | | | Updates complement.sh to read the ref from an environment variable (defaulting to master) when downloading a complement bundle for testing.
* Use buildkit's cache feature to speed up docker builds (#11691)Richard van der Hoff2022-01-121-0/+4
| | | Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
* Complement environment variable name change and update .gitignore. (#11718)Michael Kaye2022-01-101-1/+1
|
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-1/+1
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Return the stable `event` field from `/send_join` per MSC3083. (#11413)Patrick Cloke2021-11-291-1/+1
| | | | | This does not remove the unstable field and still parses both. Handling of the unstable field will need to be removed in the future.
* Remove msc2716 from the list of tests for complement. (#11389)Patrick Cloke2021-11-191-1/+1
| | | As the tests are currently failing and not run in CI.
* Fix running complement.sh script. (#11368)Patrick Cloke2021-11-171-3/+3
| | | | By reverting changes from #11166 in this script. Specifically commit 13f084eb58c379b2fc073680818d5931e94626f4.
* Fix Shellcheck SC2086: Quote to prevent splittingDan Callahan2021-10-221-3/+3
| | | | | | | | Double quote to prevent globbing and word splitting. https://github.com/koalaman/shellcheck/wiki/SC2086 Signed-off-by: Dan Callahan <danc@element.io>
* Ensure the base Docker image is rebuilt when running complement with ↵Patrick Cloke2021-08-251-7/+7
| | | | | | workers. (#10686) We now always rebuild the matrixdotorg/synapse image, then build the matrixdotorg/synapse-workers image on top of it.
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Use new go test running syntax for complement. (#10488)Patrick Cloke2021-07-271-1/+1
| | | Updates CI and the helper script t ensures all tests are run (in parallel).
* Disable msc2716 until Complement update is merged (#10463)Eric Eastwood2021-07-221-1/+1
|
* Send out invite rejections and knocks over federation (#10223)Richard van der Hoff2021-06-231-1/+1
| | | | | ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-1/+1
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Run complement with Synapse workers manually. (#10039)Patrick Cloke2021-05-241-3/+22
| | | | Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).
* Enable complement tests for MSC2946. (#9771)Patrick Cloke2021-04-091-1/+1
| | | By providing the additional build tag for `msc2946`.
* MSC3083: Check for space membership during a local join of restricted rooms. ↵Patrick Cloke2021-04-081-1/+1
| | | | | | | | (#9735) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Rewrite complement.sh (#9685)Andrew Morgan2021-03-311-11/+38
| | | | | | | This PR rewrites the original complement.sh script with a number of improvements: * We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch. * You can now specify a regex of test names to run, or just run all tests. * We now use the Synapse test blacklist tag (so all tests will pass).
* Add support for running Complement against the local checkout (#8317)Andrew Morgan2020-09-291-0/+22
This PR adds a script that: * Builds the local Synapse checkout using our existing `docker/Dockerfile` image. * Downloads [Complement](https://github.com/matrix-org/complement/)'s source code. * Builds the [Synapse.Dockerfile](https://github.com/matrix-org/complement/blob/master/dockerfiles/Synapse.Dockerfile) using the above dockerfile as a base. * Builds and runs Complement against it. This set up differs slightly from [that of the dendrite repo](https://github.com/matrix-org/dendrite/blob/master/build/scripts/complement.sh) (`complement.sh`, `Complement.Dockerfile`), which instead stores a separate, but slightly modified, dockerfile in Dendrite's repo rather than running the one stored in Complement's repo. That synapse equivalent to that dockerfile (`Synapse.Dockerfile`) in Complement's repo is just based on top of `matrixdotorg/synapse:latest`, which we opt to build here locally. Thus copying over the files from Complement's repo wouldn't change any functionality, and would result in two instances of the same files. So just using the dockerfile in Complement's repo was decided upon instead.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-08 15:03:48 +0000