diff options
Diffstat (limited to 'docs/development')
-rw-r--r-- | docs/development/contributing_guide.md | 30 | ||||
-rw-r--r-- | docs/development/dependencies.md | 22 | ||||
-rw-r--r-- | docs/development/reviews.md | 41 | ||||
-rw-r--r-- | docs/development/url_previews.md | 61 |
4 files changed, 92 insertions, 62 deletions
diff --git a/docs/development/contributing_guide.md b/docs/development/contributing_guide.md index 2b3714df66..4e1df51164 100644 --- a/docs/development/contributing_guide.md +++ b/docs/development/contributing_guide.md @@ -62,6 +62,8 @@ pipx install poetry but see poetry's [installation instructions](https://python-poetry.org/docs/#installation) for other installation methods. +Synapse requires Poetry version 1.2.0 or later. + Next, open a terminal and install dependencies as follows: ```sh @@ -304,6 +306,29 @@ To run a specific test, you can specify the whole name structure: COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages/parallel/Historical_events_resolve_in_the_correct_order ``` +The above will run a monolithic (single-process) Synapse with SQLite as the database. For other configurations, try: + +- Passing `POSTGRES=1` as an environment variable to use the Postgres database instead. +- Passing `WORKERS=1` as an environment variable to use a workerised setup instead. This option implies the use of Postgres. + +To increase the log level for the tests, set `SYNAPSE_TEST_LOG_LEVEL`, e.g: +```sh +SYNAPSE_TEST_LOG_LEVEL=DEBUG COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestImportHistoricalMessages +``` + +### Prettier formatting with `gotestfmt` + +If you want to format the output of the tests the same way as it looks in CI, +install [gotestfmt](https://github.com/haveyoudebuggedit/gotestfmt). + +You can then use this incantation to format the tests appropriately: + +```sh +COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -json | gotestfmt -hide successful-tests +``` + +(Remove `-hide successful-tests` if you don't want to hide successful tests.) + ### Access database for homeserver after Complement test runs. @@ -328,7 +353,7 @@ To prepare a Pull Request, please: 3. `git push` your commit to your fork of Synapse; 4. on GitHub, [create the Pull Request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request); 5. add a [changelog entry](#changelog) and push it to your Pull Request; -6. for most contributors, that's all - however, if you are a member of the organization `matrix-org`, on GitHub, please request a review from `matrix.org / Synapse Core`. +6. that's it for now, a non-draft pull request will automatically request review from the team; 7. if you need to update your PR, please avoid rebasing and just add new commits to your branch. @@ -504,10 +529,13 @@ From this point, you should: 1. Look at the results of the CI pipeline. - If there is any error, fix the error. 2. If a developer has requested changes, make these changes and let us know if it is ready for a developer to review again. + - A pull request is a conversation, if you disagree with the suggestions, please respond and discuss it. 3. Create a new commit with the changes. - Please do NOT overwrite the history. New commits make the reviewer's life easier. - Push this commits to your Pull Request. 4. Back to 1. +5. Once the pull request is ready for review again please re-request review from whichever developer did your initial + review (or leave a comment in the pull request that you believe all required changes have been done). Once both the CI and the developers are happy, the patch will be merged into Synapse and released shortly! diff --git a/docs/development/dependencies.md b/docs/development/dependencies.md index 8ef7d357d8..b356870f27 100644 --- a/docs/development/dependencies.md +++ b/docs/development/dependencies.md @@ -237,3 +237,25 @@ poetry run pip install build && poetry run python -m build because [`build`](https://github.com/pypa/build) is a standardish tool which doesn't require poetry. (It's what we use in CI too). However, you could try `poetry build` too. + + +# Troubleshooting + +## Check the version of poetry with `poetry --version`. + +The minimum version of poetry supported by Synapse is 1.2. + +It can also be useful to check the version of `poetry-core` in use. If you've +installed `poetry` with `pipx`, try `pipx runpip poetry list | grep +poetry-core`. + +## Clear caches: `poetry cache clear --all pypi`. + +Poetry caches a bunch of information about packages that isn't readily available +from PyPI. (This is what makes poetry seem slow when doing the first +`poetry install`.) Try `poetry cache list` and `poetry cache clear --all +<name of cache>` to see if that fixes things. + +## Try `--verbose` or `--dry-run` arguments. + +Sometimes useful to see what poetry's internal logic is. diff --git a/docs/development/reviews.md b/docs/development/reviews.md new file mode 100644 index 0000000000..d0379949cb --- /dev/null +++ b/docs/development/reviews.md @@ -0,0 +1,41 @@ +Some notes on how we do reviews +=============================== + +The Synapse team works off a shared review queue -- any new pull requests for +Synapse (or related projects) has a review requested from the entire team. Team +members should process this queue using the following rules: + +* Any high urgency pull requests (e.g. fixes for broken continuous integration + or fixes for release blockers); +* Follow-up reviews for pull requests which have previously received reviews; +* Any remaining pull requests. + +For the latter two categories above, older pull requests should be prioritised. + +It is explicit that there is no priority given to pull requests from the team +(vs from the community). If a pull request requires a quick turn around, please +explicitly communicate this via [#synapse-dev:matrix.org](https://matrix.to/#/#synapse-dev:matrix.org) +or as a comment on the pull request. + +Once an initial review has been completed and the author has made additional changes, +follow-up reviews should go back to the same reviewer. This helps build a shared +context and conversation between author and reviewer. + +As a team we aim to keep the number of inflight pull requests to a minimum to ensure +that ongoing work is finished before starting new work. + +Performing a review +------------------- + +To communicate to the rest of the team the status of each pull request, team +members should do the following: + +* Assign themselves to the pull request (they should be left assigned to the + pull request until it is merged, closed, or are no longer the reviewer); +* Review the pull request by leaving comments, questions, and suggestions; +* Mark the pull request appropriately (as needing changes or accepted). + +If you are unsure about a particular part of the pull request (or are not confident +in your understanding of part of the code) then ask questions or request review +from the team again. When requesting review from the team be sure to leave a comment +with the rationale on why you're putting it back in the queue. diff --git a/docs/development/url_previews.md b/docs/development/url_previews.md deleted file mode 100644 index 154b9a5e12..0000000000 --- a/docs/development/url_previews.md +++ /dev/null @@ -1,61 +0,0 @@ -URL Previews -============ - -The `GET /_matrix/media/r0/preview_url` endpoint provides a generic preview API -for URLs which outputs [Open Graph](https://ogp.me/) responses (with some Matrix -specific additions). - -This does have trade-offs compared to other designs: - -* Pros: - * Simple and flexible; can be used by any clients at any point -* Cons: - * If each homeserver provides one of these independently, all the HSes in a - room may needlessly DoS the target URI - * The URL metadata must be stored somewhere, rather than just using Matrix - itself to store the media. - * Matrix cannot be used to distribute the metadata between homeservers. - -When Synapse is asked to preview a URL it does the following: - -1. Checks against a URL blacklist (defined as `url_preview_url_blacklist` in the - config). -2. Checks the in-memory cache by URLs and returns the result if it exists. (This - is also used to de-duplicate processing of multiple in-flight requests at once.) -3. Kicks off a background process to generate a preview: - 1. Checks the database cache by URL and timestamp and returns the result if it - has not expired and was successful (a 2xx return code). - 2. Checks if the URL matches an [oEmbed](https://oembed.com/) pattern. If it - does, update the URL to download. - 3. Downloads the URL and stores it into a file via the media storage provider - and saves the local media metadata. - 4. If the media is an image: - 1. Generates thumbnails. - 2. Generates an Open Graph response based on image properties. - 5. If the media is HTML: - 1. Decodes the HTML via the stored file. - 2. Generates an Open Graph response from the HTML. - 3. If a JSON oEmbed URL was found in the HTML via autodiscovery: - 1. Downloads the URL and stores it into a file via the media storage provider - and saves the local media metadata. - 2. Convert the oEmbed response to an Open Graph response. - 3. Override any Open Graph data from the HTML with data from oEmbed. - 4. If an image exists in the Open Graph response: - 1. Downloads the URL and stores it into a file via the media storage - provider and saves the local media metadata. - 2. Generates thumbnails. - 3. Updates the Open Graph response based on image properties. - 6. If the media is JSON and an oEmbed URL was found: - 1. Convert the oEmbed response to an Open Graph response. - 2. If a thumbnail or image is in the oEmbed response: - 1. Downloads the URL and stores it into a file via the media storage - provider and saves the local media metadata. - 2. Generates thumbnails. - 3. Updates the Open Graph response based on image properties. - 7. Stores the result in the database cache. -4. Returns the result. - -The in-memory cache expires after 1 hour. - -Expired entries in the database cache (and their associated media files) are -deleted every 10 seconds. The default expiration time is 1 hour from download. |