From d518b05a8667943bd0aa9ab1edc91eec0a8283fe Mon Sep 17 00:00:00 2001 From: Eric Eastwood Date: Thu, 22 Jul 2021 05:58:24 -0500 Subject: Move dev/ docs to development/ (#10453) --- docs/SUMMARY.md | 6 +- docs/dev/cas.md | 64 --------------- docs/dev/git.md | 148 ---------------------------------- docs/dev/git/branches.jpg | Bin 72228 -> 0 bytes docs/dev/git/clean.png | Bin 110840 -> 0 bytes docs/dev/git/squash.png | Bin 29667 -> 0 bytes docs/dev/saml.md | 41 ---------- docs/development/cas.md | 64 +++++++++++++++ docs/development/git.md | 148 ++++++++++++++++++++++++++++++++++ docs/development/img/git/branches.jpg | Bin 0 -> 72228 bytes docs/development/img/git/clean.png | Bin 0 -> 110840 bytes docs/development/img/git/squash.png | Bin 0 -> 29667 bytes docs/development/saml.md | 41 ++++++++++ 13 files changed, 256 insertions(+), 256 deletions(-) delete mode 100644 docs/dev/cas.md delete mode 100644 docs/dev/git.md delete mode 100644 docs/dev/git/branches.jpg delete mode 100644 docs/dev/git/clean.png delete mode 100644 docs/dev/git/squash.png delete mode 100644 docs/dev/saml.md create mode 100644 docs/development/cas.md create mode 100644 docs/development/git.md create mode 100644 docs/development/img/git/branches.jpg create mode 100644 docs/development/img/git/clean.png create mode 100644 docs/development/img/git/squash.png create mode 100644 docs/development/saml.md (limited to 'docs') diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index db4ef1a44e..f1bde91420 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -67,7 +67,7 @@ # Development - [Contributing Guide](development/contributing_guide.md) - [Code Style](code_style.md) - - [Git Usage](dev/git.md) + - [Git Usage](development/git.md) - [Testing]() - [OpenTracing](opentracing.md) - [Database Schemas](development/database_schema.md) @@ -77,8 +77,8 @@ - [TCP Replication](tcp_replication.md) - [Internal Documentation](development/internal_documentation/README.md) - [Single Sign-On]() - - [SAML](dev/saml.md) - - [CAS](dev/cas.md) + - [SAML](development/saml.md) + - [CAS](development/cas.md) - [State Resolution]() - [The Auth Chain Difference Algorithm](auth_chain_difference_algorithm.md) - [Media Repository](media_repository.md) diff --git a/docs/dev/cas.md b/docs/dev/cas.md deleted file mode 100644 index 592b2d8d4f..0000000000 --- a/docs/dev/cas.md +++ /dev/null @@ -1,64 +0,0 @@ -# How to test CAS as a developer without a server - -The [django-mama-cas](https://github.com/jbittel/django-mama-cas) project is an -easy to run CAS implementation built on top of Django. - -## Prerequisites - -1. Create a new virtualenv: `python3 -m venv ` -2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate` -3. Install Django and django-mama-cas: - ``` - python -m pip install "django<3" "django-mama-cas==2.4.0" - ``` -4. Create a Django project in the current directory: - ``` - django-admin startproject cas_test . - ``` -5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas -6. Setup the SQLite database: `python manage.py migrate` -7. Create a user: - ``` - python manage.py createsuperuser - ``` - 1. Use whatever you want as the username and password. - 2. Leave the other fields blank. -8. Use the built-in Django test server to serve the CAS endpoints on port 8000: - ``` - python manage.py runserver - ``` - -You should now have a Django project configured to serve CAS authentication with -a single user created. - -## Configure Synapse (and Element) to use CAS - -1. Modify your `homeserver.yaml` to enable CAS and point it to your locally - running Django test server: - ```yaml - cas_config: - enabled: true - server_url: "http://localhost:8000" - service_url: "http://localhost:8081" - #displayname_attribute: name - #required_attributes: - # name: value - ``` -2. Restart Synapse. - -Note that the above configuration assumes the homeserver is running on port 8081 -and that the CAS server is on port 8000, both on localhost. - -## Testing the configuration - -Then in Element: - -1. Visit the login page with a Element pointing at your homeserver. -2. Click the Single Sign-On button. -3. Login using the credentials created with `createsuperuser`. -4. You should be logged in. - -If you want to repeat this process you'll need to manually logout first: - -1. http://localhost:8000/admin/ -2. Click "logout" in the top right. diff --git a/docs/dev/git.md b/docs/dev/git.md deleted file mode 100644 index 87950f07b2..0000000000 --- a/docs/dev/git.md +++ /dev/null @@ -1,148 +0,0 @@ -Some notes on how we use git -============================ - -On keeping the commit history clean ------------------------------------ - -In an ideal world, our git commit history would be a linear progression of -commits each of which contains a single change building on what came -before. Here, by way of an arbitrary example, is the top of `git log --graph -b2dba0607`: - -clean git graph - -Note how the commit comment explains clearly what is changing and why. Also -note the *absence* of merge commits, as well as the absence of commits called -things like (to pick a few culprits): -[“pep8”](https://github.com/matrix-org/synapse/commit/84691da6c), [“fix broken -test”](https://github.com/matrix-org/synapse/commit/474810d9d), -[“oops”](https://github.com/matrix-org/synapse/commit/c9d72e457), -[“typo”](https://github.com/matrix-org/synapse/commit/836358823), or [“Who's -the president?”](https://github.com/matrix-org/synapse/commit/707374d5d). - -There are a number of reasons why keeping a clean commit history is a good -thing: - - * From time to time, after a change lands, it turns out to be necessary to - revert it, or to backport it to a release branch. Those operations are - *much* easier when the change is contained in a single commit. - - * Similarly, it's much easier to answer questions like “is the fix for - `/publicRooms` on the release branch?” if that change consists of a single - commit. - - * Likewise: “what has changed on this branch in the last week?” is much - clearer without merges and “pep8” commits everywhere. - - * Sometimes we need to figure out where a bug got introduced, or some - behaviour changed. One way of doing that is with `git bisect`: pick an - arbitrary commit between the known good point and the known bad point, and - see how the code behaves. However, that strategy fails if the commit you - chose is the middle of someone's epic branch in which they broke the world - before putting it back together again. - -One counterargument is that it is sometimes useful to see how a PR evolved as -it went through review cycles. This is true, but that information is always -available via the GitHub UI (or via the little-known [refs/pull -namespace](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally)). - - -Of course, in reality, things are more complicated than that. We have release -branches as well as `develop` and `master`, and we deliberately merge changes -between them. Bugs often slip through and have to be fixed later. That's all -fine: this not a cast-iron rule which must be obeyed, but an ideal to aim -towards. - -Merges, squashes, rebases: wtf? -------------------------------- - -Ok, so that's what we'd like to achieve. How do we achieve it? - -The TL;DR is: when you come to merge a pull request, you *probably* want to -“squash and merge”: - -![squash and merge](git/squash.png). - -(This applies whether you are merging your own PR, or that of another -contributor.) - -“Squash and merge”[1](#f1) takes all of the changes in the -PR, and bundles them into a single commit. GitHub gives you the opportunity to -edit the commit message before you confirm, and normally you should do so, -because the default will be useless (again: `* woops typo` is not a useful -thing to keep in the historical record). - -The main problem with this approach comes when you have a series of pull -requests which build on top of one another: as soon as you squash-merge the -first PR, you'll end up with a stack of conflicts to resolve in all of the -others. In general, it's best to avoid this situation in the first place by -trying not to have multiple related PRs in flight at the same time. Still, -sometimes that's not possible and doing a regular merge is the lesser evil. - -Another occasion in which a regular merge makes more sense is a PR where you've -deliberately created a series of commits each of which makes sense in its own -right. For example: [a PR which gradually propagates a refactoring operation -through the codebase](https://github.com/matrix-org/synapse/pull/6837), or [a -PR which is the culmination of several other -PRs](https://github.com/matrix-org/synapse/pull/5987). In this case the ability -to figure out when a particular change/bug was introduced could be very useful. - -Ultimately: **this is not a hard-and-fast-rule**. If in doubt, ask yourself “do -each of the commits I am about to merge make sense in their own right”, but -remember that we're just doing our best to balance “keeping the commit history -clean” with other factors. - -Git branching model -------------------- - -A [lot](https://nvie.com/posts/a-successful-git-branching-model/) -[of](http://scottchacon.com/2011/08/31/github-flow.html) -[words](https://www.endoflineblog.com/gitflow-considered-harmful) have been -written in the past about git branching models (no really, [a -lot](https://martinfowler.com/articles/branching-patterns.html)). I tend to -think the whole thing is overblown. Fundamentally, it's not that -complicated. Here's how we do it. - -Let's start with a picture: - -![branching model](git/branches.jpg) - -It looks complicated, but it's really not. There's one basic rule: *anyone* is -free to merge from *any* more-stable branch to *any* less-stable branch at -*any* time[2](#f2). (The principle behind this is that if a -change is good enough for the more-stable branch, then it's also good enough go -put in a less-stable branch.) - -Meanwhile, merging (or squashing, as per the above) from a less-stable to a -more-stable branch is a deliberate action in which you want to publish a change -or a set of changes to (some subset of) the world: for example, this happens -when a PR is landed, or as part of our release process. - -So, what counts as a more- or less-stable branch? A little reflection will show -that our active branches are ordered thus, from more-stable to less-stable: - - * `master` (tracks our last release). - * `release-vX.Y` (the branch where we prepare the next release)[3](#f3). - * PR branches which are targeting the release. - * `develop` (our "mainline" branch containing our bleeding-edge). - * regular PR branches. - -The corollary is: if you have a bugfix that needs to land in both -`release-vX.Y` *and* `develop`, then you should base your PR on -`release-vX.Y`, get it merged there, and then merge from `release-vX.Y` to -`develop`. (If a fix lands in `develop` and we later need it in a -release-branch, we can of course cherry-pick it, but landing it in the release -branch first helps reduce the chance of annoying conflicts.) - ---- - -[1]: “Squash and merge” is GitHub's term for this -operation. Given that there is no merge involved, I'm not convinced it's the -most intuitive name. [^](#a1) - -[2]: Well, anyone with commit access.[^](#a2) - -[3]: Very, very occasionally (I think this has happened once in -the history of Synapse), we've had two releases in flight at once. Obviously, -`release-v1.2` is more-stable than `release-v1.3`. [^](#a3) diff --git a/docs/dev/git/branches.jpg b/docs/dev/git/branches.jpg deleted file mode 100644 index 715ecc8cd0..0000000000 Binary files a/docs/dev/git/branches.jpg and /dev/null differ diff --git a/docs/dev/git/clean.png b/docs/dev/git/clean.png deleted file mode 100644 index 3accd7ccef..0000000000 Binary files a/docs/dev/git/clean.png and /dev/null differ diff --git a/docs/dev/git/squash.png b/docs/dev/git/squash.png deleted file mode 100644 index 234caca3e4..0000000000 Binary files a/docs/dev/git/squash.png and /dev/null differ diff --git a/docs/dev/saml.md b/docs/dev/saml.md deleted file mode 100644 index a9bfd2dc05..0000000000 --- a/docs/dev/saml.md +++ /dev/null @@ -1,41 +0,0 @@ -# How to test SAML as a developer without a server - -https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great -resource for being able to tinker with the SAML options within Synapse without needing to -deploy and configure a complicated software stack. - -To make Synapse (and therefore Riot) use it: - -1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab. -2. Copy the XML to your clipboard. -3. On your Synapse server, create a new file `samling.xml` next to your `homeserver.yaml` with - the XML from step 2 as the contents. -4. Edit your `homeserver.yaml` to include: - ```yaml - saml2_config: - sp_config: - allow_unknown_attributes: true # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388 - metadata: - local: ["samling.xml"] - ``` -5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`: - ```yaml - public_baseurl: http://localhost:8080/ - ``` -6. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure - the dependencies are installed and ready to go. -7. Restart Synapse. - -Then in Riot: - -1. Visit the login page with a Riot pointing at your homeserver. -2. Click the Single Sign-On button. -3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`. - The response must also be signed. -4. Click "Next". -5. Click "Post Response" (change nothing). -6. You should be logged in. - -If you try and repeat this process, you may be automatically logged in using the information you -gave previously. To fix this, open your developer console (`F12` or `Ctrl+Shift+I`) while on the -samling page and clear the site data. In Chrome, this will be a button on the Application tab. diff --git a/docs/development/cas.md b/docs/development/cas.md new file mode 100644 index 0000000000..592b2d8d4f --- /dev/null +++ b/docs/development/cas.md @@ -0,0 +1,64 @@ +# How to test CAS as a developer without a server + +The [django-mama-cas](https://github.com/jbittel/django-mama-cas) project is an +easy to run CAS implementation built on top of Django. + +## Prerequisites + +1. Create a new virtualenv: `python3 -m venv ` +2. Activate your virtualenv: `source /path/to/your/virtualenv/bin/activate` +3. Install Django and django-mama-cas: + ``` + python -m pip install "django<3" "django-mama-cas==2.4.0" + ``` +4. Create a Django project in the current directory: + ``` + django-admin startproject cas_test . + ``` +5. Follow the [install directions](https://django-mama-cas.readthedocs.io/en/latest/installation.html#configuring) for django-mama-cas +6. Setup the SQLite database: `python manage.py migrate` +7. Create a user: + ``` + python manage.py createsuperuser + ``` + 1. Use whatever you want as the username and password. + 2. Leave the other fields blank. +8. Use the built-in Django test server to serve the CAS endpoints on port 8000: + ``` + python manage.py runserver + ``` + +You should now have a Django project configured to serve CAS authentication with +a single user created. + +## Configure Synapse (and Element) to use CAS + +1. Modify your `homeserver.yaml` to enable CAS and point it to your locally + running Django test server: + ```yaml + cas_config: + enabled: true + server_url: "http://localhost:8000" + service_url: "http://localhost:8081" + #displayname_attribute: name + #required_attributes: + # name: value + ``` +2. Restart Synapse. + +Note that the above configuration assumes the homeserver is running on port 8081 +and that the CAS server is on port 8000, both on localhost. + +## Testing the configuration + +Then in Element: + +1. Visit the login page with a Element pointing at your homeserver. +2. Click the Single Sign-On button. +3. Login using the credentials created with `createsuperuser`. +4. You should be logged in. + +If you want to repeat this process you'll need to manually logout first: + +1. http://localhost:8000/admin/ +2. Click "logout" in the top right. diff --git a/docs/development/git.md b/docs/development/git.md new file mode 100644 index 0000000000..9b1ed54b65 --- /dev/null +++ b/docs/development/git.md @@ -0,0 +1,148 @@ +Some notes on how we use git +============================ + +On keeping the commit history clean +----------------------------------- + +In an ideal world, our git commit history would be a linear progression of +commits each of which contains a single change building on what came +before. Here, by way of an arbitrary example, is the top of `git log --graph +b2dba0607`: + +clean git graph + +Note how the commit comment explains clearly what is changing and why. Also +note the *absence* of merge commits, as well as the absence of commits called +things like (to pick a few culprits): +[“pep8”](https://github.com/matrix-org/synapse/commit/84691da6c), [“fix broken +test”](https://github.com/matrix-org/synapse/commit/474810d9d), +[“oops”](https://github.com/matrix-org/synapse/commit/c9d72e457), +[“typo”](https://github.com/matrix-org/synapse/commit/836358823), or [“Who's +the president?”](https://github.com/matrix-org/synapse/commit/707374d5d). + +There are a number of reasons why keeping a clean commit history is a good +thing: + + * From time to time, after a change lands, it turns out to be necessary to + revert it, or to backport it to a release branch. Those operations are + *much* easier when the change is contained in a single commit. + + * Similarly, it's much easier to answer questions like “is the fix for + `/publicRooms` on the release branch?” if that change consists of a single + commit. + + * Likewise: “what has changed on this branch in the last week?” is much + clearer without merges and “pep8” commits everywhere. + + * Sometimes we need to figure out where a bug got introduced, or some + behaviour changed. One way of doing that is with `git bisect`: pick an + arbitrary commit between the known good point and the known bad point, and + see how the code behaves. However, that strategy fails if the commit you + chose is the middle of someone's epic branch in which they broke the world + before putting it back together again. + +One counterargument is that it is sometimes useful to see how a PR evolved as +it went through review cycles. This is true, but that information is always +available via the GitHub UI (or via the little-known [refs/pull +namespace](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally)). + + +Of course, in reality, things are more complicated than that. We have release +branches as well as `develop` and `master`, and we deliberately merge changes +between them. Bugs often slip through and have to be fixed later. That's all +fine: this not a cast-iron rule which must be obeyed, but an ideal to aim +towards. + +Merges, squashes, rebases: wtf? +------------------------------- + +Ok, so that's what we'd like to achieve. How do we achieve it? + +The TL;DR is: when you come to merge a pull request, you *probably* want to +“squash and merge”: + +![squash and merge](img/git/squash.png). + +(This applies whether you are merging your own PR, or that of another +contributor.) + +“Squash and merge”[1](#f1) takes all of the changes in the +PR, and bundles them into a single commit. GitHub gives you the opportunity to +edit the commit message before you confirm, and normally you should do so, +because the default will be useless (again: `* woops typo` is not a useful +thing to keep in the historical record). + +The main problem with this approach comes when you have a series of pull +requests which build on top of one another: as soon as you squash-merge the +first PR, you'll end up with a stack of conflicts to resolve in all of the +others. In general, it's best to avoid this situation in the first place by +trying not to have multiple related PRs in flight at the same time. Still, +sometimes that's not possible and doing a regular merge is the lesser evil. + +Another occasion in which a regular merge makes more sense is a PR where you've +deliberately created a series of commits each of which makes sense in its own +right. For example: [a PR which gradually propagates a refactoring operation +through the codebase](https://github.com/matrix-org/synapse/pull/6837), or [a +PR which is the culmination of several other +PRs](https://github.com/matrix-org/synapse/pull/5987). In this case the ability +to figure out when a particular change/bug was introduced could be very useful. + +Ultimately: **this is not a hard-and-fast-rule**. If in doubt, ask yourself “do +each of the commits I am about to merge make sense in their own right”, but +remember that we're just doing our best to balance “keeping the commit history +clean” with other factors. + +Git branching model +------------------- + +A [lot](https://nvie.com/posts/a-successful-git-branching-model/) +[of](http://scottchacon.com/2011/08/31/github-flow.html) +[words](https://www.endoflineblog.com/gitflow-considered-harmful) have been +written in the past about git branching models (no really, [a +lot](https://martinfowler.com/articles/branching-patterns.html)). I tend to +think the whole thing is overblown. Fundamentally, it's not that +complicated. Here's how we do it. + +Let's start with a picture: + +![branching model](img/git/branches.jpg) + +It looks complicated, but it's really not. There's one basic rule: *anyone* is +free to merge from *any* more-stable branch to *any* less-stable branch at +*any* time[2](#f2). (The principle behind this is that if a +change is good enough for the more-stable branch, then it's also good enough go +put in a less-stable branch.) + +Meanwhile, merging (or squashing, as per the above) from a less-stable to a +more-stable branch is a deliberate action in which you want to publish a change +or a set of changes to (some subset of) the world: for example, this happens +when a PR is landed, or as part of our release process. + +So, what counts as a more- or less-stable branch? A little reflection will show +that our active branches are ordered thus, from more-stable to less-stable: + + * `master` (tracks our last release). + * `release-vX.Y` (the branch where we prepare the next release)[3](#f3). + * PR branches which are targeting the release. + * `develop` (our "mainline" branch containing our bleeding-edge). + * regular PR branches. + +The corollary is: if you have a bugfix that needs to land in both +`release-vX.Y` *and* `develop`, then you should base your PR on +`release-vX.Y`, get it merged there, and then merge from `release-vX.Y` to +`develop`. (If a fix lands in `develop` and we later need it in a +release-branch, we can of course cherry-pick it, but landing it in the release +branch first helps reduce the chance of annoying conflicts.) + +--- + +[1]: “Squash and merge” is GitHub's term for this +operation. Given that there is no merge involved, I'm not convinced it's the +most intuitive name. [^](#a1) + +[2]: Well, anyone with commit access.[^](#a2) + +[3]: Very, very occasionally (I think this has happened once in +the history of Synapse), we've had two releases in flight at once. Obviously, +`release-v1.2` is more-stable than `release-v1.3`. [^](#a3) diff --git a/docs/development/img/git/branches.jpg b/docs/development/img/git/branches.jpg new file mode 100644 index 0000000000..715ecc8cd0 Binary files /dev/null and b/docs/development/img/git/branches.jpg differ diff --git a/docs/development/img/git/clean.png b/docs/development/img/git/clean.png new file mode 100644 index 0000000000..3accd7ccef Binary files /dev/null and b/docs/development/img/git/clean.png differ diff --git a/docs/development/img/git/squash.png b/docs/development/img/git/squash.png new file mode 100644 index 0000000000..234caca3e4 Binary files /dev/null and b/docs/development/img/git/squash.png differ diff --git a/docs/development/saml.md b/docs/development/saml.md new file mode 100644 index 0000000000..a9bfd2dc05 --- /dev/null +++ b/docs/development/saml.md @@ -0,0 +1,41 @@ +# How to test SAML as a developer without a server + +https://capriza.github.io/samling/samling.html (https://github.com/capriza/samling) is a great +resource for being able to tinker with the SAML options within Synapse without needing to +deploy and configure a complicated software stack. + +To make Synapse (and therefore Riot) use it: + +1. Use the samling.html URL above or deploy your own and visit the IdP Metadata tab. +2. Copy the XML to your clipboard. +3. On your Synapse server, create a new file `samling.xml` next to your `homeserver.yaml` with + the XML from step 2 as the contents. +4. Edit your `homeserver.yaml` to include: + ```yaml + saml2_config: + sp_config: + allow_unknown_attributes: true # Works around a bug with AVA Hashes: https://github.com/IdentityPython/pysaml2/issues/388 + metadata: + local: ["samling.xml"] + ``` +5. Ensure that your `homeserver.yaml` has a setting for `public_baseurl`: + ```yaml + public_baseurl: http://localhost:8080/ + ``` +6. Run `apt-get install xmlsec1` and `pip install --upgrade --force 'pysaml2>=4.5.0'` to ensure + the dependencies are installed and ready to go. +7. Restart Synapse. + +Then in Riot: + +1. Visit the login page with a Riot pointing at your homeserver. +2. Click the Single Sign-On button. +3. On the samling page, enter a Name Identifier and add a SAML Attribute for `uid=your_localpart`. + The response must also be signed. +4. Click "Next". +5. Click "Post Response" (change nothing). +6. You should be logged in. + +If you try and repeat this process, you may be automatically logged in using the information you +gave previously. To fix this, open your developer console (`F12` or `Ctrl+Shift+I`) while on the +samling page and clear the site data. In Chrome, this will be a button on the Application tab. -- cgit 1.5.1