From 7c5dd6ffb85f208632f7a2018a922b5ef2083c18 Mon Sep 17 00:00:00 2001 From: anoadragon453 Date: Tue, 22 Mar 2022 16:30:53 +0000 Subject: deploy: 6b26536a52f77aa5573b4d2afedae448fac31b7c --- latest/development/cas.html | 2 +- latest/development/contributing_guide.html | 11 +- latest/development/database_schema.html | 8 +- latest/development/demo.html | 289 +++++++++++++++++++++ latest/development/experimental_features.html | 2 +- latest/development/git.html | 6 +- .../development/internal_documentation/index.html | 2 +- latest/development/releases.html | 2 +- latest/development/room-dag-concepts.html | 66 ++++- latest/development/saml.html | 2 +- latest/development/url_previews.html | 2 +- 11 files changed, 364 insertions(+), 28 deletions(-) create mode 100644 latest/development/demo.html (limited to 'latest/development') diff --git a/latest/development/cas.html b/latest/development/cas.html index c5a0d9973a..b7c71304a3 100644 --- a/latest/development/cas.html +++ b/latest/development/cas.html @@ -99,7 +99,7 @@ diff --git a/latest/development/contributing_guide.html b/latest/development/contributing_guide.html index f7420d9bd4..8476d01bab 100644 --- a/latest/development/contributing_guide.html +++ b/latest/development/contributing_guide.html @@ -99,7 +99,7 @@ @@ -512,6 +512,15 @@ accept anonymous contributions at this time.

Git allows you to add this signoff automatically when using the -s flag to git commit, which uses the name and email set in your user.name and user.email git configs.

Private Sign off

If you would like to provide your legal name privately to the Matrix.org +Foundation (instead of in a public commit or comment), you can do so +by emailing your legal name and a link to the pull request to +dco@matrix.org. +It helps to include "sign off" or similar in the subject line. You will then +be instructed further.

Once private sign off is complete, doing so for future contributions will not +be required.

10. Turn feedback into better code.

Once the Pull Request is opened, you will see a few things:

Any new boolean column must be added to the BOOLEAN_COLUMNS list in -scripts/synapse_port_db. This tells the port script to cast the integer -value from SQLite to a boolean before writing the value to the postgres -database.
+synapse/_scripts/synapse_port_db.py. This tells the port script to cast +the integer value from SQLite to a boolean before writing the value to the +postgres database.
Before SQLite 3.23, TRUE and FALSE were not recognised as constants by diff --git a/latest/development/demo.html b/latest/development/demo.html new file mode 100644 index 0000000000..7345070101 --- /dev/null +++ b/latest/development/demo.html @@ -0,0 +1,289 @@ + + + + + + Demo scripts - Synapse + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+ +
+ +
+ +
+
+
+ + +
+
+
+
+
+
+
+ + + +
+ +
Synapse
+ +
+ + + + + + + + + + + + + + + + +
+
+ + +
+
+ +
+
+
+
+
+
+
+ + + + + +
+
+ +
+
+
+ +
Synapse demo setup
+
DO NOT USE THESE DEMO SERVERS IN PRODUCTION
+
Requires you to have a Synapse development environment setup.
+
The demo setup allows running three federation Synapse servers, with server +names localhost:8080, localhost:8081, and localhost:8082.
+
You can access them via any Matrix client over HTTP at localhost:8080, +localhost:8081, and localhost:8082 or over HTTPS at localhost:8480, +localhost:8481, and localhost:8482.
+
To enable the servers to communicate, self-signed SSL certificates are generated +and the servers are configured in a highly insecure way, including:
+
+
Not checking certificates over federation.
+
Not verifying keys.
+
+
The servers are configured to store their data under demo/8080, demo/8081, and +demo/8082. This includes configuration, logs, SQLite databases, and media.
+
Note that when joining a public room on a different HS via "#foo:bar.net", then +you are (in the current impl) joining a room with room_id "foo". This means that +it won't work if your HS already has a room with that name.
+
Using the demo scripts
+
There's three main scripts with straightforward purposes:
+
+
start.sh will start the Synapse servers, generating any missing configuration. +
+
This accepts a single parameter --no-rate-limit to "disable" rate limits +(they actually still exist, but are very high).
+
+
+
stop.sh will stop the Synapse servers.
+
clean.sh will delete the configuration, databases, log files, etc.
+
+
To start a completely new set of servers, run:
+
./demo/stop.sh; ./demo/clean.sh && ./demo/start.sh +
+ +
+ +
+ + + + + + + + + + + + + +
+
+
+
+ +
+ + + + + + + + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/latest/development/experimental_features.html b/latest/development/experimental_features.html index 87e15ceb73..402e552d32 100644 --- a/latest/development/experimental_features.html +++ b/latest/development/experimental_features.html @@ -99,7 +99,7 @@
-
+
diff --git a/latest/development/git.html b/latest/development/git.html index 2960ab267a..e31c03e678 100644 --- a/latest/development/git.html +++ b/latest/development/git.html @@ -99,7 +99,7 @@
-
+
@@ -315,7 +315,7 @@ the history of Synapse), we've had two releases in flight at once. Obviously, - + @@ -333,7 +333,7 @@ the history of Synapse), we've had two releases in flight at once. Obviously, - + diff --git a/latest/development/internal_documentation/index.html b/latest/development/internal_documentation/index.html index d869fd373a..996acd6d62 100644 --- a/latest/development/internal_documentation/index.html +++ b/latest/development/internal_documentation/index.html @@ -99,7 +99,7 @@
-
+
diff --git a/latest/development/releases.html b/latest/development/releases.html index e73e6ba9a2..86041a1005 100644 --- a/latest/development/releases.html +++ b/latest/development/releases.html @@ -99,7 +99,7 @@
-
+
diff --git a/latest/development/room-dag-concepts.html b/latest/development/room-dag-concepts.html index 59cead08ab..183a250718 100644 --- a/latest/development/room-dag-concepts.html +++ b/latest/development/room-dag-concepts.html @@ -99,7 +99,7 @@
-
+
@@ -205,24 +205,62 @@ incrementing integer, but backfilled events start with stream_ordering=-1< rather than skipping any that arrived late; whereas if you're looking at a historical section of timeline (i.e. /messages), you want to see the best representation of the state of the room as others were seeing it at the time. +Outliers +We mark an event as an outlier when we haven't figured out the state for the +room at that point in the DAG yet. They are "floating" events that we haven't +yet correlated to the DAG. +Outliers typically arise when we fetch the auth chain or state for a given +event. When that happens, we just grab the events in the state/auth chain, +without calculating the state at those events, or backfilling their +prev_events. +So, typically, we won't have the prev_events of an outlier in the database, +(though it's entirely possible that we might have them for some other +reason). Other things that make outliers different from regular events: + + +We don't have state for them, so there should be no entry in +event_to_state_groups for an outlier. (In practice this isn't always +the case, though I'm not sure why: see https://github.com/matrix-org/synapse/issues/12201). + + +We don't record entries for them in the event_edges, +event_forward_extremeties or event_backward_extremities tables. + + +Since outliers are not tied into the DAG, they do not normally form part of the +timeline sent down to clients via /sync or /messages; however there is an +exception: +Out-of-band membership events +A special case of outlier events are some membership events for federated rooms +that we aren't full members of. For example: + +invites received over federation, before we join the room +rejections for said invites +knock events for rooms that we would like to join but have not yet joined. + +In all the above cases, we don't have the state for the room, which is why they +are treated as outliers. They are a bit special though, in that they are +proactively sent to clients via /sync. Forward extremity -Most-recent-in-time events in the DAG which are not referenced by any other events' prev_events yet. -The forward extremities of a room are used as the prev_events when the next event is sent. +Most-recent-in-time events in the DAG which are not referenced by any other +events' prev_events yet. (In this definition, outliers, rejected events, and +soft-failed events don't count.) +The forward extremities of a room (or at least, a subset of them, if there are +more than ten) are used as the prev_events when the next event is sent. +The "current state" of a room (ie: the state which would be used if we +generated a new event) is, therefore, the resolution of the room states +at each of the forward extremities. Backward extremity The current marker of where we have backfilled up to and will generally be the prev_events of the oldest-in-time events we have in the DAG. This gives a starting point when backfilling history. -When we persist a non-outlier event, we clear it as a backward extremity and set -all of its prev_events as the new backward extremities if they aren't already -persisted in the events table. -Outliers -We mark an event as an outlier when we haven't figured out the state for the -room at that point in the DAG yet. -We won't necessarily have the prev_events of an outlier in the database, -but it's entirely possible that we might. -For example, when we fetch the event auth chain or state for a given event, we -mark all of those claimed auth events as outliers because we haven't done the -state calculation ourself. +Note that, unlike forward extremities, we typically don't have any backward +extremity events themselves in the database - or, if we do, they will be "outliers" (see +above). Either way, we don't expect to have the room state at a backward extremity. +When we persist a non-outlier event, if it was previously a backward extremity, +we clear it as a backward extremity and set all of its prev_events as the new +backward extremities if they aren't already persisted as non-outliers. This +therefore keeps the backward extremities up-to-date. State groups For every non-outlier event we need to know the state at that event. Instead of storing the full state for each event in the DB (i.e. a event_id -> state diff --git a/latest/development/saml.html b/latest/development/saml.html index 49d9b4227b..218c3b550a 100644 --- a/latest/development/saml.html +++ b/latest/development/saml.html @@ -99,7 +99,7 @@ - Introduction Welcome and Overview Setup Installation Using Postgres Configuring a Reverse Proxy Configuring a Forward/Outbound Proxy Configuring a Turn Server Delegation Upgrading Upgrading between Synapse Versions Usage Federation Configuration Homeserver Sample Config File Logging Sample Config File Structured Logging Templates User Authentication Single-Sign On OpenID Connect SAML CAS SSO Mapping Providers Password Auth Providers JSON Web Tokens Refresh Tokens Registration Captcha Application Services Server Notices Consent Tracking URL Previews User Directory Message Retention Policies Pluggable Modules Writing a module Spam checker callbacks Third-party rules callbacks Presence router callbacks Account validity callbacks Password auth provider callbacks Background update controller callbacks Porting a legacy module to the new interface Workers Using synctl with Workers Systemd Administration Admin API Account Validity Background Updates Delete Group Event Reports Media Purge History Register Users Registration Tokens Manipulate Room Membership Rooms Server Notices Statistics Users Server Version Federation Manhole Monitoring Understanding Synapse Through Grafana Graphs Useful SQL for Admins Database Maintenance Tools State Groups Request log format Admin FAQ Scripts Development Contributing Guide Code Style Release Cycle Git Usage Testing OpenTracing Database Schemas Experimental features Synapse Architecture Log Contexts Replication TCP Replication Internal Documentation Single Sign-On SAML CAS Room DAG concepts State Resolution The Auth Chain Difference Algorithm Media Repository Room and User Statistics Scripts Other Dependency Deprecation Policy Running Synapse on a Single-Board Computer + Introduction Welcome and Overview Setup Installation Using Postgres Configuring a Reverse Proxy Configuring a Forward/Outbound Proxy Configuring a Turn Server Delegation Upgrading Upgrading between Synapse Versions Usage Federation Configuration Homeserver Sample Config File Logging Sample Config File Structured Logging Templates User Authentication Single-Sign On OpenID Connect SAML CAS SSO Mapping Providers Password Auth Providers JSON Web Tokens Refresh Tokens Registration Captcha Application Services Server Notices Consent Tracking URL Previews User Directory Message Retention Policies Pluggable Modules Writing a module Spam checker callbacks Third-party rules callbacks Presence router callbacks Account validity callbacks Password auth provider callbacks Background update controller callbacks Porting a legacy module to the new interface Workers Using synctl with Workers Systemd Administration Admin API Account Validity Background Updates Delete Group Event Reports Media Purge History Register Users Registration Tokens Manipulate Room Membership Rooms Server Notices Statistics Users Server Version Federation Manhole Monitoring Understanding Synapse Through Grafana Graphs Useful SQL for Admins Database Maintenance Tools State Groups Request log format Admin FAQ Scripts Development Contributing Guide Code Style Release Cycle Git Usage Testing Demo scripts OpenTracing Database Schemas Experimental features Synapse Architecture Log Contexts Replication TCP Replication Internal Documentation Single Sign-On SAML CAS Room DAG concepts State Resolution The Auth Chain Difference Algorithm Media Repository Room and User Statistics Scripts Other Dependency Deprecation Policy Running Synapse on a Single-Board Computer diff --git a/latest/development/url_previews.html b/latest/development/url_previews.html index e225fca096..85a72ebc72 100644 --- a/latest/development/url_previews.html +++ b/latest/development/url_previews.html @@ -99,7 +99,7 @@ - Introduction Welcome and Overview Setup Installation Using Postgres Configuring a Reverse Proxy Configuring a Forward/Outbound Proxy Configuring a Turn Server Delegation Upgrading Upgrading between Synapse Versions Usage Federation Configuration Homeserver Sample Config File Logging Sample Config File Structured Logging Templates User Authentication Single-Sign On OpenID Connect SAML CAS SSO Mapping Providers Password Auth Providers JSON Web Tokens Refresh Tokens Registration Captcha Application Services Server Notices Consent Tracking URL Previews User Directory Message Retention Policies Pluggable Modules Writing a module Spam checker callbacks Third-party rules callbacks Presence router callbacks Account validity callbacks Password auth provider callbacks Background update controller callbacks Porting a legacy module to the new interface Workers Using synctl with Workers Systemd Administration Admin API Account Validity Background Updates Delete Group Event Reports Media Purge History Register Users Registration Tokens Manipulate Room Membership Rooms Server Notices Statistics Users Server Version Federation Manhole Monitoring Understanding Synapse Through Grafana Graphs Useful SQL for Admins Database Maintenance Tools State Groups Request log format Admin FAQ Scripts Development Contributing Guide Code Style Release Cycle Git Usage Testing OpenTracing Database Schemas Experimental features Synapse Architecture Log Contexts Replication TCP Replication Internal Documentation Single Sign-On SAML CAS Room DAG concepts State Resolution The Auth Chain Difference Algorithm Media Repository Room and User Statistics Scripts Other Dependency Deprecation Policy Running Synapse on a Single-Board Computer + Introduction Welcome and Overview Setup Installation Using Postgres Configuring a Reverse Proxy Configuring a Forward/Outbound Proxy Configuring a Turn Server Delegation Upgrading Upgrading between Synapse Versions Usage Federation Configuration Homeserver Sample Config File Logging Sample Config File Structured Logging Templates User Authentication Single-Sign On OpenID Connect SAML CAS SSO Mapping Providers Password Auth Providers JSON Web Tokens Refresh Tokens Registration Captcha Application Services Server Notices Consent Tracking URL Previews User Directory Message Retention Policies Pluggable Modules Writing a module Spam checker callbacks Third-party rules callbacks Presence router callbacks Account validity callbacks Password auth provider callbacks Background update controller callbacks Porting a legacy module to the new interface Workers Using synctl with Workers Systemd Administration Admin API Account Validity Background Updates Delete Group Event Reports Media Purge History Register Users Registration Tokens Manipulate Room Membership Rooms Server Notices Statistics Users Server Version Federation Manhole Monitoring Understanding Synapse Through Grafana Graphs Useful SQL for Admins Database Maintenance Tools State Groups Request log format Admin FAQ Scripts Development Contributing Guide Code Style Release Cycle Git Usage Testing Demo scripts OpenTracing Database Schemas Experimental features Synapse Architecture Log Contexts Replication TCP Replication Internal Documentation Single Sign-On SAML CAS Room DAG concepts State Resolution The Auth Chain Difference Algorithm Media Repository Room and User Statistics Scripts Other Dependency Deprecation Policy Running Synapse on a Single-Board Computer -- cgit 1.5.1

Synapse