Introduction

Have you used software like Google Docs, Etherpad or Hedgedoc? It's a very direct way to collaborate – multiple people can type into a document at the same time, and you can see each other's cursors.

Ethersync enables a workflow like that, but for local text files, using your favorite text editors like Neovim or VS Code!

⚠️ Warning:

Ethersync is still in active development. At this point in time it is usable (we use it every day!) but has a lot of subtle things to know about and things that you might expect to work that do not yet work. Consider it as a proof of concept, and make sure to have backups.

A main reason we have written a lot of documentation is to give you the ability to learn about this system and with that allow you to use it smoothly despite the caveats.

Current Features

  • 👥 Real-time collaborative text editing
  • 📍 See other people's cursors
  • 🗃️ Work on entire projects
  • 🛠️ Sync changes done by text editors and external tools
  • ✒️ Local-first: You always have full access, even offline
  • 🇳 Fully-featured Neovim plugin
  • 🧩 Simple protocol for writing new editor plugins
  • 🌐 Peer-to-peer connections, no need for a server
  • 🔒 Encrypted connections secured by a shared password

Planned features

  • 🪟 VS Code plugin
  • 🔄 Individual undo/redo (we probably won't work on this soon)

Documentation overview

The main part of this documentation is aimed at users:

  • Getting started shows you how to install Ethersync, and how to make your first steps in it.
  • Concepts goes into the fundamentals of how Ethersync operates, which is important for using it effectively.
  • Features explains various things you can do with it (and some thing you can't do yet).
  • Ethersync in practice contains detailed advice for how to use Ethersync for certain workflows.
  • Related projects lists other software which have attempted to build similar systems.

There is also a section aimed at people who want to help improving Ethersync:

  • Writing new editor plugins specifies the protocol we use for communicating between the daemon and editors, and lists other things a plugin needs to do.
  • What to learn from us might be interesting to you if you want to build new software like this, especially if you find this after Ethersync has died. 💀

Getting started

To get started you might first want to install Ethersync on your system. You'll need a daemon and an Editor plugin, like Neovim. Then you can go ahead and try the first steps.

Installation

Every user of Ethersync needs to install two components:

  • The Ethersync daemon runs on your local machine, and is responsible for synchronization with other peers.
  • A plugin for your preferred text editor – currently, we offer a Neovim plugin. The plugin will connect to the daemon, send it what you type, and receive other peoples' changes.

Daemon

You might be able to use one of the following packages:

Arch Linux

Install the ethersync-bin package from the AUR.

Nix

💡 Tip

You can use the Nix package on any Linux or MacOS system!

This repository provides a Nix flake. You can put it in your PATH like this:

nix shell github:ethersync/ethersync

If you want to install it permanently, you probably know what your favorite approach is.

Binary releases

The releases on GitHub come with precompiled static binaries for Linux and macOS. Download one and put it somewhere in your shell's PATH, so that you can run it with ethersync.

Via Cargo

If you have a Rust installation, you can install Ethersync with cargo:

cargo install ethersync

Confirm the installation

To confirm that the installation worked, try running:

ethersync

This should show the available options.

Neovim Plugin

Again, we have several options of how to install the Neovim plugin:

Lazy

If you're using the Lazy plugin manager, you can use a configuration block like this:

{
  "ethersync/ethersync",
  config = function(plugin)
      -- Load the plugin from a subfolder:
      vim.opt.rtp:append(plugin.dir .. "/vim-plugin")
      require("lazy.core.loader").packadd(plugin.dir .. "/vim-plugin")
  end,
  keys = { { "<leader>j", "<cmd>EthersyncJumpToCursor<cr>" } },
  lazy = false,
}

Nix

For testing purposes, you can run an Ethersync-enabled Neovim like this:

nix run github:ethersync/ethersync#neovim

Manual installation

If you're not using a plugin manager, here's a "quick and dirty" way to install the plugin:

Clone the Ethersync repository:

git clone git@github.com:ethersync/ethersync

Link to the plugin directory from nvim:

mkdir -p $HOME/.local/share/nvim/site/pack/plugins/start
cd ethersync # make sure you're in the root of the project
ln -s $PWD/vim-plugin $HOME/.local/share/nvim/site/pack/plugins/start/ethersync

Confirm the installation

To confirm that the plugin is installed, try running the :EthersyncInfo command in Neovim. It should show a message like "Not connected to the Ethersync daemon".

VS Code plugin

We're offering two places from where you can install the plugin:

thus you should be able to install it independent whether you're running VS Code or VS Codium.

Confirm the installation

To confirm that the plugin is installed, try starting VS Code. It should show a message like "Connection to Ethersync daemon lost."

First steps

Here's how to try out Ethersync!

🖥 Try Ethersync on your own computer

1. Create an example project directory

Our current convention is to have a subdirectory called .ethersync in an Ethersync-enabled directory. So create them both:

mkdir -p playground/.ethersync
cd playground
touch file

2. Start the Ethersync daemon

ethersync daemon

You should see some log output indicating that things are initialized etc.

3. See changes across editors

Open the file in a new terminal:

nvim file

You should see Ethersync activated! in Neovim, and a Client connected message in the logs of the daemon.

💡 Tip

If that doesn't work, make sure that the ethersync command is in the PATH in the terminal where you run Neovim.

Next, in order to see Ethersync working, you can open the file again in a third terminal:

nvim file

The edits you make in one editor should now appear in both!

Note that using two editors is not the main use-case of Ethersync. We show it here for demonstrating purposes.

🧑‍🤝‍🧑 Invite other people

If a friend now wants to join the collaboration from another computer, they need to follow these steps:

1. Prepare the project directory

mkdir -p playground/.ethersync
cd playground

2. Exchange the information required to connect

Your friend will need to know two things:

  • When your daemon started, it printed a connection address ("multiaddress") like /ip4/192.168.23.42/tcp/58063/p2p/12D3KooWPNj7mom3X2D6NiSyxbFa5hHfzxDFP98ZL52yYnkEVmDv. If your friend is in the same local network, they can just use that address. If they're in another local network, see these instructions.
  • When your daemon started, it generated a secret passphrase, and printed it in the logs. Only people who know that passphrase are allowed to connect to it via the network.

In order to allow them to connect, we assume that you sent these two things to your friend (if you're not local, a secure channel is recommended).

3. Start the daemon

The command for joining another peer will look something like this:

ethersync daemon --peer <multiaddress> --secret <passphrase>

If a connection can be made, both sides will indicate success with a log message "Peer connected" and "Connected to peer" respectively. If you don't see it, double check the previous steps.

4. Start collaborating in real-time!

If everything worked, connected peers can now collaborate on existing files through opening them in their editors. Type somethings and the changes will be transferred over! If you're on nvim you should also see your peer's cursor.

Concepts

  • System overview provides you a deep dive into our architecture: how components are working together and why.
  • File ownership is a concept that we are using to decide which part of the system is allowed to make edits to a file.
  • Connection making explains how Ethersync daemons connect to each other.

System overview

Ethersync is a system for real-time local-first collaboration on text files, where

  • real-time means that edits and cursor movements should appear immediately while you are in a connection with your peer
  • local-first means that it's also possible to continue working on the project while you're (temporarily) offline
  • and the collaboration is restricted to text-files only (unfortunately, we don't support docx, xlsx, images, or other binary files yet).

Here's a diagram of the components that are involved:

  daemon <---(Internet)---> daemon
  ^    ^                    ^    ^
  |    |                    |    |
  v    |                    v    |
 file  |                   file  |
system |                  system |
   ^   |                      ^  |
   |   |                      |  |
   v   v                      v  v
   editor                    editor

Text editor

Text editors (with an installed Ethersync plugin) is what users most directly communicate with. If they make a change to a file, the editor instantly communicates every single character edit to the daemon.

The plugins also display other peoples' cursors in real-time.

Daemon

On each participant's computer, there's an Ethersync daemon, keeping the file's content in a data structure called "CRDT".

The daemon collects changes being communicated by the connected editor, and syncs them with other peers. If conflicts arise, because two edits happened at the same time, they will be resolved by the daemon automatically.

If the daemon is offline, it records the change locally and will communicate it to the other peers later.

The project

When collaborating with your peers, we assume that you are working on a set of files which are in a common directory. We call this directory the project. You can compare it, if you're familiar with that, with a git repository.

The tracking of, and communication about changes happens only inside the realm of that directory and whatever it contains recursively (which means it includes sub-directories and the files therein). Most files are synchronized, except for ignored files.

Currently, you will need to start one daemon per project. When you start the daemon, you have the option to provide the directory as an optional parameter:

ethersync daemon [OPTIONS] [DIRECTORY]

If you leave it out, the current directory is selected.

File ownership

Ethersync synchronizes edits immediately to each peer. Sometimes the peer has the file already open in an editor, sometimes not. In order to deal with different situations, we are using a concept called "ownership". Either the daemon or the editor can have it, it's like a token who is allowed to change the file.

Daemon has ownership

The daemon has ownership of a file if it is not open in an editor on that computer. In this case the daemon is allowed to write the changes some other connected daemon makes to a file directly to the disk.

Editor has ownership

However, once that file has been opened in an editor, that is undesirable – text editors are not happy if you change their files while they're running. So by opening a file in an editor with Ethersync plugin, that editor takes "ownership" of the file – the daemon will not write to them anymore. Instead, it will communicate changes to the editor plugin, which is then responsible for updating the editor buffer.

Once you close the file, the daemon will write the correct content to the file again. This means that, in an Ethersync-enabled directory, saving files manually is not required – you can do it if you want, but your edits will be communicated to your peers immediately anyway.

This is true as long as the daemon is running, so in case you're wrapping up your session, always make sure that you close all editors first and then the daemon, otherwise you might risk accidentally losing some of the edits to your buffer.

Opening the same file with multiple editors

As you have seen in the "first steps" section it is possible to open the same file in two editors and get the changes in both places. But keep in mind that our ownership detection is rather "basic" and does not have the ability to differentiate between those two editors. If you are opening the file with the second editor it's loading the content from disc, which might be out of date if you're not careful, leading to an out-of-sync state.

In short: We are not recommending this feature for the day to day use. It mainly exists to prevent crashes and allow debugging while testing.

Connection making

In order to make sense of how Ethersync daemons connect to each other a little bit of Networking background (IP addresses, TCP ports) is helpful. You should still be able to get going within one local network (such as two computers in the same Wi-Fi) by just copy pasting things, but connecting to other peers over the internet might currently require some configurations. We're aiming to give you the right keywords to look for in case you've not encountered that yet.

Multiaddress

Ethersync uses libp2p for making a connection. To identify another daemon we're using a connection address like /ip4/192.168.23.42/tcp/58063/p2p/12D3KooWPNj7mom3X2D6NiSyxbFa5hHfzxDFP98ZL52yYnkEVmDv. This is what libp2p calls a multiaddress – it contains your IP address, the TCP port, and a "peer ID" (which is used by connecting peers to make sure that they're actually connecting to the correct peer, and not to a "man in the middle").

Port

By default Ethersync selects a random private port, but in this case you're trying to set up port forwarding or a cloud peer, it's probably helpful to fix the port for the hosting peers. This can be done through the --peer option or the configuration file as explained here.

Peer to peer

If you want to connect across different local networks where each of you is behind a router. This way of connecting is more "ad hoc" and useful if you want to collaborate over a short period of time (as described in more detail in the pair programming scenario).

You need to enable port forwarding on your router. Specifically, the hosting peer needs to configure their router in such a way that it forwards incoming connections on the port you're using with Ethersync to their local machine. Also the port might be blocked by a network firewall.

Cloud peer

When you want to have an "always online" host, such that every user can connect to it at the time of their liking, let's say you're collaborating in a group on taking notes.

Other systems solve this with a client-server architecture, where the server is always online, and the clients connect to it as needed.

But Ethersync is fundamentally peer-to-peer, so what we suggest to use is what the research group Ink & Switch call a "cloud peer": You run an Ethersync peer on a public server, and all users will then connect to that server.

This is only recommended for people who are comfortable setting up services on a server. But the nice part is that if someone did this for you, you can just connect to it not worrying about the nitty-gritty networking details.

Local first

After you've initially synced with someone, your copy of the shared directory is fully independent from your peer. You can make changes to it, even when you don't have an Internet connection, and once you connect again, the daemons will sync in a more or less reasonable way. We can do this thanks to the magic of CRDTs and the Automerge library.

Behind the scenes

A CRDT is some kind of database that every peer maintains on their own, but when peers connect to each other, they will synchronize each other on their content.

The way Ethersync does this is by storing the CRDT in a file in your project. You can find it at .ethersync/doc. This file contains the edit history of all the files by each of the peers.

Sometimes it can make sense to delete this file to start with a clean state. This will especially become relevant in the case of switching to a different peer for example in a pair-programming setup.

Features

  • File events explains how Ethersync picks up changes to the filesystem, like adding or deleting files.
  • Offline support explains what is possible (or not) when you are going offline.
  • Configuration files shows you all the ways to configure Ethersync.
  • Workarounds are sometimes necessary and this section explains how to deal with them.

File Events

Ethersync tries to sync not only file changes done by supported editors, but also by external tools.

⚠️ Warning:

When one peer edits a file from an editor, and another peer changes it with an external tool at the same time, the latter change might get lost.
This is a restriction that seems hard to avoid. If you want to make sure changes by external tools are recorded correctly, do them while the daemon is not running, and make use of Ethersync's offline support.

Creating files

  • Opening a new file with an Ethersync-enabled text editor (this will create the file in the directory of connected peers).

    Example: nvim new_file

  • Creating a file directly on the file system.

    Example: touch new_file

  • Copying in a file from outside the project.

    Example: cp ../somewhere/else/file .

Changing files

  • Editing a file in an Ethersync-enabled text editor.

    Example: nvim existing_file

  • Changing files with external tools. Examples:

    • echo new stuff >> file
    • sort -o file file (sorting a file in place)
    • git restore file

Deleting files

  • Deleting a file directly from the file system.

    Example: rm new_file

  • Moving a file out of the project.

    Example: mv file ../somwhere/else/

Ignored files

Some files and directories are ignored by default. Also you have the option to specify files that should be ignored. Files that might contain sensitive information, like secrets, that should not be shared with your peers. Also Ethersync doesn't handle binary files, so maybe it makes sense to exclude them too.

Ethersync

  • ignores .git and everything in it.
  • ignores .ethersync and everything in it.
  • it respects everything that Git would ignore.

Offline Support

A core idea of Ethersync is that you can still work on a shared project, even when disconnected from your peers.

Ethersync uses a data structure called "Conflict-free replicated data type" (CRDT) to enable this, specifically, the Automerge library. The CRDT describes the current file contents, and the edits that were made to it, and allows smoothly syncing with other peers later.

Making changes while disconnected to peers

You can make changes to a project while disconnected from the Internet. If the daemon is running, the changes you make to files will already be put into the CRDT as you type them. If you then connect to other peers which worked on the same project, your changes will smoothly be integrated with theirs.

Making changes while the Ethersync daemon is not running

You can also make changes to a project while the Ethersync daemon is not running! When you start the daemon later, it will compare the file contents with its CRDT state, calculate a diff, and integrate the patches into its CRDT. This means that from Ethersync's perspective the files are the source of truth. After Ethersync has been restarted, its CRDT content will exactly match the file content.

Starting from scratch

Ethersync saves its CRDT state to .ethersync/doc. If you ever want to discard that state, you can delete that file. You might want to do this, for example, if you have previously paired on a project with person A, but now you want to join a shared session hosted by unrelated person B. Because B's document history has nothing to do with the one you currently have, syncing them will not work. So by deleting .ethersync/doc, you can "start from scratch", and join B.

What do you mean by "more or less reasonable" syncing?

The syncing will not always give 100% semantically correct results:

  • When two people create a file with the same name at the same time, one of the two copies will win, and the other one will be overwritten. The daemon's log will tell you which copy won. We're planning to give you more choices or make a backup.
  • When two people edit the same place of a source code, version control software like Git would show this as a "conflict", and ask you to resolve it manually. Ethersync, on the other hand, allows the changes to smoothly integrate. The result is like the combination of their insertions and deletions. So the result will not necessarily compile.

However, the syncing should always guarantee that all peers have the same directory content.

Configuration

There are two ways to configure what an Ethersync daemon will do.

Command line flags

You can provide the options on the command line, after ethersync daemon:

  • --peer <multiaddr> specifies which peer you want to try connect to.
  • --secret <passphrase> specifies the shared secret passphrase. Peers must use the same passphrase to be allowed to connect.
  • --port <port for your daemon> specifies which port the daemon should listen for incoming connections.

Configuration files

If you keep starting Ethersync with the same options, you can also put any of these options into a configuration file at .ethersync/config:

peer = <multiaddr you want to try connecting to>
secret = <the shared secret>
port = <port for your daemon>

Common pitfalls and workarounds

Some things about Ethersync are currently still a bit annoying. Let us show you how to work around them!

Sharing multiple projects requires configuring the socket

Ethersync currently only supports sharing a single project directory per daemon. If you want to sync more than one project, you can do so by starting a second daemon. The trick is to use a different socket for the editors to connect to.

  1. When starting the second daemon, use the --socket-path option, like this:

    ethersync daemon --socket-path /tmp/ethersync2

  2. Before opening a file in the second project directory, set the ETHERSYNC_SOCKET environment variable to the correct path, like this:

    export ETHERSYNC_SOCKET=/tmp/ethersync2

Restarting the daemon requires restarting the editor

The editor plugins currently only try to connect to Ethersync when they first start. If you need to restart the daemon for any reason, you will also need to restart all open editors to reconnect.

Editing a file with tools that don't have Ethersync support

We are planning to support this in a smoother way, but currently it's recommended to:

  • turn off the daemon
  • make your edits
  • start the daemon again.

It will then compare the "last seen" state with what you have on disk and synchronize your edits to other peers.

Ethersync in practice

Here we describe some use cases and workflows:

  • Pair programming is interesting if you're collaborating with someone on code on an ad-hoc basis.
  • Shared Notes describes a more long-term way of collaborating.
  • Working with Git describes some background and workflows in the context of using Git with Ethersync.

Using Ethersync for pair programming

One use case for Ethersync is to do a short collaboration session on an existing project. You could do this, for example, to pair program with another person, editing code together at the same time.

This would also work for more than two people. One person will start the session, and others can connect to it.

Step-by-step guide

1. Starting conditions

If both people already have a copy of the project, make sure you're on the same state – for example, by making sure that you're on the same commit with a clean working tree. If the joining peer has a different state, those changes will be overwritten.

An alternative is that the joining peer starts from scratch, with an empty directory.

Make sure you're both inside the project directory on the command line.

Also this guide assumes you're in the same local network. For other connections consider reading the section on connection making.

2. Create the .ethersync directory

This is our convention to mark a project as shareable: It needs to have a directory called .ethersync in it. So both peers should make sure that it exists:

mkdir .ethersync

Note that this directory, similar to a .git directory will not be synchronized.

3. First peer

To start the session, run:

ethersync daemon

This will print, among other initialization information, two things you need to tell the other peers:

  • The multiaddress which looks like /ip4/192.168.23.42/tcp/58063/p2p/12D3KooWPNj7mom3X2D6NiSyxbFa5hHfzxDFP98ZL52yYnkEVmDv.
  • A secret passphrase, that is randomly generated each time you start the daemon. If you want to use a stable secret, we recommend putting it into the configuration file.

4. Other peers

To join a session, run:

ethersync demon --peer <multiaddr> --secret <secret>

This should show you a message like "Connected to peer ...". The hosting daemon should show a message like "Peer connected".

If you prefer, it's also possible to use the configuration file to provide multiaddress and secret.

5. Collaborate!

Connected peers can now open files and edit them together. Note the common pitfalls.

6. Stop Ethersync

To stop collaborating, stop the daemon (by pressing Ctrl-C in its terminal). Both peers will still have the code they worked on, and can continue their work independently.

7. Reconnect later

If you later want to do another pairing session, make sure that you understand Ethersync's offline support feature and the local first concept. When you re-start Ethersync, it will scan for changes you've made in the meantime, and try to send them to the other peer. It is probably safest if you delete the CRDT state in .ethersync/doc as a joining peer. The hosting peer doesn't need to do that, it will simply update their state to the latest file content and share that with others.

Using Ethersync for writing shared notes

Another use case for Ethersync is to have a long-lasting collaboration session on a directory of text files (over the span of months or years). This is similar to how you would use Google Docs, Ethersync or Hedgedoc to work on text. It would be suited for groups who want to write notes or documentation together.

This use case is different from the "pair-programming" use case, because there, all peers are online at the same time. When you're working on a directory of notes for a longer time, it might happen that you make a change to a file, and then go offline, while the other peers are also offline. Still, you want other peers to be able to receive your changes.

We suggest to use a "cloud peer", a peer that is always online.

Step-by-step guide

You need to have access to a server on the Internet, and install the Ethersync daemon there.

1. Set up the directory

On the server, create a new directory for your shared project, as well as an .ethersync directory inside it:

mkdir my-project/.ethersync
cd my-project

2. Configure the daemon

You'll want to use a stable secret passphrase and a stable port on the server, so put those into the configuration file:

echo "secret=your-passphrase-here" >> .ethersync/config
echo "port=4242" >> .ethersync/config

3. Start the daemon

Launch the daemon in a way where it will keep running once you disconnect from your terminal session on the server. You could use screen, tmux, write a systemd service, or, in the easiest case, launch it with nohup:

nohup ethersync daemon &

4. Collaborate!

Other peers can now connect to the "cloud peer". It is most convenient for them to also use a configuration file like this:

echo "secret=your-passphrase-here" >> .ethersync/config
echo "peer=/ip4/<server ip>/tcp/<port>/p2p/<peerid>" >> .ethersync/config

Then, they can connect anytime using

ethersync daemon

Working with Git

While Ethersync currently doesn't have dedicated "Git integration" features, you can use it together with Git pretty well.

This section explains what possible workflows look like, and how Ethersync and Git concepts are interacting.

For the workflows below, we assume that you already have an established Git repository among the collaborating peers.

Ignoring .ethersync directories

In Ethersync-enabled projects, you will have a directory called .ethersync.

If you always want to ignore these directories, you can add it to your global .gitignore file like this:

mkdir -p ~/.config/git/
echo ".ethersync/" >> ~/.config/git/ignore

How Ethersync and Git interact

Ethersync tracks changes that you make to files in editors with an Ethersync plugin, and with external tools.

However, any change to the .git directory and the staging area (which is in fact also tracked in the .git repository) is ignored by Ethersync. This means that Ethersync does not sync

  • commits you create,
  • files that you stage or unstage, or
  • changes you're making to the HEAD.

This means that most Git operations you might try will not have an effect on connected peers.

Git commands that are safe to run

These commands will not lead to inconsistent behaviour (however, changes in your index or in your commits will not be shared with peers):

  • Checking what you have been doing so far with git diff / git status.
  • Use git add and the like to stage changes.
  • Use git commit to, well, create a commit in the current branch.

Git commands that are not safe to run

Because these commands might change file contents (without going through an editor), they might lead to different file contents, compared to your peers:

  • Synchronizing with a remote repository with git push and git fetch.
  • Use git switch/git checkout to switch to a different branch or get a specific file state from history.
  • Use git reset --soft or git reset --mixed to modify the staging area and the HEAD "manually".
  • Use git restore/git checkout -- <pathspec>/git reset --hard HEAD to undo your changes or get a different content of a file from the Git history.

If you need to run these commands while pairing, temporarily turn off Ethersync, make the change, and then reconnect. The daemon will then pick up changes, as described in the section about offline support.

When you start the daemon, make sure all peers are starting on the same commit with a clean staging area.

When you want to make a commit together, all peers should stop typing/editing files, then, one person should create the commit:

Committer

As a committer, create a commit like you usually would, and push it to a remote repository:

git push

Other peers

  1. Any other peer can then fetch the changes without applying them. Note: The changes are already applied to their working tree, through Ethersync.

    git fetch

  2. Now each peer can update the HEAD. The easiest option is to run:

    git reset @{u}

    What does this command do?

    • git reset will move your current branch to the given commit, and also set your index to the content of that commit. Notably, it does not touch the working directory (because it already contains exactly the content we want).
    • @{u} is an abbreviation for the upstream branch which the current branch is tracking. For example, it could mean origin/main (but it always refers to the correct upstream branch).

    As an effect, this command brings you to the same Git state like the committer.

All peers can then use git status/git diff to double check that they have the same diff now (if the commit contains all changes, the diff should be empty).

In the note taking use case, you can use Git for keeping your own local backup copy of the note's contents. You can then use it, to track which parts have been changed by others, for example while you were offline.

Let's say you have initially added and committed all notes.

  • Whenever you are reconnecting to the cloud peer and are getting some changes, you can revise them by looking at the git diff.
  • Then you can add and commit them with an unimportant commit message to set a "savepoint" for next time

It's also a nice little back-up in case anything goes wrong with the sync. Which might happen given that this is very new and bleeding edge software, be it through bugs or misunderstandings.

Related Projects

There have been a number of attempts of enabling collaborative text editing! If you think a project is missing, feel free to submit an issue or a PR to add them to the list!

Open-sourceActively developed1Peer-to-peerLocal-first2Editor-agnostic
Tandem✅❌✅✅✅ (Sublime, Neovim, Vim)
Open Collab Tools✅✅❌✅✅ (VS Code, Eclipse Theia)
crdt.el✅❌✅✅❌ (Emacs)
instant.nvim✅❌✅✅❌ (Neovim)
Teletype✅❌✅✅❌ (Atom)
Etherpad✅✅❌❌❌ (Web)
HedgeDoc✅✅❌✅3❌ (Web)
CryptPad✅✅❌❌❌ (Web)
Nextcloud Text✅✅❌✅❌ (Web)
Rustpad✅✅❌❌❌ (Web)
SubEthaEdit✅❌✅❌❌ (Standalone)
Gobby✅❌✅❌❌ (Standalone)
Floobits❌4❌❌❌✅ (Sublime, Atom, Neovim, Vim, IntelliJ, Emacs)
Google Docs❌✅❌❌❌ (Web)
Visual Studio Live Share❌✅❌❌❌ (Visual Studio, VS Code)
IntelliJ's Code With Me❌✅❌❌❌ (IntelliJ)
1

As of September 2024

2

This column indicates that the software uses CRDTs, we haven't checked for good offline support

3

Starting with the (upcoming) version 2.0

4

Open-source plugins, proprietary server

Editor plugin development guide

This document describes the protocol between the Ethersync daemon and the text editors. It should contain everything you need to implement a plugin for a new editor!

How to connect to the daemon

Ethersync consists of two parts: The daemon and editor plugins.

The daemon takes care of synchronization with other peers. An editor, on the other hand, has to communicate with the daemon to send it changes made to the file by the user, as well as changes of cursor positions. The daemon will send other people's changes and cursor positions back to the editor.

To make this as easy as possible for your plugin, we're using the same protocol as the Language Server Protocol: JSON-RPC. So if your editor plugin system allows connecting to an LSP, you'll hopefully can re-use the component opening a JSON-RPC connection.

The editor plugin will need to spawn the command ethersync client (which is our helper tool to connect to a running Ethersync daemon), and speak JSON-RPC (with Content-Length headers) with the standard input/output of that process. Think of ethersync client as the LSP Server when looking at it from the editor's perspective.

File ownership

Ethersync has the concept of file ownership. By default, the daemon has ownership, which means that, as connected peers make changes to files, it will write the changes directly to the disk.

But when an editor sends an "open" message, it takes ownership; all changes to the file by other sources will now be sent through the editor plugin. This is because text editor usually don't like it if you change to files they have opened.

When the last editor gives up ownership by sending a "close" message, the daemon takes ownership again.

Editor revision and daemon revision

For each open file, editors store two integers:

  • The editor revision describes how many changes the user has made to the file. It needs to be incremented after each edit made by the user.
  • The daemon revision describes how many changes the editor received from the daemon. It needs to be incremented after receiving an edit from the daemon.

How the editor recognizes Ethersync-enabled directories

Similar how Git repositories have a .git directory at the top level, Ethersync-enabled directories have an .ethersync directory at the top level. The editor must only send messages for files inside Ethersync-enabled directories.

The daemon-editor protocol

Here's the nitty-gritty details of what messages the daemon and the editor use to talk to each other.

Basic data types

The protocol uses a couple of basic data types (we're using the same syntax to specify them as the LSP specification:

  • DocumentUri = string

    This is an absolute file URI, for example: "file:///home/user/bla/fu.txt"`.

  • Position: {line: number, character: number}

    A position inside a text document. Characters are counted in Unicode characters (as opposed to UTF-8 or UTF-16 byte counts).

  • Range: {start: Position, end: Position}

    A range inside a text document. For cursor selections, the end is the part of the selection where the active/movable end of the selection is.

  • Delta: {range: Range, replacement: string}[]

    A complex text manipulation, similar to LSP's TextEdit[]. Like in LSP, all ranges refer to the starting content, and must never overlap, see the linked LSP documentation.

  • RevisionedDelta: {delta: Delta, revision: number}

    This attaches a revision number to a delta. The semantics are that the delta applies to (is intended for) that specified revision.

Messages sent by the editor to the daemon

These should be sent as JSON-RPC requests, so that the daemon can send back errors.

"open" {uri: DocumentUri}

  • Sent when the editor opens a document. The daemon will respond either with a success, or with an error describing why the file could not be opened (for example, because it is an ignored file, or if it's not part of the daemons shared project).
  • When an open succeeds, the editor gets ownership of the file, and the daemon will start sending updates for it as they come in.
  • The editor has to initialize its editor revision and daemon revision for that document to 0.

"close" {uri: DocumentUri}

  • Sent when the editor closes the file. It is no longer interested in receiving updates.

"edit" {uri: DocumentUri, delta: RevisionedDelta}

  • Sent when the user edits an open document, or when the text editor makes a change to a text buffer content for any reason.
  • Pitfall: Make sure to not send these messages when you incorporate a remote edit into the buffer content. You have to find a way to filter out the edits you cause as a plugin, for example, by setting a temporary ignore flag while the edit is being made, or by remembering which edits you perform, and checking against them when the editor notifies you of a buffer change.
  • The revision attribute of RevisionedDelta is the last revision seen from the daemon.
  • After each user edit, the editor must increase its editor revision.

"cursor" {uri: DocumentUri, ranges: Range[]}

  • Sends current cursor position/selection(s). Replaces the previous cursor ranges.

Messages sent by the daemon to the editor

These should be sent as notifications, there is no need to reply to them.

"edit" {uri: DocumentUri, delta: RevisionedDelta}

  • revision in the RevisionedDelta is the last revision the daemon has seen from the editor.
  • If this is not the editor revision stored in the editor, the editor must ignore the edit. The daemon will send an updated version later.
  • After applying the received edit, the editor must increase its daemon revision.

"cursor" {userid: integer, name?: string, uri: DocumentUri, ranges: Range[]}

  • The daemon sends this message when user's cursor positions or selections change, regardless of whether the file has been opened in the editor. The editor can use this information to display in which files other people work.

Tools to help you develop and debug a new plugin

Sending an example message to the daemon

To send messages to the daemon manually, you can try the following. Assuming you can start the daemon on a playground as described in the first steps, now we add some debugging output:

ethersync daemon playground -d
# Note for below: You will see some output like "Listening on UNIX socket: /tmp/ethersync"

You can then start the client, in another terminal:

ethersync client

This will already produce an output in the daemon which indicates that an Editor connected. This happens because the client connects to the /tmp/ethersync socket. Killing it shows the opposite: "Editor disconnected".

Next, you could manually send some JSON-RPC. We included a Python script to help you create messages in the correct format: Run it to see what an open message could look like:

python tools/dummy-jsonrpc.py playground/file

You can send it to the daemon like this:

python tools/dummy-jsonrpc.py playground/file | ethersync client

Seeing what an existing Ethersync plugin sends

On the other hand, not running any daemon, you can see what the plugin "wants" to communicate as follows. In the demon console (stop it), we now just plainly listen on the socket for incoming data:

# nc can only bind to existing sockets, so we'll drop potentially existing ones
rm /tmp/ethersync; nc -lk -U /tmp/ethersync

In the client console, start nvim on a file, move the cursor and edit something:

nvim playground/file

Starting two local daemons

For testing purposes, it can be useful to simulate having two peers connecting to each other.

Do do that on a single machine, follow these steps:

  1. Start one daemon regularly, we will call its directory the "first directory".
  2. Create a new, empty shared directory (with an .ethersync directory in it) for the second daemon.
  3. Start the second daemon:
    • The directory should be the additional directory you created.
    • Set the --peer option to the address of the first daemon.
    • Set --socket-path to a new value like /tmp/ethertwo.
  4. Connect an editor to the first daemon by opening a file in the first directory.
  5. Connect an editor to the second daemon by setting the environment variable ETHERSYNC_SOCKET to the new value before opening a file in the second directory.
    • Example: ETHERSYNC_SOCKET=/tmp/ethertwo nvim directory2/file

Things you type into the first editor should now appear in the second editor, and vice versa.

What to learn from us (after Ethersync is dead)

This page is written for people in the future, who discover Ethersync, and want to learn from it, to start their own project.

Maybe you don't agree with some of our technical choices. Or maybe you're unhappy with how we lead the project. But the most probable cause you're reading this is that we stopped to maintain and improve Ethersync, and the project is essentially dead. Looking at similar projects, many of them had the same fate.

Because of that, we want to give you as much useful information as we can, so that you can learn from our ideas and mistakes.

The documentation

This documentation already gives you many insights into how Ethersync works and behaves. The list of related projects might be helpful for you to discover other approaches.

The Architecural Decision Records

In addition to the documentation, we wrote a number of "architectural decision records", which each look at a certain problem we were facing, document the possible options, which one we picked, and why. We believe these records might be useful for future projects facing the same problems. You can find them here.

The editor plugins

Because of the not-invented-here syndrome, you're likely to code your own "core" for your project.

But one part of our existing code that could be most helpful to you, we think, is the code for the editor plugins. They speak quite a simple protocol, and maybe you can adapt them to your own needs, or look up how we solved certain problems (like Vim's infamous "implicit newline at the end of the file").

Open invitation: Ask us anything!

We care about the problem we're trying to solve, and we probably will care in the future. Even if we're not actively working on it anymore, you're welcome to get in touch and ask us anything. The best way to reach us is by email: Moritz Neeb <nt4u@kpvn.de>, blinry <mail@blinry.org>