b0VIM 7.3O*jjoeygnu~joey/src/git-annex/doc/design/assistant/syncing.mdwnutf-8 U3210#"! Utp1$3TadC1ml]\w. d  u 1  e B k j # S R  ?> :21H32g=D@`_ih## other considerationscomplete (necessary for tracking uploads).This status file can also be updated periodically to show amount of transferwhich will avoid races.Can use file locking on these status files to claim upload/download rights,(and other git-annex transfer processes) should write a status file to disk.For files being sent, there's no filesystem indication. So git-annex-shellthere's no good way to find the pid (and I'd rather not kill blindly).For files being received, it can see the temp file, but other than lsofapp manage transfers as user desires. downloading it from us, or vice versa, as well as to in future let the webThis is important to avoid uploading content to a remote that is alreadysending (triggered by another remote), so it can add data for those too.The assistant needs to find out when `git-annex-shell` is receiving or -- add [Transfer] to DaemonStatus data Transfer = Transfer Direction Key TransferID EpochTime Integer data Direction = Uploading | Downloading data TransferID = TransferThread ThreadID | TransferProcess PidMay sometimes want multiple threads downloading, or uploading, or even both.Transfer threads started/stopped as necessary to move data. that lack content.* ToDownload added by the watcher thread when it seens new symlinks* ToUpload added by the watcher thread when it adds content. type ToTransferChan = TChan [ToTransfer] data ToTransfer = ToUploTTTTTTTTTTTT### transfer trackinganyway.get the basic functionality of the assistant to work. And we'll need thisreachable remote. This is worth doing first, since it's the simplest way toBefore mapping the network, the best we can do is flood all files out to every### first pass: flood syncingThis probably will need lots of refinements to get working well.and with appropriate rate limiting and control facilities.need to control those data transfers, sending to the cheapest nodes first,With the map, we can determine which nodes to push new content to. Then weexcept with a directed graph, so really a Arborescence problem.eventually, with the least cost. This is a minimum spanning tree problem,built, we want to find paths through the network that reach all nodesMapping the network can reuse code in `git annex map`. Once the map isdecide what to sync when.There are two parts to data syncing. First, map the network and second,## data syncing and do its own signaling.4. Add a hook, so when there's a change to sync, a program can be run signaling a change out of band.3. Investigate the XMPP approach like dvcs-autosync does, or other ways of so conflicts never break a sync. **done**2. Use a git merge driver that adds both conflicting files, and repush. **done**1. Also, detect if a push failed due to not being up-to-date, pull,1. Periodically retry pushes that failed. **done** (every half an hour) **done** any special mechanism to notify a remote that it's been synced to.] [The watching can be done with the existing inotify code! This avoids needing changes (sending them to any other remotes). that merges in received changes, and follow it by the part that pushes out another node via `git annex sync`), and run the part of `git annex sync`1. Watch `.git/refs/remotes/` for changes (which would be pushed in from out changes. **done**; changes are pushed out to all remotes in parallel When a change is committed, launch the part of `git annex sync` that pushes1. Can use `git annex sync`, which already handles bidirectional syncing.## git syncingall the other git clones, at both the git level and the key/value level.Once files are added (or removed or moved), need to send those changes toad)\)([[cloud]] needs to be used to bridge between LANs.This assumes the network is connected. It's often not, so thesyncing starts automatically. Use dbus on Linux?It would be nice if, when a USB drive is connected,ad 4 $Xut3  Y  | U T y , + y N M 5 4 ## other considerationscomplete (necessary for tracking uploads).This status file can also be updated periodically to show amount of transferwhich will avoid races.Can use file locking on these status files to claim upload/download rights,(and other git-annex transfer processes) should write a status file to disk.For files being sent, there's no filesystem indication. So git-annex-shellthere's no good way to find the pid (and I'd rather not kill blindly).For files being received, it can see the temp file, but other than lsofapp manage transfers as user desires. downloading it from us, or vice versa, as well as to in future let the webThis is important to avoid uploading content to a remote that is alreadysending (triggered by another remote), so it can add data for those too.The assistant needs to find out when `git-annex-shell` is receiving or -- add [Transfer] to DaemonStatus data Transfer = Transfer Direction Key TransferID EpochTime Integer data Direction = Uploading | Downloading data TransferID = TransferThread ThreadID | TransferProcess PidMay sometimes want multiple threads downloading, or uploading, or even both.Transfer threads started/stopped as necessary to move data. that lack content.* ToDownload added by the watcher thread when it seens new symlinks* ToUpload added by the watcher thread when it adds content. type ToTransferChan = TChan [ToTransfer] data ToTransfer = ToUpload Key Remote | ToDownload Key Remote