BitTorrent Sync creates private, peer-to-peer Dropbox, no cloud required

No size limits, no cloud: Hands-on with BitTorrent's new file syncing software.

Read the whole story

 

[yiuin]

letao[/url]":1pum79st]I'll add a bit to the sentiment that this shouldn't be represented as an alternative to cloud-based services like DropBox. Another common use case for DropBox is accessing my files from anywhere, for example when I'm on vacation or at someone else's house. For BitTorrent Sync to be a viable replacement, I would have to leave my home computer on 24/7 and open it to the internet as a file server of sorts, which is way outside my normal usage pattern and is something I'd be hesitant to do.

Yes you would have to have at least one device on at all times. I have a work machine and a home server, but my home server has crappy bandwidth compared to my work machine, anyway this is an elegant solution because it doesn't matter which machine is running, as long as there is one.

 

[groghunter]

Hinton[/url]":2mp1ptg6]

RockDaMan[/url]":2mp1ptg6]Would this be this safe to install?

It killed my cat.

Speak for yourself, it brought my great-grandparents back from the dead.

They say hello.

 

[Dilbert]

nmalinoski[/url]":2jk6nojt]

Dilbert[/url]":2jk6nojt]So it's like turning all your files into thousands of chunks, and then copying those chunks to little USB thumb drives in a way that makes any one chunk appear on several drives. Then you divide all those USB thumb drives amongst all the people in your neighborhood.

Not quite. It starts out like that, but then all those USB flash drives get shared amongst everyone else who got a flash drive until each person has a full copy of the original set of files.

Impossible. That would stick everyone with everyone else's files. No one has that kind of local disk space.

This, used on the BT network as a whole, is free but unreliable storage.

This, used on a few private machines, is redundant because there are many other synch options available.

 

[kris2lee]

I hoped they have a mobile app too.

 

[wagnerrp]

Maldoror[/url]":2aewsk68]I think many people are missing the point. I share many work-related files with a large number of colleagues. Currently these are stored on a central server. We can't use (variants of) Dropbox for security reasons. If we were to use BTSync I see two immediate benefits: speed (the more users, the faster) and not having to rely on third parties for storage or the IT department for setting up a local cloud.

Edit: I realize that security with P2P is only as good as the least trustworthy peer...

What's wrong with just using rsync on that central server? Seriously, why have people gotten such a distaste of servers?

 

[miniBill]

What's the license? Because if it's not free software I'm never going to recommend this over git-annex

 

[bvz_1]

Scorp1us[/url]":3kguprig]I'm over on the forum asking the devs "Why? and how is this different from RSync" They don't have any really good answers.

They point out with a moderately sized cloud downloads would be faster as you get some combination of upstream bandwidth. But no one is talking about what happens when a file changes. How does BT handle that? Rsync would send delta blocks. Do you need a whole new file?

For those in micro-clouds of 2 or 3 stations, there isn't much benefit.

I have very little experience actually using rsync (and none using this new bittorrent sync) but one of the things I am looking for in a syncing client is encryption. I.e. my data synced from machine A to machine B but encrypted so that nobody between the machines could read it AND nobody on machine B can read it. Sort of using sync to do remote, encrypted backups.

rsync by itself kind of falls down here as far as my research has told me (though I am open to hearing otherwise) because once you want to encrypt on the remote side the "only sync those parts of the file that have changed" part of rsync stops working. Duplicity is one option to overcome this (and it is built on top of rsync).

I wonder whether this sync tool would allow that kind of a use case. If so, that would be a huge difference. I just haven't had time to dig into it enough yet to find out.


Edit: did a bit of searching and it looks like the encrypted remote folder is not a part of this (not saying that it even should be).

 

[Entegy]

Oh goody! I ended up shelling out for a year of Cubby for DirectSync. I still miss Windows Live Mesh. I will never understand why Microsoft killed that product.

Once my 1-year sub of Cubby is up, I'll be switching to this. Should be nice and mature since then. I'm not adverse to the cloud at all, there are just certain files that don't belong in the cloud or are too big (hello Sims 3 save folder that sits at 1.8GB. Yeah, sync that multiple times a day over the cloud!)

 

[doalwa]

Sounds kinda cool...perhaps a poor-mans DoubleTake...I'll give it a try for my porn collection!

 

[daneren2005]

albeec13[/url]":g21tkui0]So what's the benefit of this over, say, OwnCloud?

Well, if it actually works with large collections and doesn't die scanning files with odd characters, that would be one. We shall have to wait and see though.

 

[leexgxreal]

jbrodkin[/url]":4jt9kfg0]

Darkice1111[/url]":4jt9kfg0]From your screenshot it appears that download speed was 6.6 megabytes per second - 52.5 Mbps. And downloading on one pc meant uploading on the other. What's your actual upload speed limit?

On the other computer, I had an upload speed of 7MBps when I was transferring the large file. (I have a screenshot, although it's not in the story).

My ISP's upload limit is about 6Mbps, so I presume it was using the local network. My router does simultaneous dual-band, with 300 + 450 Mbps.

the presume part means your not understanding some stuff, so is your ISP upload speed 6mb/s , 60mb/s or 6MB/s? (last two are about the same and unlikely that you have that much upload speed)

most dual band routers are 2.4+5ghz and they are not bonded so you have 2 WiFi networks

if you was doing theses tests on the same network then your ISP has nothing to do with the test you have done (unless you have an 100mb upload speed) if all your tests was local and if both computers was on WiFi then 6MB/s (70mb/s ish ) is very good for WiFi

do the same tests on wired (wifi off) you find you be limited by cpu and disk speed (if 1gb network ports)

I find out once I get back I have an play with it

 

[tokyojerry]

I would like to clarify if I understand the function of this correctly. If I am working on a video project, say in Final Cut Pro or some other video post production package on my computer at home, and I have various files in a folder related to that project, and then want to have that project's folder content available to me on the road with my notebook to pick up on the project where I left off, will this utility serve the purpose to accommodate this objective? Conversely, edits and updates I have done while on the road, could that folder be synchronized back to the original folder of project files on the machine at home to have the latest version of the project files available to continue the project when I return home? If so, this can be quite useful. Currently, to accomplish this, I write all the project files to a USB stick. I use that while on the road. Then, when I return home, I copy the changed files on the USB stick back to the original location on the home computer.

 

[leexgxreal]

tokyojerry[/url]":28xh84ie]I would like to clarify if I understand the function of this correctly. If I am working on a video project, say in Final Cut Pro or some other video post production package on my computer at home, and I have various files in a folder related to that project, and then want to have that project's folder content available to me on the road with my notebook to pick up on the project where I left off, will this utility serve the purpose to accommodate this objective? Conversely, edits and updates I have done while on the road, could that folder be synchronized back to the original folder of project files on the machine at home to have the latest version of the project files available to continue the project when I return home? If so, this can be quite useful. Currently, to accomplish this, I write all the project files to a USB stick. I use that while on the road. Then, when I return home, I copy the changed files on the USB stick back to the original location on the home computer.

quick answer yes it syncs the folder you have selected (the newest version is always synced) , do note if your syncing large files it eat data very fast, so be careful what size of the files you are changing/editing when on mobile broadband

http://forum.bittorrent.com/topic/16410 ... -sync-faq/

your probably better using drop box as its not alpha and is less likely to blow up in your face, like delete all your files (you can recover older changed or deleted files with drop box)

 

[Plasdom]

^ Except if his video file is 120GB (not likely for him since he already uses a thumb drive but you get the idea). Also if he does not want traces of the video on any cloud service.

 

[_bk201]

So this is for LAN transfers only? I don't see how this could work outside your LAN, with just a 21 byte key.

 

[tokyojerry]

leexgxreal[/url]":1h25n03c]

tokyojerry[/url]":1h25n03c]I would like to clarify if I understand the function of this correctly. If I am working on a video project, say in Final Cut Pro or some other video post production package on my computer at home, and I have various files in a folder related to that project, and then want to have that project's folder content available to me on the road with my notebook to pick up on the project where I left off, will this utility serve the purpose to accommodate this objective? Conversely, edits and updates I have done while on the road, could that folder be synchronized back to the original folder of project files on the machine at home to have the latest version of the project files available to continue the project when I return home? If so, this can be quite useful. Currently, to accomplish this, I write all the project files to a USB stick. I use that while on the road. Then, when I return home, I copy the changed files on the USB stick back to the original location on the home computer.

quick answer yes it syncs the folder you have selected (the newest version is always synced) , do note if your syncing large files it eat data very fast, so be careful what size of the files you are changing/editing when on mobile broadband

http://forum.bittorrent.com/topic/16410 ... -sync-faq/

your probably better using drop box as its not alpha and is less likely to blow up in your face, like delete all your files (you can recover older changed or deleted files with drop box)

Thanks for that feedback and link.
I have used Dropbox, and that is one route to go if the video project sizes are not enormously large. (Usually personal video clips shot on camera so perhaps not over 4-5GB or so for any one project.) With a Dropbox solution, two caveats: (1) limited space (versus whatever one has in personal control) and (2) the trust factor. Not that Dropbox is untrustworthy, but, whenever personal data traverses vis-a-vis a 3rd party, it is a considerable factor. Anyway, it's exciting times as all these enabling technologies and opportunities emerge.

 

[A.W.]

_bk201[/url]":us78plqj]So this is for LAN transfers only? I don't see how this could work outside your LAN, with just a 21 byte key.

21 bytes is 168 bits, which happens to be the maximum key size for TripleDES. I'm assuming that's probably what they're using. TripleDES isn't as good as AES with 256-bit key, but is generally considered "good enough."

 

[kbhola]

Cool tech, good article.

However, the speed issue mentioned and paired with the screenshot needs to be clarified. It's clearly in MB/s (which is now states in the article) but you also say it's far below your network speed. What is your max network speed in your current configuration? It's kind of an important detail if you're going to be reviewing the product and calling it out on the speed of transfer but not giving us any information on why it's so low (for reference, I don't think ~6-7MB/s is bad for a wifi home network doing a comp-comp xfer).

 

[Fritzr]

Azdle[/url]":n7l2q4cs]Actually, on flooey's question, is it possible to sync the encrypted data with a server that doesn't have the key? That would be perfect for syncing to a server that I don't fully trust, such as a server that I can share with a few friends without sharing all my data with them.

From looking at the examples, you can create a "Public Share" folder & assign it a new secret key. Send the new key to the untrusted computer who will create a receiving folder using this key. When the transfer succeeds then you can delete the "Public Share" folder since the secret key for accessing it is compromised for any purpose other than additional transfers to/from the untrusted computer.

One of the examples in the article asks if you want to generate a new key or reused the existing key when adding an additional folder.

 

[bilditup1]

You know, I was wondering why this didn't exist yet. I had the idea to do something like this based on ftp, with some nice interface that hides all the technical crap from users.

But with BitTorrent, of course, you could potentially leverage the distribution of your data across multiple locations, making transfers quicker, and it could potentially have fewer limitations in general than an ftp-based solution. Good stuff.

 

[Fritzr]

wagnerrp[/url]":qksgcl44]

Maldoror[/url]":qksgcl44]I think many people are missing the point. I share many work-related files with a large number of colleagues. Currently these are stored on a central server. We can't use (variants of) Dropbox for security reasons. If we were to use BTSync I see two immediate benefits: speed (the more users, the faster) and not having to rely on third parties for storage or the IT department for setting up a local cloud.

Edit: I realize that security with P2P is only as good as the least trustworthy peer...

What's wrong with just using rsync on that central server? Seriously, why have people gotten such a distaste of servers?

When the Central Server is offline then all services dependent on it are offlline. With the BT version if 1 computer anywhere in the cloud has a copy of the piece you need, then it is online even if the Central Server is currently involved in an incident that will cause a few days to pass while a new one is installed and backups restored.

A micro-cloud might be the home office computer, your laptop, the wife's laptop, the kitchen computer and the kid's computer(s). It does not matter if most of them spend their time shut down, one of the others will serve the files. The ones that are offline will catch up with the changes the next time they connect and query the P2P private cloud.

 

[deet]

g011um[/url]":11lhuxlo]

albeec13[/url]":11lhuxlo]So what's the benefit of this over, say, OwnCloud?

It's peer-to-peer instead of client-server, so it doesn't require you to run an OwnCloud server somewhere. It syncs content directly between devices. Sounds like it would be great for real-time backup to another computer/NAS in your house, or sharing picture/video content with family members etc.

I'm constantly looking for ways to maintain redundant offsite backup and continuity.

Cross-platform, free, unlimited... Yes, please.

 

[deet]

Particularly as data sets grow beyond what's reasonable to backup with portable disks and tape, being able to keep live clones in two places is increasingly important, yet kind of a holy grail. This is a very promising approach.

 

[Chuck Knucka]

I've been using this for a few days now. It's definitely a cool application. You won't find too many free applications that can do what BTSync does. But it is not an rsync replacement. Rsync will be able to increase and decrease the file size if it changes. BTSync currently resyncs the entire file if the file size changes. Rsync is typically used more for pushing out diffs while BTSync is bidirectional. Rsync also requires some substantial research to use while BTSync is pretty much install-and-go.

So you could, for example, keep an entire VM disk in sync between a work and home computer provided you use fixed disks. If you use dynamic disks, BTSync will end up syncing the entire disk. For large files, this becomes an issue especially if you lack free space on the target. They do provide a trash folder to recover though.

It's a very powerful tool and the fact that it's free must sting some other companies.

 

[_bk201]

A.W.[/url]":22rb3sbh]

_bk201[/url]":22rb3sbh]So this is for LAN transfers only? I don't see how this could work outside your LAN, with just a 21 byte key.

21 bytes is 168 bits, which happens to be the maximum key size for TripleDES. I'm assuming that's probably what they're using. TripleDES isn't as good as AES with 256-bit key, but is generally considered "good enough."

I mean its a one way transfer, how could i send a file from my work network to my home network with just one peice of info(21 byte key), wouldn't you have to open ports on your firewall, give a destination IP and most people would be using a dynamic IP, dropbox works becuase you connect to dropbox, not the other way around

 

[Deleted member 174040]

Hinton[/url]":3enftp4r]

RockDaMan[/url]":3enftp4r]Would this be this safe to install?

It killed my cat.

Thank you for replying to my hidden post. It keeps me from having to repost it.

 

[pierrebuz]

Wow. These guys read my mind. I was just trying to figure out a good way to sync extremely large folders over the WAN. The thought of torrents did cross my mind but seemed to difficult. I was going to try to rig up a powershell script using BITS over an openvpn tunnel, but this is much better.

I should add that the real reason this is useful (for my problem) is resiliency. If my wan drops out on a 10gb file I want to be able to pickup where it left off. That and it really looks like this might essentially be set and forget style simplicity.

One question: does anyone know if I can start with two identical copies on each side of the WAN?

 

[Decker-Mage]

Nibblers[/url]"acp7a1t]Now we have a "legitimate" reason to demand that our internet providers do not block P2P traffic.

I say legitimate because until now if I have ever complained about this practice more people than I expected turn around and say: why do you need p2p? Isn't that just used by internet hackers to doenload illegal stuff? Sigh

I can definitely make a case here surrounded as I am with people who simply won't back things up while I'm sitting on a large (!) amount of file storage space and available bandwidth. They can easily grasp the pass-phrase concept now that they dimly understand WiFi requires it (well, here anyway). That is, of course, if Comcast doesn't six strike us to death for using BT-protocol for legitimate uses (mostly tailored distro's and VM's like backtrack, & al. currently).

Free, fairly reliable, modestly secure, might even get some cooperation around here. Yeah, it could grow on me.

 

[abcslayer]

Haha,
This is what I have waited for Bittorrent. I used Live Mesh, then looking for someone implement bittorrent protocol for direct syncing between devices. Now I test AeroFS & use Cubby. Well, I will try this new services when I have time. I think the app is still quite 'simple', I hope they can have something as simple & powerful like Cubby.

 

[sttot]

Reminds me of this peer to peer digital preservation storage prototype: http://naturalspublishing.com/files/pub ... 22n92v.pdf
It basically worked by you making available as much storage for others to use as you want to have duplicated in the cloud, the software would automatically manage backing up your data to others' computers on the portion they had allocated for others to use.

 

[Greho]

My first thought was Cubby as well. I think I will try BT side-by-side with it.

LogMeIn Backup also works this way, with direct P2P incremental backups, and no cloud.

 

[sporkwitch]

Darkice1111[/url]":71y9gx7r]

jbrodkin[/url]":71y9gx7r]

ars_technician[/url]":71y9gx7r]"which topped out around 7Mbps"

I think you made a unit error. The screenshot was MBps, not Mbps, so the speed should have been ~50+Mbps.

Yes, you're right, changed it to the big B. Embarrassing mistake, as I am usually the one explaining the difference between bits and bytes to people I know.

Not good enough. You are still writing that it's far below your network speed. So what is your upload speed? It's kind of an important detail as to actually know how good or bad is the performance of the client.

Minimum of 100Mb/s symetrical, but since he's a geek, most likely he has a gigabit router. "upload speed" isn't relevant here, because it's not going to the internet, he clearly stated that it was on his internal network.

This actually raises MY question, though, which is can it sync through NAT over the internet, or do I need to be on the same local network?

Even if it requires being on the local network, this shows a lot of potential for getting rid of everyone's favourite headache: SAMBA config. Seems like it would be a hell of a lot easier to simply install this on all my machines and just punch in the secrets, than fighting with SAMBA and Windows to play nice with password-protected sharing.

 

[Decker-Mage]

sporkwitch[/url]":2ap3xqog]
This actually raises MY question, though, which is can it sync through NAT over the internet, or do I need to be on the same local network?

Even if it requires being on the local network, this shows a lot of potential for getting rid of everyone's favourite headache: SAMBA config. Seems like it would be a hell of a lot easier to simply install this on all my machines and just punch in the secrets, than fighting with SAMBA and Windows to play nice with password-protected sharing.

Which is where I am. How do I cross NAT, especially when I do NOT have control of the routers/firewalls in question. This still has to meet the dead-easy criteria. ... still wading through the forum there.

Aside: sporkwitch? Talk about cognitive dissonance!

 

[sporkwitch]

Decker-Mage[/url]":30ep2kpm]

sporkwitch[/url]":30ep2kpm]
This actually raises MY question, though, which is can it sync through NAT over the internet, or do I need to be on the same local network?

Even if it requires being on the local network, this shows a lot of potential for getting rid of everyone's favourite headache: SAMBA config. Seems like it would be a hell of a lot easier to simply install this on all my machines and just punch in the secrets, than fighting with SAMBA and Windows to play nice with password-protected sharing.

Which is where I am. How do I cross NAT, especially when I do NOT have control of the routers/firewalls in question. This still has to meet the dead-easy criteria. ... still wading through the forum there.

Aside: sporkwitch? Talk about cognitive dissonance!

I would imagine it would function in some kind of similar way to magnet links, but I don't know much about them other than that they somehow dynamically match you up with peers without needing an actual torrent file first. Obviously these let you connect through NAT. I don't see how control over the routers would be particularly relevant, since it can either get through the NAT or it can't (if you have control you can open ports, but that defeats the point, since you would, presumably, have multiple machines running it, not a specific one you could set forwarding for.) If you don't have control then it doesn't much matter anyway: they could just as easily block it outright.

Either way, it's gotta be easier than making Windows play nice with SAMBA, never mind kludges like dynamicDNS (which are TOS violations on pretty much all american ISPs).

 

[Kazper]

So. Went and installed this on (so far) my desktop, my NAS (I love that they have an ARM version ready that "just works") and my laptop. So far it works very well at what it does keeping all three in sync at all times. Running it on a NAS or a 24/7 server is definitely still a plus, as you can count on at least that repository ALWAYS being up to date and ready to sync to other clients, yet if it should go offline for a while it wouldn't be a disaster as you could still do direct peering. I could see this as potentially huge for semi-large amount of computers (ideally global due to timezones), where you can always count on at least some of them being online, whether it's development, other projects or something else.

The ability to share a read-only key is also very nice.

It's clearly still an Alpha version - rough around the edges, and with limited settings/preferences (especially for the ARM version), but it shows promise. I'm curious how it will deal with the more restrictive firewall settings at work though - my guess is that it won't, which is still a problem, but one that is hard to solve.

So far I'm also thinking of using this in my family. Install it on my parent's computers and have them do auto-sync (backup) since I have plenty of space on my NAS. It would also give us all an easy way to share vacation photos and videos - things that usually exceed the size we get on something like Dropbox.

 

[harold31415]

A 21-byte key? Are they using Triple DES?

 

[Kazper]

harold31415[/url]":9cdxhstl]A 21-byte key? Are they using Triple DES?

Maybe, or maybe they are using 256bit AES and are just truncating the secret/key because they consider that safe "enough" and less intimidating for laypeople? Which I would tend to agree with unless you are sharing some state secrets or similar, which I don't think BTSync is meant for

But yes some more information on what encryption they are using would be nice. TripleDES sounds like an odd choice - being relatively slow without giving any security benefits in return. But I guess there could be some odd programmatical reason why they would use that if it's better/easier supported on some of their platforms. I'm still betting on 256bit AES until we hear otherwise, but then I don't use it for anything sensitive at this point

 

[onkeljonas]

Both SparkleShare and Git-annex do pretty much the same thing (both opensource).

 

[Kazper]

onkeljonas[/url]":2xfk6n10]Both SparkleShare and Git-annex do pretty much the same thing (both opensource).

Except they both (if I understand them correctly) require you to set up a server to act as a central hub for your peers. The point here is that no such central server is needed (although in a small enough cloud it'd probably be a benefit to have one).

 

[sporkwitch]

onkeljonas[/url]":3azn6xzm]Both SparkleShare and Git-annex do pretty much the same thing (both opensource).

Do they have clients for all three major platforms and what basically amounts to one-click setup? If not, they're not fulfilling the niche that this is targeting. (at work, so the links are blocked).