This is a long-running project that started as the proverbial need to scratch my own itch and then somehow evolved into a full-time job of the past 6 years.
I thought I'd do a Show HN for two reasons.
1. I obviously want to show my baby to those who haven't seen it.
Its primary purpose is to do very fast file replication. If you are familiar with "robocopy /mir" - same idea, but on steroids. Lots of them. It can be used for both mirroring and archiving backups, among other things.
It is light, very small and it is really quite fast. Half of the development time was sunk into the UI/UX design, so there's that too.
Existing version is a result of 5 years of a _very_ careful evolution, focusing more on perfecting existing features rather than adding new ones. Said No to more feature requests than I can remember. The blog captures some of that in a form of development screenshots, sketches and what not [1].
2. Secondly, I wanted to add an anecdotal data point that the desktop software development is still very much an option despite of all the nasty rumors. The demand for well-written Windows software is still there. The biggest takeaway has been that there is LOTS of people, on Windows, that recognize software quality as a feature. They acknowledge and compliment it, and they are actively looking for it.
That's the niche. If you are thinking of trying the Windows ISV path, I'd aim there.
--
By the way of introduction - I'm in my mid 40s. I've been a programmer for my entire life, mostly on the sysdev side of things - firewalls, network stacks, VPNs, etc. - which is one of the reasons I still like things to be as small and as fast as possible. I'm also the original author of Hamachi VPN, there's a chance you might've heard of it.
Any thoughts or comments on the program itself, would appreciate to hear them. If anyone has any questions, I'd be happy to answer them if I can as well.
Thanks for creating Hamachi. I used it for older games that didn't have internet play, but had LAN functionality, with friends online. I never knew it was a solo project before LogmeIn bought it (and made it buggy/unreliable). What a joyfully simple program that was!
Oh wow, I've used both Hamachi and Bvckup, but never realized they were made by the same developer. Two great pieces of software.
We used a couple of alternatives after LogMeIn bought Hamachi and retargeted it at enterprise users, but none of them were ever quite as good. And as games have continued to strip out LAN support and/or gotten better at working through NATs, it's been a while since I needed anything like it.
No longer using Bvckup (my photo library doesn't live on my Windows machine anymore) but it was a nice change of pace to run something where it felt like the developer actually thought about the interface. With most utilities on Windows that's really not the case.
Hamachi had its place back in mid-00's, but I agree that a need for zeroconf LAN bridging has gone down substantially since then.
I do think however that extracting its p2p tunneling bit into a separate service could be a good thing to do even now. Its NAT punching was server-driven, which was (and still is!) a superior way to do it. Better timing, tighter port prediction. Heck, it could even establish _TCP_ tunnels through the simultaneous open trick.
Hey apankrat, thanks for your software! I used it for personal projects and considered it when I was doing full time IT work.
I didn't end up deploying it because for Hyper-V, performance of the differential scans was really poor, and IIRC there were some issues with cluster shared volumes.
Have you considered adding Windows' resilient change tracking support to bvckup2? I think with RCT and hooking into the API for backup quiescence, you could make this an enterprise grade backup product for SOHO IT needs.
Do you remember the details? First time I hear of something like this.
Re: RCT - I read up on it a while back and the overall impression I got was that it was an application-level re-implementation of NTFS write change journaling. So filed it under 'exotics' and moved on.
Skimmed through the specs again now and I think that it's a good feature to have. Basically, RCT gives the program a change list for a VD for free IF it knows how to retrieve it. I made a note, will see what I can do. Thanks for the nudge.
The full scan of say, a production database server's VHD negatively impacted performance to an unacceptable degree. It's something that probably could have been fixed with a more powerful storage subsystem.
RCT not only gives the change list, but I believe it also ensures that on a cluster shared volume (CSV) there is an easy way to safely get the changed blocks. Normally a VSS operation on a cluster shared volume puts the drive into redirected mode, in which all IO has to go through the host that owns the CSV at that time. That also negatively affects performance, effectively limiting throughput to whatever the network speed is instead of the usually much faster SAS bus.
OK, thanks. The second part is something that I, not being a practicing sysadmin, have zero insight into. Very helpful.
Re: full scan issues - that's likely due to bvckup2 being too aggressive with the IO. This is something that's easy to change [1], though I realize it's not a very good excuse and it should not require manual tweaking to begin with.
Just wanted to say I originally heard about Bvckup 2 when some one posted a link to your little IO Monitoring Tool [1]. I love running it now and then, always felt it was the best little tool to use drive usage! I feel that tool almost deserves its own little dedicated page, else I have to keep searching through the blog to find it.
Thanks for making Bvckup, its one of those tools I never knew I needed till I got it.
Ah, yes. Any excuse to do some data vis is worth pursuing. It needs quite a bit of cleanup to be worthy of a separate site though. The UI still needs a magnifying glass on high-DPI monitors and apparently not everyone is OK with that.
I have a question, it might not have one concrete answer.
If my destination doesnt actually have a copy of the data, but only the metadata (onedrive sync with filespace freed up, nasuni with files purged from the cache) can bvckup write only the changed blocks, or for example, would the nasuni need to pull the file from the cloud master back into the cache before it can patch it with the delta? Maybe this is such a tech specific question it doesnt have an answer.
I am not familiar with nasuni, but the general idea is that once bvckup2 has the block hashes for the source file, it will write only modified blocks to the backup copy.
Block hashes are computed either when the file is first copied to destination (by bvckup2) OR when the exact copy already exists at destination and the backup job is set to do "delta pre-comp" [1].
If in your case the latter applies, you may in fact get away with setting up delta copying without pulling any data off the cloud storage.
Nice software. Indie Windows software very much resonates with me. One question. How do you deal with Microsoft's requirement to sign the software? Is it expensive to get past the certification?
You get a code signing cert from any CA that sells them and sign your executables yourself. Certs can be as cheap as $50 a year / $250 of extended validation ones. Stay away from Comodo though, they are staffed by predominantly incompetent idiots and their validation process is a circus and nightmare in equal parts.
If you get an EV cert, it gives you an instant pass through Microsoft's SmartScreen, meaning that the warning the Edge (or what it's called bow) shows upon downloading is less scary.
The whole thing is in C++, but more in C-with-classes style. Very little of meta programming outside of STL, flat class hierarchy, etc.
In terms of dependencies - just the Windows API and a bit of Native API, if available. No WMI, very little of COM (for shell and shadow copying). It does however use quite a few exotics from the Windows API, e.g. SetFileIoOverlappedRange, SetFileCompletionNotificationModes, etc.
It does tree-to-tree comparison for deducing exact changes, and it can also, optionally, use file system change notifications to schedule the actual runs.
I'm going to defer to you on this one, but my hunch is that NTFS being stupid and crusty is a limiting factor? In my experience, it's dog-slow (particularly on small files). Have you tested on ReFS and compared performance?
From what I've seen NTFS is a really well-made file system. There is indeed a fixed per-file overhead that becomes an issue when working through lots of small files, but NTFS handles multi-threaded access very efficiently, so that's a very easy workaround.
It's not NTFS that's slow, it's something in the Windows I/O subsystem common across file systems. And it's not that it's slow on small files (the I/O performance is fine), but that there's a large overhead for repeatedly opening and closing files, especially by paths. Apparently Linux gets around this with some sort of inode cache (hashtable?) whereas Windows doesn't implement this for whatever reasons. (Possibly the filter drive model used to support virus scanning etc. doesn't allow it, though I don't quite understand the details of why.) Not even sure Mac is as fast as Linux in this regard.
On the other hand we have, over the years, seen plenty of fights between application and Linux kernel devs over the degree to which the POSIX spec makes it OK for Linux to discard application data, so perhaps that speed comes at a cost.
I thought I'd do a Show HN for two reasons.
1. I obviously want to show my baby to those who haven't seen it.
Its primary purpose is to do very fast file replication. If you are familiar with "robocopy /mir" - same idea, but on steroids. Lots of them. It can be used for both mirroring and archiving backups, among other things.
It is light, very small and it is really quite fast. Half of the development time was sunk into the UI/UX design, so there's that too.
Existing version is a result of 5 years of a _very_ careful evolution, focusing more on perfecting existing features rather than adding new ones. Said No to more feature requests than I can remember. The blog captures some of that in a form of development screenshots, sketches and what not [1].
2. Secondly, I wanted to add an anecdotal data point that the desktop software development is still very much an option despite of all the nasty rumors. The demand for well-written Windows software is still there. The biggest takeaway has been that there is LOTS of people, on Windows, that recognize software quality as a feature. They acknowledge and compliment it, and they are actively looking for it.
That's the niche. If you are thinking of trying the Windows ISV path, I'd aim there.
By the way of introduction - I'm in my mid 40s. I've been a programmer for my entire life, mostly on the sysdev side of things - firewalls, network stacks, VPNs, etc. - which is one of the reasons I still like things to be as small and as fast as possible. I'm also the original author of Hamachi VPN, there's a chance you might've heard of it.Any thoughts or comments on the program itself, would appreciate to hear them. If anyone has any questions, I'd be happy to answer them if I can as well.
[1] https://bvckup2.com/wip