The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
TechInterview.org
CityDesk
FogBugz
Fog Creek Copilot


The Old Forum


Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

What backup scheme to recommend to professional friends?

In a world where people take their backup in their own hands only to find the home-brewed backup scheme to be not without flaw*, they might come and ask you and me, to give them some advice. What would you tell your doctor- lawyer- and dentist- friends? How would you accomodate the backup of a small workgroup or a particular power user? How would you help them achieve the primary goals of disaster recovery, security/secrecy, while making data readily available on the go? How would you handle secondary goals like backups that works cross-platform? Or folks who hoard a lot of data (perhaps someone like Francis Ford Coppola*, who may want to edit his own digital film stocks) out necessity? Discuss! :-)

* http://news.bbc.co.uk/1/hi/entertainment/7019644.stm
Li-fan Chen Send private email
Sunday, September 30, 2007
 
 
Backup is a personal thing.  I don't want to back up my music because it is all on my iPod.  My wife does, though, because she doesn't use a large capacity MP3 player.


When people ask about backup I usually give them a custom solution.  A clickable script that will back up their data, or one that is scheduled to run at an interval without their intervention.

For me, at least, the combination of rsync and a language like Ruby is the most powerful backup tool I know.
John Cromartie Send private email
Sunday, September 30, 2007
 
 
The only ideal backup scheme in my opinion is one based off tape backups + CD/DVD redundency. Flash Drives are great for small things, but too small and expensive for most people's needs. HDDs are big, but also not so cheap and have the tendency to just die without warning if it happens to be defective.

Unfortunately Tapes have fallen out of favor in the consumer market and a good product isn't readily available from what I've seen. Please correct me if I'm wrong.

I think CD/DVDs are good, but not for super-long term storage as I think there's a problem with them degrading over the years, so be sure to update those backups regularly.

Otherwise, it's a good idea to buy one of those small waterproof/fireproof safes and update it's contents regularly with your latest backups, and keep that hidden away somewhere that robbers won't find. Plus a more convienient on site backup next to your computer.

If your data is REALLY important to you, you could do some sort of bi-monthly backup storage in a bank vault. That would definitly provide some peace of mind :)
Jason M.
Sunday, September 30, 2007
 
 
Two points I'd like to make.  I realize they don't directly address the question of "what scheme to recommend," but ...:

First, if you asked your doctor friend for advice about your bad back, he'd tell you to schedule an appt on Monday and you would be charged for a doctor visit.

If you asked your lawyer friend for advice about your will, he would essentially tell you the same thing: my professional advice is not free.

You should not be giving professional advice to your friends for free, either (assuming that was your intent in asking this question).  Your response should be, "what does Tuesday morning at 10 look like for you?  I'll come to your office, discuss your needs and we'll set up a working solution for you."

Second, the backup needs of your "professional" friends are no different from that of your gravedigger or deli clerk friends.  You could argue that medical or legal data is more "important" than personal data, but just try and tell that to someone who just lost five years of digital photos of their grandkids.  If you have data in a computer system it is at risk unless it is backed up regularly, regardless of your job, skills or "profession."
Karl Perry Send private email
Sunday, September 30, 2007
 
 

Sunday, September 30, 2007
 
 
I don't like tapes since when you find they have failed, the company that made the tape drive has gone out of business or 'no longer supports' that model, which used some proprietary encryption and compression scheme that is not documented anywhere. Too many tape backup failures.

Best solution is cloning then syncing the drive. That's the way to go, plus put really important stuff on DVDs regularly.
Tony Chang
Sunday, September 30, 2007
 
 
I just set up a new computer for an affluet professional couple in the medical field. (Paid work.)

They already have a WD external drive sitting on their desk which they manage to fill with misnamed copies of their original files (I should add that they have no concept of simple operations like "Browse" buttons in common dialogs, and do things like try to hand enter complex file names letter by letter and wind up creating tons of empty duplicate files, etc. which they have no clue how to deal with. Argh!)

So I strongly recommended to them the last time I was there that they invest $50 or so in Acronis backup and start a program of periodically backing up to their underutilized external drive.

Their immediate response was "what about the software that the backup drive came with?" I assured them that 1) Acronis is the gold standard for reliable backup 2) I had successfully restored a completely trashed machine (partitions wiped out) from an Acronis backup and 3) the SW that comes with the external drives is always an inferior "pay extra to upgrade to the real commercial version" piece of dreck.

But they couldn't and wouldn't accept this. The bundled non compehensive shitware had them mesmerized. "Oooooh, we paid for something just like that already. Ooooooooh." They basically couldn't comprehend that there was any qualitative difference between thrown-in freeware and a piece of commercial software that is a scaled down enterprise solution. They also didn't seem to accept my statement of the solidity of the solution that I was presenting.

Smart professional people can be utterly clueless and can shut down on needed information.  I'm saying that with PCs, there is too much disinformation around that prevents a recommendation from having any real clarity.

You can lead a horse to water... !!!!

PS: I agree that if they aren't paying you, don't offer advice like this. I would simply say that I had good experience with product X and end it there.
Bored Bystander Send private email
Sunday, September 30, 2007
 
 
No external hard drive. I lost mine with all my projects :( because of stupid windows explorer which could not handle copying several files at the same time to an USB device.

CD/DVD for things that does not change.
Online or secondary drive for things that do change.
Copies of important things to best friends because you never know.
Victor N. Send private email
Sunday, September 30, 2007
 
 
*do not change*
Victor N. Send private email
Sunday, September 30, 2007
 
 
I can only suggest what I use at home: two 500GB external hard drives and Retrospect configured to back up my personal files every night. I rotate the external hard drives every couple of weeks or so, and keep the one not in use in a locked drawer at work. External hard drives are cheap now, relatively fast, and big enough for personal or small business use.
Charles E. Grant
Sunday, September 30, 2007
 
 
Offsite storage is also recommended. Even in the fireproof safes, tape/disk media may not survive a fire. I've found http://www.strongspace.com useful for offsite storage and reasonably priced.
James C Send private email
Sunday, September 30, 2007
 
 
Many/most fireproof safes are designed to have enough insulation to protect paper, but not necessarily more insulation than is required to protect paper, and it turns out that paper will tolerate somewhat higher temperatures than typical digital media.  You need to read the documentation for your fireproof safe carefully.
Joel N. Weber II Send private email
Sunday, September 30, 2007
 
 
"What would you tell your doctor- lawyer- and dentist- friends?"

My brother's a dentist and I set him up with a long USB cable he snakes behind his office desk with a thumb drive at the end. Plus I put together an autorun utility which copies his software's data directory every night at midnight.

I figure if a thief breaks in he'll probably rip out all the cables and take just the CPU, which is why the thumb drive is at the end of the cable which he'd probably leave. Then I told my brother to swap the thumb drive with a second one about once every couple weeks or a month, and keep it at home in case of office fire.
Mac
Monday, October 01, 2007
 
 
I use the home edition of acronis true image. (so, +1 to whoever mentioned that). It has both just file backup, or full hard drive image ability. And, it supports popular email settings for backsups also.

Further, dvd writing, and usb hard drives are also supported (even when booting from the boot disk – something that Ghost from Symantec was always weak at).

For daily backups, there is also incremental backups ability (very fast, since you don't backup everything again)..

I not used the pro version, but it is a great product for sure..and it the best $50 I spent…

I don’t have a off site backup, and I think after reading this…I shall adopt something…

Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
Albert D. Kallal Send private email
Monday, October 01, 2007
 
 
+1 for Acronis. I use two external USB drives and swap them every couple of weeks into a data quality fireproof safe.  I'm considering using a third drive and swapping into a nieghbor's garage just in case my house burns to the ground!



Bored,

Did you try explaining to them:
  - Acronis can back up in-use files. Most other software won't do that. (And copying an in-use file is like trying to clean the floor you're standing on.  That part of the floor is already "in use" and you have to work some magic to amake that happen.
- The quality of your healthcare varies. Not all doctors are of the same skill.

- How much is thier data worth to them? 


But, you're right, people often make irrational decisions about money. I've had people offer to DRIVE an hour to pick up one our CDs to save the $6.50 S & H.  All I can figure is that they LOVE driving because you'll pay that in gas. These are the same people who can't be bothered to spend 1 minute reading the instructions.  People take the path of least resistance.
Mr. Analogy Send private email
Monday, October 01, 2007
 
 
"The bundled non comprehensive shitware had them mesmerized."

Tell me about it.

I've (almost) lost count of the number of times I've removed Norton/McAffee from a machine and replaced it with AVG, only to go back a few weeks later and find that some "friend" had put it back again.

Once people get the name of a product in their head there's no shaking it - especially if the replacement is free/cheaper.
Jimmy Jones
Monday, October 01, 2007
 
 
That really depends on the "friend".  Where is the data located (server vs. Multiple PCs)?  If laptops are involved, do they typically take them home in the eve?
Are we just looking at data recovery or system recovery?
What is the data worth to them; i.e. are they out of business if the data is gone?  What is the max amount of  data they can afford to loose?
What timeframe do they have for recovery?
Can they be bothered with rotating tapes/CD's/Thumbdrives?
Do they bother with restore testing?

My most common 2 SMB solutions, both based on Acronis Workstation/Server.
1) External HD, ideally of a central location like a server.  I come in weekly to rotate drives/perform restore testing.
2) Set up a RAID 5 external HD solution like the Buffalo Terra station, no rotation.

That being said, if you are looking into longterm archiving, etc., nothing yet beats tapes on a cost basis.

Re. "Professionals" mesmerized by freeby crapware.  Ask them "So if I had a problem with (insert their speciality), would you be comfortable referring me to somebody who does this on a lark, for free?"
If they say yes, maybe you should re-evaluate their friendship.
Silent Wing Send private email
Monday, October 01, 2007
 
 
More considerations:

1) Hard drives these days are cheap because they fail. Frequently and hard. You must use an additional storage medium.

2) People who don't understand browsing also won't have a UPS on their systems. They also don't understand "Safely Remove Hardware" procedures. Chances of external devices getting fried are high.

3) Off-site storage. Copies of data must be stored elsewhere to prevent total loss from theft, fire and other natural disasters.

4) Backups must be tested.

5) On-line backups. Fills the roles of points 1, 2, and 3. See Amazon S3, www.rsync.net, and similar. Somehow you must pick a service which will not go out of business when you need them the most; which probably excludes the freebie services.
IT guy
Monday, October 01, 2007
 
 
Mr. Analogy - You're describing a subtlety that would be lost on them. They didn't even get the big picture of why to use a proper tool.

Jimmy Jones - your anecdote fits here. In fact, that's the whole problem with computer support work as a business. Everyone, all the time, is always a freakin' expert. You're always "fighting" with unseen parties who provide free and usually conflicting advice that directly undermines whatever you are trying to accomplish.

I also believe that tech-weenie end users who elect themselves "protectors" of their computer-illiterate relations and friends, just love to fight over this stuff, particularly with the overpriced "paid" guy.

One reason this couple was "fleeing" their old PC was because it was becoming incredibly slow. This is because they have several well-meaning "friends" including their own kids-in-law, who do "helpful" things like install Opera (I never got into asking them when they used IE and when they used Opera??), Google Desktop search engine, the fully locked down Zone Alarm, etc. As well as the full stack of free computer shit like Kodak's do-something-with-images task bar utility, and a whole bunch of other utilities that they allowed to be installed by default themselves. So you not only have risks and extra crap introduced by user error but also a lot of extra baggage introduced by unpaid "helpers".

I expect their new PC to become completely unusable again over the next 18 months as their computer whiz son in law insists on removing AVG, reinstalling Norton, installing Zone Alarm with its 15000 popups nagging you if running Outlook is OK, etc.

Argh!

I took the job on (the first paid "computer service" I have done in the past 2 years) as a favor. I certainly earned enough from the markup of their PC to be happy, and the hourly fees on top of that were gravy. I had decided going in that if I couldn't mark up their new PC for at least $500, the business just wasn't worth taking on. Well, I made that number... and more.

I do NOT recommend that business.

Just a slightly off topic aside to entertain everyone. :)
Bored Bystander Send private email
Monday, October 01, 2007
 
 
Well, not that I'm really adding much here to the good advice above, but...

How valuable is the data? If it's really that important - RAID 1 to start - I like mirroring.

Then software backup solutions as above...

Backups can be costly.

However, I just heard of some biotech thing that will allow removeable storage devices to have 50 TB of storage and may be commercially available in 18 months... Wow. 50 TB! THAT is some serious backup potential. Wonder what the cost will be though...
Ryan Smyth Send private email
Monday, October 01, 2007
 
 
About RAID - major PITA. I personally think that unless you set up RAID for a living, it is not a great option, until they make it easier to configure. I favor regular backups.

Anecdote:

I set up my new PC a couple of years ago with RAID mirroring.

Then one day I accidentally rebooted (my foot pushed the reset button or something like that.) When it restarted, BIOS complained that the drives were not in-sync. It did let me get back to Windows but one of the RAID HDDs was flagged as "ignored" - basically the RAID volume was then automatically broken at the BIOS level.

So I used Acronis (per my previous rant) to make a full backup. (I just felt that I should.) I went back to BIOS and attempted to restore the volume.

Setting up the RAID volume in the first place, with empty drives, was pretty straightforward. But there was NO freaking documentation on how to restore a RAID volume from a known good drive and a suspect mirror.

I wound up accidentally deleting the main partitions of both physical RAIDed drives. Thanks a f***ing lot, RAID! :(

So I decided to dump raid and just have a second, seldom used HDD. I restored from the Acronis backup to the single non RAID drive in a few hours and I was good to go.

I guess that RAID isn't supposed to be restored from existing disk contents without using a backup on external media. It's really difficult to find out what you're supposed to do when a volume is broken due to a non catastrophic error.
Bored Bystander Send private email
Monday, October 01, 2007
 
 
Just to throw out a competing view, network storage. By that I mean something like Amazon's S3 with brad fitzpatrick's brackup tool, rsync.net, and so on.  I'm sure there's more options out there for this.  Basically it means that all the hardware concerns of backup are Somebody Else's Problem (and further, you get off-site for free as well as the user never having to remember to swap out media).  The constraint of course is bandwidth... trying to fully image a few hundred gigabytes of data over a typical home DSL or cable connection would be a little painful.  (However, how many people actually have hundreds of GB of data that needs backup?  By which I mean, data you couldn't get back from original media or at the worst buying a new copy of $OS, $APP, etc.  Tallying up in my head, I can only think of a few hundred MB of data like old tax returns, papers I liked writing in school, bits of code and such I wrote on my own time, etc.  Further, if the tools support an incremental approach the actual full transfers will be limited.)  Yes this approach would probably involve some degree of scripting/integration/etc. on the part of the installer but find me a backup program that doesn't involve this already even for local operations.
son of anon
Monday, October 01, 2007
 
 
Wow, amazing stories and great feedbacks. Keep 'em coming!

(In a way, I am asking this because yours truly is also one of those user who has been burnt by his/her own homebrewed backup scheme. Hoping to learn something from the experience by learning to ask questions from you fine folks.

I used to have a pretty elaborate backup system to USB drives, but I am going to shun away from it because somewhere in the link things failed and I ended up with sufficient data corruption to make the whole scheme untrustworthy. Today it might be an innocent photo getting corrupted, but what if it knocks out a password wallet tomorrow or an acrobat pro scan of an important contract?

During this investigative phase of what scheme to try next, I am looking carefully at building a low-cost dedicated mini-tower serving a RAID based system. Probably a low-end Dell Vostro with an aftermarket low-end Promise SATA RAID card. How wrong can one go with this setup?*

* Famous last words...

It could be SAMBA, AFS, or iSCSI--whatever is reliable. At least the link between the storage and the PC will be relatively tried and true--say a proper Gig-E setup. From here, I'll try to listen to the great advice here and start with some of the commercial quality backup software available. And see if they are reliable.

Will commercial backup software generally care enough about the potential of data corruptions to do MD5 on data blocks? Thanks again ahead of time.

I think you are doing everyone a service, if a professional finds this thread off of Google they can get a firsthand view of how complicated the setups can be and and varied the advice can be. It would ground them well, let them know that hiring a good SOHO-networking/backup consultant is going to be worth the money.
Li-fan Chen Send private email
Monday, October 01, 2007
 
 
"I guess that RAID isn't supposed to be restored from existing disk contents without using a backup on external media. It's really difficult to find out what you're supposed to do when a volume is broken due to a non catastrophic error. "

Actually that is exactly what RAID is suspose do.  You just did it wrong.  When I have a hard drive fail like that the first thing I do is pick up the phone and call Dell's tech support and have them walk me through trouble shooting the failure and testing the hardware and then doing the restore.

If you didn't have RAID and this happened or something like it, you'd have had a non-bootable computer and still have your data on it.
Sneeker.
Monday, October 01, 2007
 
 
"During this investigative phase of what scheme to try next, I am looking carefully at building a low-cost dedicated mini-tower serving a RAID based system. Probably a low-end Dell Vostro with an aftermarket low-end Promise SATA RAID card. How wrong can one go with this setup?*"

So by low end you are not worried about the data that much? Harsh I know but thats how you are going about it?

For $860 you can get a Dell Power Edge 840 server with RAID 1 and 2x 250 gig hard drives.  Confirm with Dell what the phone support on that unit will be, I called in for some assistance on a 6 year old server when one of its drives failed.  Worst case you may have to pay for it, but atleast you know its the people who build your hardware that you are talking too.

If you are going through the effort of reseaching a good backup solution, do you want to risk your data for the sake of a few hundred $$?
Sneeker.
Monday, October 01, 2007
 
 
So, Sneeker: basically what you are saying is that you need paid telephone support in order to service problems with a RAID drive?

I don't dispute this - and in fact I realize that I did something wrong because I lost the partitions.

But I built my system myself --- I did not have a hotline that I can call. Just the motherboard's RAID documentation. Which said nothing about dealing with a live system.

IMO this is a not a good recommendation for an end user, unless they are willing to pay for a consultant when something screws up.
Bored Bystander Send private email
Monday, October 01, 2007
 
 
How about virus infections? I am reluctant to use an external HD as my backup, because it can be erased in no-time by a virus. Using CD/DVD feels more secure due to their read-once nature.

Do there exist external USB harddisks with a read-only switch? I have googled for it numereous times, but haven't found anything yet...
Yoriz Send private email
Monday, October 01, 2007
 
 
I recommend Carbonite.com for backup.
C Send private email
Monday, October 01, 2007
 
 
"So, Sneeker: basically what you are saying is that you need paid telephone support in order to service problems with a RAID drive?"

No, I could have likely found all the required documentation and done it myself.  But 250 gigs of data on the companies server I figured warrented a phone call.  Which was still covered under the Dell support contact that was on that server, must have been a lifetime contract at the time.

Had that server crashed out right and required rebuilding, I would have spent alot more time re-setting it up, going to the last backup, reinstalling all the services & software on it, etc. 

When I get a new server in, every now and then before I put the OS or just after I do, fake a hard drive failure and rebuild the array for my own education. 

The moral is, RAID 1 is only good if you test how to rebuild it.  I have also hosed the good drive on a RAID 1 (on a promise controller), luckly this was not a critial machine, since then I test how to do it before putting a machine into production.
Sneeker.
Monday, October 01, 2007
 
 
From multiple posts:

"About RAID - major PITA. I personally think that unless you set up RAID for a living, it is not a great option, until they make it easier to configure. I favor regular backups."

RAID is *NO* substitution for a backup. I say it again, please read it from my lips: RAID IS NO BACKUP!

"Just the motherboard's RAID documentation."

RAID systems integrated on the motherboard are usually cheap toys -- if they work at all as expected.

"But there was NO freaking documentation on how to restore a RAID volume from a known good drive and a suspect mirror."
"Will commercial backup software generally care enough about the potential of data corruptions to do MD5 on data blocks?"

Beside that RAID is no backup (I've already said that), most backup software and systems on the lower scale are great for doing backups, but they s**k on long-term data integrity (checksums and redundance) and restore. Hmm... seems to be a reason it is called "Backup", not "Data Safety".
Secure
Tuesday, October 02, 2007
 
 
"For $860 you can get a Dell Power Edge 840 server with RAID 1 and 2x 250 gig hard drives."

I hear you. Will take a careful look at that class of computers for redundancy of live data.
Li-fan Chen Send private email
Tuesday, October 02, 2007
 
 
"For $860 you can get a Dell Power Edge 840 server with RAID 1 and 2x 250 gig hard drives."

For about the same price, Dell also sells the Buffalo Technologies TeraStation 2 Terrabyte NAS, complete with the 4 500 GB hard drives, support for several varieties of RAID, ftp and http support, access via a couple different means (including TCP/IP). We just ordered one (actually, started the purchasing process here, which means we're probably still at least a month away from actually being able to place the order).
Ken White Send private email
Tuesday, October 02, 2007
 
 
Nice Ken,

If you have an existing good server to plug it into thats the way to go.

We got a Dell 2U with 1 TB in it (RAID5), and have room for 3 more drives when we out grow that.

We needed a new server at the time which is why we opted for a new server rather then NAS.
anon
Wednesday, October 03, 2007
 
 
+1 Acronis.

I have a 'barebones' Linux (Zenwalk) server over in the corner. It's my web testing server, and also (with Samba) acts as my backup site. Acronis is set for a full backup every month, and incremental every day.

The ability to open a backup from (say) last week is very useful in its own right. Saves me having to worry about version control!
Craig Welch Send private email
Wednesday, October 03, 2007
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz