Managing Large (and lots of!) Files

I received an email recently from a friend who went to China with me in May. He was shooting with the new Nikon D800, and had a question:

“How do you cope with the mass storage requirements of the D800?  With the volume of photography you’ve done lately I have to believe you have a more severe problem than I.  During the China trip I ended up with 388 Gigs of images.  I only had 225 Gigs of memory cards with me and my netbook only had a 250 gig hard drive.  Fortunately, I had a one terabyte drive with me.  While I have a three terabyte NAS at home it’s almost full from previous shooting sessions.  What do you do for storage and archiving?  In my case I only have a DVD burner at the moment.  At a max of 8 Gigs per disk it’s going to take a lot of disks to backup the 388 Gigs from China alone.  I’d be

This shot with the D800, at 36 megapixels, weighs in at about 40 megabytes.
NIKON D800, ISO 200, 1/30, f/4.5, -2.3 EV, 30mm. Photo copyright Reed Hoffmann.

interested to know how you deal with this.”

Obviously these new high-resolution cameras are forcing changes to our workflows. I shoot my D800 set for 14-bit compressed NEF files. That means every time I press the shutter button I’m creating a file that’s about 40MB in size. Forty megabytes! I’m now shooting 32GB cards, which was unthinkable just a few years ago. But as my friend points out, the bigger problem is what to do with the files afterwards. For me, that means I’ve turned my workflow upside down.

In the past, with cheap storage, I never bothered deleting pictures. It wasn’t worth the time. I’d simply go through my shoot and mark my favorites, back those up to a “Best of” collection, archive everything to hard drives and DVDs and move on. Now I’ve changed that.

After downloading (and batch captioning) I go through and mark any picture I think might have a future. It could be a photo I love or just

Even the entry-level Nikon D3200, at 24 megapixels, creates RAW files that are around 20MB in size.
NIKON D3200, ISO 100, 1/1.6, f/11, -1.0 EV, 10mm. Photo copyright Reed Hoffmann.

one that makes a good teaching example. Next step – delete everything else. At that point I rename what’s left. Is there a danger to doing it this way? Absolutely. You’ve got to be careful you don’t delete a picture you might want. That’s one reason I don’t format my cards right away – if I realize later I don’t see a photo I remember shooting, I can always go back to the card. That means I try to bring enough cards so I don’t have to re-format any until after the trip, when everything’s home and backed up.

For the China trip my friend was writing about, I had nine full days shooting, plus one evening on arrival  and one short morning before leaving. Since I was teaching, I wasn’t shooting as much as I might have otherwise, but still by the end of the trip, using the workflow above, I returned with 2965 images filling 86.7 GB. And that’s about three times what I normally come home with. Remember the line from Jaws, “You’re going to need a bigger boat,”? Well, we’re going

I have two servers in my basement, each with four hard drives in it.

to need bigger hard drives. So that presents the next problem –storage.

With storage I always think short-term and long-term. Short term is multiple hard drives, even on the road. Once I get home the photos are moved to my server, a RAID array that gives me redundant storage. Every month or two I burn everything new to DVDs. Those DVDs go on a shelf in my office, and are part of my long-term solution. At the end of the year I transfer everything from the server to one hard drive large enough to hold all of it, and it then joins a shelf of hard drives from the past twelve years. So I try to keep everything on both hard drives for fast access and DVDs for long-term (because hard drives crash, but DVDs don’t).

Some of the “Best of” folders I’ve created on one of my hard drives.

Most importantly, I create small collections of my best photos. I have a “best of” collection for every year, then “best of” collections from any large shoots I did each year. That means for 2012 I have a folder on the server titled “2012 Best of,” plus folders with “best of” collections from trips to Yosemite, Peru, China, Oregon, Chicago, Lake Tahoe and soon New Mexico. And those “best of” collections are not just on my server, but on several hard drives, including one in a safe deposit box. As long as I have those “best of” collections, then I could live with losing the DVDs and hard drives that have everything I’ve ever shot.

Photography’s always involved keeping track of pictures and making decisions on what’s worth keeping and what isn’t. That hasn’t changed. But our ability to shoot endless numbers of pictures without needing to buy film and get it processed has changed how we shoot. We’re taking more pictures than ever, and for me, that means being more selective than ever over which ones are worth keeping.