Digital asset management as part of a Raw workflow

This is just a summary of my digital asset management (DAM) strategy with my current setup (a Canon 5D and Capture One raw processor). This also works with Rawshooter Premium. I don't know about other raw processors. I would recommend buying the DAM book by Peter Krogh. This workflow is based on my reading of that book and a lot of these ideas are not mine, but based on Krogh's scalable archiving techniques. First off you will need the following hardware in addition to your CF cards and digital wallet etc. for storage in the field:

External hard drive(s)
DVD burner (or CD burner)

You also need an image cataloguing system. I use iMatch software for this. This method is not 100% infallible, no method is, and also I have cut a few of the corners mentioned in the DAM book. I am not a professional photographer and don't necessarily need the same level of security as others might. If I were working on a professional level I would up the level of security outlined below. However, I feel that this workflow is enough for me at the present time.

1. Take the pictures and store in the field

It is best according to Krogh to set the camera to have consecutive file numbering turned on in the camera. This way it is easier to keep track of which order the pictures were taken in on any particular trip. I believe most digital cameras have an option to set this. So my 5D remembers the number of the last shot taken and increments by one every new shot rather than starting at 0001 again every time a new CF card is inserted.

I have a few compact flash (CF) cards that I take with me when I go out shooting. I shoot all my pictures in RAW mode. If all the cards get full I download all of them onto a portable hard drive. In my case this is an XS 20MB drive. I then start to use the cards again, reformatting them in the camera as I go. If I have some shots that are particularly valuable on a card I might keep that card unused as a backup to the portable drive until I get home.

So the moral here is if you really want to be extra safe either get two portable drives and back up your cards twice in the field or get a lot of CF cards - I tend to prefer many smaller 512MB and 1G cards rather than a few huge capacity cards. I think this gives you more flexibility and also if a card fails you have not lost 4Gs worth of shots in one go. Of course it also depends on what type of things you shoot. If you get through lots of frames in a short time then you obviously might prefer bigger cards.

2. Downloading to the computer

Using Capture One (LE or Pro) when I attach the card reader with CF card into the USB it automatically detects and offers to download the files and rename them (Rawshooter Premium also does this). If I use the portable drive I have to copy manually. I download to a drive on the PC itself into a directory specifically for work in progress.

The files and directories should be renamed according to a formula where they will be listed in the order they were shot. The formula I use for directories is:

MRTOML_yymmdd_nnn

Where MRTOML is an abbreviation of my name (it can be anything really), yyddmm is the date of the download in year, month, day format, and nnn is the number of the directory downloaded on that day (e.g., 001, 002, 003... etc.)

So if I downloaded 2 cards on the 16th of July 2005 there will be two directories:

MRTOML_050716_001 and MRTOML_050716_002

I sometimes append a note to the end of the directory to remind me what is in there. So 'MRTOML_060606_001_family' refers to pictures I took of my family on the 6th of June 2006. Note that appending a name like this has no effect on the way the directories will be listed in date order by your operating system. In reality if you use a decent cataloguing system such as iMatch (see below), tagging your directories like this should become redundant because the catalogue will know where everything is based on keywords/categories. However, I find for work in progress it sometimes helps to locate stuff quickly if you have a shorthand tag at the end of the directory name telling you what is in there.

Within these directories the individual raw pictures will be given names according to the following formula:

MRTOML_yymmdd_nnnn.cr2

In this case MRTOML and yymmdd are the same as the directory name and nnnn is the number already assigned by the camera to each shot. Using this method it is easy to figure out when and what was shot in which order. cr2 is just the Canon raw extension. (Note: Freeware can be downloaded that renames files which is much better than Windows explorer - I use ExplorerXP freeware.)

3. Online storage?

At this point many people would burn a DVD or transfer to some external drive. I don't do this, because I am not a professional and if I lose a couple of directories of shots it is not the end of the world. If I have some stuff that I really feel is important I will upload it to a virtual online drive at this point. For example, streamloader. Then I can always get the files back in the event of catastrophe. I also don't wipe the cards/portable drive until my next outing. So I also have a temporary backup in any case.

4. Processing the shots

What I do now is work on the shots in my raw processor and, if needed, Photoshop. I set the raw processor to develop the pictures in a subdirectory of the original directory. In Camera One this is in a directory called 'develops' and in rawshooter 'converted'. Some software produces further directories to store the raw conversion settings - this is no problem if these directories are within the original folder. Again if the work is very important I might make an intermediate backup of the shots (to streamloader or whatever).

5. Transfer to archive

When I think I have probably finished all the editing I am going to do on a set (directory) of shots in my work in progress directory, I transfer them to an external hard drive. This external drive also forms the basis of my DVD archive. (A similar procedure can also be used for CDs or BluRay disks or whatever you use.)

On the external hard drive I have directories set up to match each DVD I will burn (this is again Krogh's method). I know a DVD can hold 4.38GB of data so I just copy the directories over from my PC (including all the subdirectories with the Raw settings and developed Raw files) into a directory on the external drive until the directory approaches 4.38GB in size. Then I create a new folder for the next DVD and so on.

So I have a set of directories on the external drive that looks like this:

DVD_001
DVD_002
DVD_003
etc...

And within each DVD directory there will be your image file directories copied from your work in progress on the PC so the final directory structure looks something like this:

DVD_001
- MRTOML_060606_001
- MRTOML_060606_002
- MRTOML_060609_001_family
- MRTOML_060611_001
DVD_002
- MRTOML_060613_001_macroflowers
- MRTOML_060614_001
...

and so on...


6. Burn the DVD (or a CD)

When a DVD directory gets close to the 4.38GB limit I burn the whole directory onto two DVDs. One DVD gets stored at home and the other at the office. Once the DVDs are burned (and verified) I delete the files from my main PC 'work in progress' drive leaving them on the external drive.

The advantage of this method is that it is scalable. If in the future I decide to switch my archive to BluRay disks or whatever I can create more directories the size of a BluRay disk and simply copy sufficient DVD directories into the BluRay directories and then burn a BluRay disk instead. If the external hard drive gets full I buy another drive, add it onto the existing setup, and simply carry on using the new drive in addition to the old one.

This is the great advantage of the method described by Peter Krogh in the DAM book. As technology marches on and storage media get bigger you can easily keep up if you have a directory structure on you hard drives that matches the media you use for archiving. So you have a working solution where your images are available all the time (via your external hard drives) and if a drive fails you can easily restore it from your DVD collection.

If you want to consolidate at some point and replace external smaller drives with one bigger one, say, again it is easy as you have the DVD backups. You simply take off the old drives and put on the new one. Then simply copy the DVDs back onto the new drive into their respective DVD directories.

If you want to reburn DVDs periodically for safety reasons then again it is simply a matter of burning each DVD directory again. I write on the case of the DVD when it was burned and aim to reburn every 2 or 3 years. Although next time there will probably be a better storage solution out there. Again this doesn't matter with a scalable solution based on directories like this.

Image cataloguing

The key to making this work though is your cataloguing software. At some point you have to catalogue your images and specify the location of the files in the image management database. I use iMatch to catalogue my work when it reaches the external hard drive stage at the latest. I have a field in my database that specifies which DVD directory (and hence also which DVD) the files are on. Thus I can find the file on the external hard drive to edit or print, and I know which DVD it is burned on in case of hard drive failure. If I can find the original raw file it follows that I can also find any processed versions of the file in the subdirectories of the raw directory (it will have the same name, but with a different extension).

Also don't forget you must back up your image database to DVD and external drive every so often as well.

5D dust issues

There has been a lot of discussion on the dust issues with the Canon 5D camera.

E.g., : http://forums.dpreview.com/forums/readflat.asp?forum=1032&thread=18708842

I noticed large specks of dust appearing on mine after a few weeks even without changing the lens (I put a 50mm lens on as soon as the camera arrived and didn't take it off). Presumably there is already dust in the chamber that gets moved around when the camera is used. I tried the 'rubber blower' method recommended by Canon and this definitely made matters worse. So I suspect that the dust was already in there and the blower actually made it worse.

I eventually plucked up the courage to try the 'Copperhill method'. This is based on using a plastic spatula with a pecpad (lint free swab) wrapped around it. The swab is soaked in Eclipse fluid which I believe is methanol. The methanol does not leave smears on the sensor as it evaporates extremely quickly. The swab is passed over the sensor once per side.

It took me seven or eight tries before I got the hang of it, but it did actually work. You have to keep taking pictures inbetween swabbings to see if and where the dust is going. You can quickly spot stubborn particles and whether you are getting into the corners, especially if you use software where you can enlarge and cycle through several images at once and compare the exact positioning of the frames in each shot (I used Rawshooter Essentials for this which allows you to magnify the frame and then use the mouse wheel to cycle through successive 'cleaning shots'). The spots that don't budge appear on the computer screen in exactly the same place.

There are also several other methods out there (I am intrigued by the sensorklear option which seems to be user friendly and portable). But I am going to persevere with this method for the time being and see how it goes.

There is also apparently some advantage to regularly cleaning your sensor even if it does not have a lot of dust. This helps prevent stubborn hard to clear spots from becoming stuck to the sensor. The notes on the Copperhill site at PBase are worth reading thoroughly on this. In Europe you can get the Copperhill kits from http://www.chili-pix.ch/ . It took about a week for mine to arrive in the UK.

Esteban

DSC_0053
Originally uploaded by m_r_tomlinson.

Here is a test post of my nephew Esteban.