Finally get to do something fun at work

Started by PopeyesPappy, October 03, 2018, 04:53:21 PM

Previous topic - Next topic

PopeyesPappy

I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:



It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.
Save a life. Adopt a Greyhound.

Cavebear

Quote from: PopeyesPappy on October 03, 2018, 04:53:21 PM
I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:



It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.

Well, I recognized the Logitech Mouse...
Atheist born, atheist bred.  And when I die, atheist dead!

Baruch

The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.
Ha’át’íísh baa naniná?
Azee’ Å,a’ish nanídį́į́h?
Táadoo ánít’iní.
What are you doing?
Are you taking any medications?
Don't do that.

PopeyesPappy

Quote from: Baruch on October 04, 2018, 07:14:15 PM
The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.

Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added. It took my laptop, which is a few years old but an I7 workstation, a couple of days to process everything. This time I am going to need to do that to even more images for one project plus convert nearly 200000 TIF images to archival PDF with metadata on a separate project. It's going to take a while even with the new machine.
Save a life. Adopt a Greyhound.

Gawdzilla Sama

I did this kind of image for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

PopeyesPappy

Quote from: Gawdzilla Sama on October 05, 2018, 08:15:36 AM
I did this kind of image for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.

We won't do any of the image captures for the archives. We going to use a university with a specialized lab for that part of the job. The way they handle large format stuff like your image of the North Carolina is hang it on a large light board with a vacuum system and take a photograph of it with a very expensive camera. A single 60" x 40" Tif at 400 PPI and 24 bit color is going to be between 9 and 10 gigabytes. Other stuff is smaller, but there are going to be almost 200,000 original images before we are done. My part of this will be converting the Tif files to PDF/A, and extracting metadata from a spreadsheet and inserting it into to the PDF.

The size of some of the files is going to slow things down a bit, but the biggest factor as far as time goes is just the number of files. If it takes an average of 5 seconds to process an image we'd be looking at 277 hours or 11.6 days of processing time. If it's 15 seconds you can triple the time so a solid month plus of processing. That doesn't include the prep work either. It's just time the computer spends crunching the numbers. 
Save a life. Adopt a Greyhound.

Gawdzilla Sama

Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

Hydra009

Quote from: PopeyesPappy on October 05, 2018, 07:46:27 AM
Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added.
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.

Sal1981

Aerial photos usually require lossless quality, I know that from my time in a map making company.

PopeyesPappy

Quote from: Hydra009 on October 05, 2018, 03:57:16 PM
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.

Our image deliverables on the inventory projects is TIF and JPG. TIF for archives. JPG for working with. Documents web pages that sort of thing.
Save a life. Adopt a Greyhound.

PopeyesPappy

Quote from: Gawdzilla Sama on October 05, 2018, 01:44:07 PM
Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.

Your 8 year old Nikon probably won't do 400 ppi. Ours won't. The NPS standard for archives is 400 ppi. Inventory picks they'll accept less, but when they are archiving documents its 400.

The vacuum light box really speeds things up and holds it down flatter than you could get laying in the floor. You should see the rig they have for photographing books.
Save a life. Adopt a Greyhound.

trdsf

Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.
"My faith in the Constitution is whole, it is complete, it is total, and I am not going to sit here and be an idle spectator to the diminution, the subversion, the destruction of the Constitution." -- Barbara Jordan

Gawdzilla Sama

Quote from: Sal1981 on October 05, 2018, 04:11:34 PM
Aerial photos usually require lossless quality, I know that from my time in a map making company.
I looked at a ... few ... aerial pix when I was on active duty. Fun to be the eye in the sky, ain't it?
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

PopeyesPappy

Quote from: trdsf on October 05, 2018, 08:15:59 PM
Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.

That Samsung 970 EVO NVMe SSD should read about 6 times faster and write about 4 times faster than an SSD with a SATA interface. ;)
Save a life. Adopt a Greyhound.

PopeyesPappy

Welp, here it is. Had a hell of a time getting Windows to recognize the RAID array, but everything is up and working. We are now ready to work with a shitload of images and digitize analog audio. On the audio side we can do 1/4" reel to reel tapes on 7" and 10.5" reels, cassette tapes, micro cassette tapes, mini disks, old cd's, 5-1/4" and 3.5" floppies and zip disks.



Don't mind the mess in the background. We haven't wanted to throw out any of the boxes until we know everything worked. It does so I'll be cleaning up tomorrow.
Save a life. Adopt a Greyhound.