Author Topic: Finally get to do something fun at work  (Read 141 times)

Offline PopeyesPappy

Finally get to do something fun at work
« on: October 03, 2018, 04:53:21 PM »
I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:



It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.
Save a life. Adopt a Greyhound.


Offline Cavebear

Re: Finally get to do something fun at work
« Reply #1 on: October 04, 2018, 03:58:38 AM »
You are not allowed to view links. Register or Login
I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:



It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.

Well, I recognized the Logitech Mouse...
Atheist born, atheist bred.  And when I die, atheist dead!

Offline Baruch

Re: Finally get to do something fun at work
« Reply #2 on: October 04, 2018, 07:14:15 PM »
The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.
שלום

Offline PopeyesPappy

Re: Finally get to do something fun at work
« Reply #3 on: October 05, 2018, 07:46:27 AM »
You are not allowed to view links. Register or Login
The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.

Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added. It took my laptop, which is a few years old but an I7 workstation, a couple of days to process everything. This time I am going to need to do that to even more images for one project plus convert nearly 200000 TIF images to archival PDF with metadata on a separate project. It's going to take a while even with the new machine.
Save a life. Adopt a Greyhound.


Re: Finally get to do something fun at work
« Reply #4 on: October 05, 2018, 08:15:36 AM »
I did You are not allowed to view links. Register or Login for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

Offline PopeyesPappy

Re: Finally get to do something fun at work
« Reply #5 on: October 05, 2018, 08:37:55 AM »
You are not allowed to view links. Register or Login
I did You are not allowed to view links. Register or Login for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.

We won't do any of the image captures for the archives. We going to use a university with a specialized lab for that part of the job. The way they handle large format stuff like your image of the North Carolina is hang it on a large light board with a vacuum system and take a photograph of it with a very expensive camera. A single 60" x 40" Tif at 400 PPI and 24 bit color is going to be between 9 and 10 gigabytes. Other stuff is smaller, but there are going to be almost 200,000 original images before we are done. My part of this will be converting the Tif files to PDF/A, and extracting metadata from a spreadsheet and inserting it into to the PDF.

The size of some of the files is going to slow things down a bit, but the biggest factor as far as time goes is just the number of files. If it takes an average of 5 seconds to process an image we'd be looking at 277 hours or 11.6 days of processing time. If it's 15 seconds you can triple the time so a solid month plus of processing. That doesn't include the prep work either. It's just time the computer spends crunching the numbers. 
Save a life. Adopt a Greyhound.


Re: Finally get to do something fun at work
« Reply #6 on: October 05, 2018, 01:44:07 PM »
Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

Offline Hydra009

Re: Finally get to do something fun at work
« Reply #7 on: October 05, 2018, 03:57:16 PM »
You are not allowed to view links. Register or Login
Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added.
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.

Online Sal1981

Re: Finally get to do something fun at work
« Reply #8 on: October 05, 2018, 04:11:34 PM »
Aerial photos usually require lossless quality, I know that from my time in a map making company.
"The first principle is that you must not fool yourself - and you are the easiest person to fool" --- Richard P. Feynman

Offline PopeyesPappy

Re: Finally get to do something fun at work
« Reply #9 on: October 05, 2018, 06:26:02 PM »
You are not allowed to view links. Register or Login
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.

Our image deliverables on the inventory projects is TIF and JPG. TIF for archives. JPG for working with. Documents web pages that sort of thing.
Save a life. Adopt a Greyhound.


Offline PopeyesPappy

Re: Finally get to do something fun at work
« Reply #10 on: October 05, 2018, 06:44:52 PM »
You are not allowed to view links. Register or Login
Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.

Your 8 year old Nikon probably won't do 400 ppi. Ours won't. The NPS standard for archives is 400 ppi. Inventory picks they'll accept less, but when they are archiving documents its 400.

The vacuum light box really speeds things up and holds it down flatter than you could get laying in the floor. You should see the rig they have for photographing books.
Save a life. Adopt a Greyhound.


Offline trdsf

Re: Finally get to do something fun at work
« Reply #11 on: October 05, 2018, 08:15:59 PM »
Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.
Sir Terry Pratchett, on being told about the theory that the universe is a computer simulation: "If we all get out and in again, would it start to work properly this time?"

Re: Finally get to do something fun at work
« Reply #12 on: October 06, 2018, 06:33:06 AM »
You are not allowed to view links. Register or Login
Aerial photos usually require lossless quality, I know that from my time in a map making company.
I looked at a ... few ... aerial pix when I was on active duty. Fun to be the eye in the sky, ain't it?
We 'new atheists' have a reputation for being militant, but make no mistake  we didn't start this war. If you want to place blame put it on the the religious zealots who have been poisoning the minds of the  young for a long long time."
PZ Myers

Offline PopeyesPappy

Re: Finally get to do something fun at work
« Reply #13 on: October 07, 2018, 12:20:54 AM »
You are not allowed to view links. Register or Login
Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.

That Samsung 970 EVO NVMe SSD should read about 6 times faster and write about 4 times faster than an SSD with a SATA interface. ;)
Save a life. Adopt a Greyhound.


 

SMF spam blocked by CleanTalk