Atheistforums.com

Science Section => Science General Discussion => Math and Computers => Topic started by: PopeyesPappy on October 03, 2018, 04:53:21 PM

Title: Finally get to do something fun at work
Post by: PopeyesPappy on October 03, 2018, 04:53:21 PM
I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:

(https://i.imgur.com/zXFdSBk.jpg)

It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.
Title: Re: Finally get to do something fun at work
Post by: Cavebear on October 04, 2018, 03:58:38 AM
Quote from: PopeyesPappy on October 03, 2018, 04:53:21 PM
I've got a new project at work that is going to require processing a couple hundred thousand images. Some of them are going to be very large TIF files. 60" x 40" at 400 ppi x 24 bits. Anyway I budgeted for a new computer to do the grunt work and I ordered the parts Monday and got a shipping number today. It's been a while since I built a PC, and this ought to be fun. Here's the parts list:

(https://i.imgur.com/zXFdSBk.jpg)

It won't have any fancy graphics, but I don't need that. Just a lot of processing power, memory and disk space. This will be my first foray into NVME SSD. This thing ought to be pretty damn fast.

Well, I recognized the Logitech Mouse...
Title: Re: Finally get to do something fun at work
Post by: Baruch on October 04, 2018, 07:14:15 PM
The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 05, 2018, 07:46:27 AM
Quote from: Baruch on October 04, 2018, 07:14:15 PM
The tiff then would be between 9 and 10 gigabytes.  Not too large.  What is crucial is how many computations per pixel you are doing.  A fast fourier transform of a picture involves a lot of computations per pixel, because the processing of each pixel, depends on all the others.

Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added. It took my laptop, which is a few years old but an I7 workstation, a couple of days to process everything. This time I am going to need to do that to even more images for one project plus convert nearly 200000 TIF images to archival PDF with metadata on a separate project. It's going to take a while even with the new machine.
Title: Re: Finally get to do something fun at work
Post by: Gawdzilla Sama on October 05, 2018, 08:15:36 AM
I did this kind of image (https://www.history.navy.mil/content/dam/nhhc/research/library/online-reading-room/ships/wardamagereportno61/wdrbb55-63a.jpg) for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 05, 2018, 08:37:55 AM
Quote from: Gawdzilla Sama on October 05, 2018, 08:15:36 AM
I did this kind of image (https://www.history.navy.mil/content/dam/nhhc/research/library/online-reading-room/ships/wardamagereportno61/wdrbb55-63a.jpg) for the Naval History and Heritage Command. Had to scan it in in three pieces and paste it together.  Didn't strain my cpu at all. Used Paint Shop for the editing.

We won't do any of the image captures for the archives. We going to use a university with a specialized lab for that part of the job. The way they handle large format stuff like your image of the North Carolina is hang it on a large light board with a vacuum system and take a photograph of it with a very expensive camera. A single 60" x 40" Tif at 400 PPI and 24 bit color is going to be between 9 and 10 gigabytes. Other stuff is smaller, but there are going to be almost 200,000 original images before we are done. My part of this will be converting the Tif files to PDF/A, and extracting metadata from a spreadsheet and inserting it into to the PDF.

The size of some of the files is going to slow things down a bit, but the biggest factor as far as time goes is just the number of files. If it takes an average of 5 seconds to process an image we'd be looking at 277 hours or 11.6 days of processing time. If it's 15 seconds you can triple the time so a solid month plus of processing. That doesn't include the prep work either. It's just time the computer spends crunching the numbers. 
Title: Re: Finally get to do something fun at work
Post by: Gawdzilla Sama on October 05, 2018, 01:44:07 PM
Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.
Title: Re: Finally get to do something fun at work
Post by: Hydra009 on October 05, 2018, 03:57:16 PM
Quote from: PopeyesPappy on October 05, 2018, 07:46:27 AM
Last time around I was only working with about 1400 original NEF images. They had to be converted to TIF and JPG and have some metadata added.
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.
Title: Re: Finally get to do something fun at work
Post by: Sal1981 on October 05, 2018, 04:11:34 PM
Aerial photos usually require lossless quality, I know that from my time in a map making company.
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 05, 2018, 06:26:02 PM
Quote from: Hydra009 on October 05, 2018, 03:57:16 PM
Ooh, Nikon.

Why JPG?  Unless file size is a huge problem, I can't think of an single reason to prefer lossy jpg to say, lossless png.

Our image deliverables on the inventory projects is TIF and JPG. TIF for archives. JPG for working with. Documents web pages that sort of thing.
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 05, 2018, 06:44:52 PM
Quote from: Gawdzilla Sama on October 05, 2018, 01:44:07 PM
Hmmm, my eight year old Nikon reproduced pix 48"x48" with no problem. I just put them on the floor and shot from the edge of a table.

Your 8 year old Nikon probably won't do 400 ppi. Ours won't. The NPS standard for archives is 400 ppi. Inventory picks they'll accept less, but when they are archiving documents its 400.

The vacuum light box really speeds things up and holds it down flatter than you could get laying in the floor. You should see the rig they have for photographing books.
Title: Re: Finally get to do something fun at work
Post by: trdsf on October 05, 2018, 08:15:59 PM
Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.
Title: Re: Finally get to do something fun at work
Post by: Gawdzilla Sama on October 06, 2018, 06:33:06 AM
Quote from: Sal1981 on October 05, 2018, 04:11:34 PM
Aerial photos usually require lossless quality, I know that from my time in a map making company.
I looked at a ... few ... aerial pix when I was on active duty. Fun to be the eye in the sky, ain't it?
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 07, 2018, 12:20:54 AM
Quote from: trdsf on October 05, 2018, 08:15:59 PM
Oh, you're going to love booting from an SSD.  I'm running 64 bit Win7 Pro on a 3.5GHz quad core Intel, and the delay between power on and login is almost negligible.  My data drives are standard hard drives (3.5Tb total between two drives -- I could not afford that kind of space on SSD), but the boot drive is solid state and holy Bob that's fast.

That Samsung 970 EVO NVMe SSD should read about 6 times faster and write about 4 times faster than an SSD with a SATA interface. ;)
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 25, 2018, 04:51:08 PM
Welp, here it is. Had a hell of a time getting Windows to recognize the RAID array, but everything is up and working. We are now ready to work with a shitload of images and digitize analog audio. On the audio side we can do 1/4" reel to reel tapes on 7" and 10.5" reels, cassette tapes, micro cassette tapes, mini disks, old cd's, 5-1/4" and 3.5" floppies and zip disks.

(https://i.imgur.com/GssIHNy.jpg)

Don't mind the mess in the background. We haven't wanted to throw out any of the boxes until we know everything worked. It does so I'll be cleaning up tomorrow.
Title: Re: Finally get to do something fun at work
Post by: PopeyesPappy on October 26, 2018, 11:02:06 AM
This should work.

(https://i.imgur.com/iDIhVFD.jpg)
Title: Re: Finally get to do something fun at work
Post by: aitm on October 28, 2018, 06:34:53 PM
I was able to do a screen shot on my phone...so.....I learned that today........