[GM-help] Best performance for resizing?

[Mike Brittain]

I am planning to use GraphicsMagick to batch process our back stock of
images at Etsy (ahem, a few million images). The images are being shrunk
and cropped to fit 170x135. I have been testing the following set of
options:

/usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity
Center \
-crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg

Are there any options for "convert" that I am leaving out but should
consider using to improve the resize speed? I already ran a test of the
available filters on a sample set of images and found "triangle" to have the
best balance of image quality, speed, and smaller file sizes.

I'm using version 1.3.12.

Thanks,
Mike

P.S. John Allspaw recommended I check in here to get the best advice on
GraphicsMagick!

[Mike Brittain]

I am planning to use GraphicsMagick to batch process our back stock of
images at Etsy (ahem, a few million images). The images are being shrunk
and cropped to fit 170x135. I have been testing the following set of
options:

/usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity
Center \
-crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg

Are there any options for "convert" that I am leaving out but should
consider using to improve the resize speed? I already ran a test of the
available filters on a sample set of images and found "triangle" to have the
best balance of image quality, speed, and smaller file sizes.

I'm using version 1.3.12.

Thanks,
Mike

P.S. John Allspaw recommended I check in here to get the best advice on
GraphicsMagick!

[Bob Friesenhahn]

I am planning to use GraphicsMagick to batch process our back stock of images at Etsy (ahem, a few
million images).  The images are being shrunk and cropped to fit 170x135.  I have been testing the
/usr/bin/gm convert input.jpg -filter triangle -resize "170x135^" -gravity Center \
  -crop "170x135+0+0" -quality 90 -type TrueColor -interlace None output.jpg
Are there any options for "convert" that I am leaving out but should consider using to improve the
resize speed?  I already ran a test of the available filters on a sample set of images and found
"triangle" to have the best balance of image quality, speed, and smaller file sizes.

Yes. There are a few things. Something which is vital to know for
JPEG is that the JPEG decoder can return a smaller size
(subresolution) of the image to GraphicsMagick. That saves a lot of
time. This support is triggered by using the -size option prior to
the input file name with a geometry argument which is at least the
size that you need. You will want to specify a size at least double
the final size so that antialiasing works well.

Also, a -thumbnail option was added which works like -resize but takes
steps to make resizing to much smaller sizes a lot faster. It does
that by using the 'sample' algorithm followed by the 'resize'
algorithm using the box filter. But since it knows the image size and
the ratios, it does this intelligently.

So use something like

gm convert -size 340x270 input.jpg -thumbnail "170x135^" \
-gravity Center -crop "170x135+0+0" -quality 90 -type TrueColor \
-interlace None output.jpg

Lastly, if you are batch processing a whole lot of files in a
directory tree, you may want to use 'mogrify' instead since it is able
to execute the same commands on a large set of files without starting
a new process for each one. The -output-directory and
-create-directories options can be quite useful in order to process a
sub-directory tree of images into another subdirectory tree. See
"http://www.graphicsmagick.org/FAQ.html#how-can-i-process-many-files-at-once".

I'm using version 1.3.12.

Good.

P.S. John Allspaw recommended I check in here to get the best advice on GraphicsMagick!

Many thanks to John Allspaw. :-)

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/

[Mike Brittain]

Thanks for the info, Bob. This is super helpful.

On Thu, Mar 11, 2010 at 10:34 PM, Bob Friesenhahn <

Something which is vital to know for JPEG is that the JPEG decoder can
return a smaller size (subresolution) of the image to GraphicsMagick. That
saves a lot of time. This support is triggered by using the -size option
prior to the input file name with a geometry argument which is at least the
size that you need. You will want to specify a size at least double the
final size so that antialiasing works well.

I had tried the -size option earlier with no improvements, but now that
you've mentioned this I found that I had it placed *after* the input file.

For my testing, I have 100 files I have been resizing. Without -size, this
was taking 31 seconds. After adding the -size option, the speed improved to
10 seconds. Faster, indeed!


Also, a -thumbnail option was added which works like -resize but takes steps

to make resizing to much smaller sizes a lot faster. It does that by using
the 'sample' algorithm followed by the 'resize' algorithm using the box
filter. But since it knows the image size and the ratios, it does this
intelligently.
So use something like
gm convert -size 340x270 input.jpg -thumbnail "170x135^" \
-gravity Center -crop "170x135+0+0" -quality 90 -type TrueColor \
-interlace None output.jpg

I replaced -resize with -thumbnail as you suggest in the options above, but
in this case the speed dropped. The same set of 100 images took 14 seconds
to process when using -thumbnail.

Lastly, if you are batch processing a whole lot of files in a directory
tree, you may want to use 'mogrify' instead since it is able to execute the
same commands on a large set of files without starting a new process for
each one.

Looking at the directory options, I don't think this is going to be helpful
for my case. The file layout I have is one full-size image and a number of
derivative sizes per directory. For example, in a single directory we have
files something like: 12345_full.jpg, 12345_small.jpg, 12345_medium.jpg, and
12345_large.jpg. The new sizes I am batch processing will be derivatives of
the *_full.jpg size but must live in the same directory. Also, these are
new files, and not resizing a file in place of the original. Correct me if
I'm wrong here.

I am glad that you brought this up, however, because we have an upcoming
project that *will* touch files in place. Mogrify will probably come in use
there.

Mike

[Bob Friesenhahn]

I replaced -resize with -thumbnail as you suggest in the options above, but in this case the speed
dropped.  The same set of 100 images took 14 seconds to process when using -thumbnail.

Depending on the number of CPU cores you have active, -resize may be
faster than -thumbnail in terms of wall-clock time, but it will
usually use less total CPU. Since the JPEG library is already
returning a fairly small image, there is much less benefit from
-thumbnail.

Looking at the directory options, I don't think this is going to be helpful for my case.  The file
layout I have is one full-size image and a number of derivative sizes per directory.  For example,
in a single directory we have files something like: 12345_full.jpg, 12345_small.jpg,
12345_medium.jpg, and 12345_large.jpg.  The new sizes I am batch processing will be derivatives of
the *_full.jpg size but must live in the same directory.  Also, these are new files, and not
resizing a file in place of the original.  Correct me if I'm wrong here.

In this case, be sure to investigate the '-write filename' option
which allows you to write the current image to the specified filename
and then continue processing. In this case you might not take
advantage of the JPEG -size option, but you can use -thumbnail (or
-resize, or -scale) in a sort of pyramid fashion to produce the
various smaller sizes while reading the original image just once.

For example:

% time gm convert -verbose 100_4994.jpg -quality 80 +profile '*' \
-write original.jpg -thumbnail 60% -write large.jpg -thumbnail 60% \
-write medium.jpg -thumbnail 60% -write small.jpg -thumbnail 120x80 \
thumb.jpg
100_4994.jpg JPEG 1728x2304+0+0 DirectClass 8-bit 1.3M 0.210u 0:01 (17.3M pixels/s)
100_4994.jpg=>original.jpg JPG 1728x2304+0+0 DirectClass 8-bit 798.5K 0.290u 0:01 (13.6M pixels/s)
100_4994.jpg=>large.jpg JPG 1728x2304=>1037x1382+0+0 DirectClass 8-bit 292.5K 0.100u 0:01 (38.0M pixels/s)
100_4994.jpg=>medium.jpg JPG 1728x2304=>622x829+0+0 DirectClass 8-bit 102.8K 0.040u 0:01
100_4994.jpg=>small.jpg JPG 1728x2304=>373x497+0+0 DirectClass 8-bit 31.6K 0.020u 0:01
100_4994.jpg JPEG 1728x2304=>60x80+0+0 DirectClass 8-bit 1.780u 0:01 (4.6M pixels/s)
100_4994.jpg=>thumb.jpg JPG 1728x2304=>60x80+0+0 DirectClass 8-bit 0.000u 0:01
gm convert -verbose 100_4994.jpg -quality 80 +profile '*' -write original.jpg 1.90s user 0.09s system 184% cpu 1.078 total

Just keep in mind that there could be a small bit of additional
quality loss due to resizing an already resized image.

Regardless, when using JPEG format, the bottleneck is likely to be the
JPEG decode/encode rather than GraphicsMagick.

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/

[Mike Brittain]

On Fri, Mar 12, 2010 at 10:56 PM, Bob Friesenhahn <

Regardless, when using JPEG format, the bottleneck is likely to be the JPEG
decode/encode rather than GraphicsMagick.

Does that imply I might be able to compile GraphicsMagick against an
optimized libjpeg to get better performance, or maybe even just a newer
libjpeg than whatever is on the system I'm using?

Mike

[Bob Friesenhahn]

Regardless, when using JPEG format, the bottleneck is likely to be the JPEG decode/encode
rather than GraphicsMagick.
Does that imply I might be able to compile GraphicsMagick against an optimized libjpeg to get better
performance, or maybe even just a newer libjpeg than whatever is on the system I'm using?

There are modified libjpeg's available which use MMX instructions. I
have heard that these can cut the decode/encode time by as much as
70%. I understand that Flickr is using the Japanese libjpeg with MMX
enhancements.

JPEG 7 and JPEG 8 are actually a bit slower than 6b.

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/