optimize_thumbnail post_processor doesn't work with alternative file storages like S3

[damianmoore]

We added the THUMBNAIL_OPTIMIZE_COMMAND setting to one of our projects. Locally, we uploaded images, tested the ouputted thumbnails by saving and running again through jpegoptim. This showed that optimisation was working as the images could not be losslessly compressed any further.

We then deployed to our staging environment which has DEFAULT_FILE_STORAGE set to use S3 via Boto. On this environment we noticed that the thumbnails were failing to optimize.

Looking briefly at the post_processor I can see thumbnail.path gets used primarily and I think this does not exist for object-based storage like S3.

[jonathanherzog]

Exactly right. thumbnail.path throws an NotImplemented exception when the underlying storage is an S3BotoStorage. Thus optimize.post_processot.optimize_thumbnail() throws NotImplemented in line 41, and the function returns (line 43) without doing anything.

I have managed to get it to work with a few simple changes:

def optimize_thumbnail(thumbnail):
    '''Optimize thumbnail images by removing unnecessary data'''
    logger.debug("starting optimize_thumbnail")
    try:
        logger.debug("thumbnail.name: %s", thumbnail.name)   
        thumbnail.seek(0)     
        file_extension = determinetype("", thumbnail.read())
        logger.debug("file_extension: %s", file_extension)
        optimize_command = \
            settings.THUMBNAIL_OPTIMIZE_COMMAND[file_extension]
        if not optimize_command:
            logger.info('optimize_command not found')
            return
    except Exception:
        logger.exception("error raised while computing optimize_command")
        return
    logger.debug("optimize_command: %s", optimize_command)
    storage = thumbnail.storage
    try:
        with NamedTemporaryFile() as temp_file:
            thumbnail.seek(0)
            temp_file.write(thumbnail.read())
            temp_file.flush()
            optimize_command = optimize_command.format(filename=temp_file.name)
            logger.debug("optimize_command: %s", optimize_command)
            output = check_output(
                optimize_command, stderr=subprocess.STDOUT, shell=True)
            if output:
                logger.warn(
                    '{0} returned {1}'.format(optimize_command, output))
            else:
                logger.info('{0} returned nothing'.format(optimize_command))
            with open(temp_file.name, 'rb') as f:
                thumbnail.file = ContentFile(f.read())
                storage.delete(thumbnail.name)
                storage.save(thumbnail.name, thumbnail)
                logger.debug("Finished.")
    except Exception as e:
        logger.exception('problem optimizing the file')
        logger.error(e)

As you can tell, I added a bunch of logging statements to help me track down the problems. The real changes are

1. Passing a byte-stream to imghdr.what() instead of a file name:

        thumbnail.seek(0)     
        file_extension = determinetype("", thumbnail.read())

2. Storing the optimized file by name and not path:

                storage.delete(thumbnail.name)
                storage.save(thumbnail.name, thumbnail)

I have confirmed that this works when the storage is S3BotoStorage (easy-thumbnails 2.0.1, django 1.5.1). I have not confirmed that this works for any other storage or version.

Cheers.