The Imagine plugin

The Imagine plugin for CakePHP, which has been available from the CakeDC account on GitHub, was written as a separate plugin, but complementary to my FileStorage plugin.

In the past, the good old CakePHP 1.3 times, we’ve had our Media plugin at the CakeDC which we never released. The main purpose of the plugin was to deal with file uploads, but only to the local system and to process images.

I’ve had to work with a few client projects that used phpthumb, which I’ve learned to dislike because it had its faults and issues. Before reinventing the wheel I simply tried to ask if somebody knows about something modern, OOP, PHP 5+, with unit tests, and had some luck.

Imagine is a modern PHP 5.3 library that provides an interface to different image processing back-ends like gd, imagick and imagick shell. Others can be implemented as well. I really recommend this lib over phpthumb, and suggest you to replace it with Imagine if your project is moving on.

The plugin for CakePHP is basically a wrapper around the Imagine library, that will autoload the library (because CakePHP 2.0 does not have it’s own autoloader until 3.0) and make it available as a behavior. The behavior provides some methods that handle commonly used image operations, like cropping and thumbnails.

Besides that, the plugin comes with a helper and a component, which allows it to generate versions of images on the fly. This was more done for backward compatibility for some apps, rather than being a concept that should be used. In fact, it is not the recommended way to handle this. I’ve written two other articles related to image processing and file storage that explain the issues with that.

Instead of creating images on the fly, which is just putting load on the server all the time, you should create the versions of an image after upload. The Imagine plugin works together with the FileStorage plugin, which can use Imagine to generate whatever images you want, right after upload.

I originally contributed this plugin, which I developed in my own free time, to CakeDC. But now, it is finally going back to my GitHub account, to pull some maintenance work away from the company. I’ll continue to maintain it, and hope you take some time to check it out, and contribute if possible.

Computer vs Car – A 2013 Rematch

For more then a decade there have been jokes that compared cars with computers or to be more specific software. 2013 – I think it is time to recap this because software has evolved, cars – in my opinion – obviously not. It is a well known fact that adding complexity leads to more problems within a system. What happened to cars? Today’s cars are full of electronic assistance systems and none required gadgets like electronically adjustable mirrors and electronically controlled locks and electronic ignition.

Besides the fact that you can today hack the electronic locks with ease they’re horrible expensive to replace – 300 euro for a broken door lock. What the..!? While speaking about security…

You can access the [1] CAN bus from an electronically controlled side-mirror and gain access to the complete electronic infrastructure system of the car.
Remote-locks are easy to hack, even for non computer guys. Who ever is capable of searching the internet and reading and is not a retard can do this.

So to summarize this, modern cars:

  • Require special tools for maintenance and building
  • Are hard to maintain
  • Are expensive to repair because they are hard to maintain
  • Are a pile of possible failure due to the amount of (unnecessary) electronic gadgets
  • Have parts and components that aren’t exchangeable with other cars, very often not even within the same series and manufacturer
  • Electronic locks and ignition systems are a huge security issue
  • Don’t have fallbacks (i.e. the electronic mirrors, locks or ignition breaks you can not still use them mechanically)

How a car could be build based on best practice and principles of modern software architecture

  • All screws, bolts and whatever else is needed in the car must be assemble with standard tools that are international available
  • The engine hold must be constructed in a way that different engines can be mounted inside so that the engine can be simply replaced by another more powerful or new engine
  • All parts of the whole care must be easy to access and maintain
  • Form follows function but should be aerodynamic – form follows function

Now if all of that could be done in a way all the parts, at least the frame of the card, would become manufacturable by using modern 3D printers, that might revolutionize how cars could be build, where they could be build and also result in lower prices. Just imagine to be able to drop a few containers containing the 3D printers somewhere , everywhere in the world and to be able to 3D print most parts needed for a car that can than be manufactured by any average mechanic! In fact [2] the US Army has already started doing this.

With the evolution of industrial level 3D printing this might become a cheap way to build cars in developing countries for a comparatively cheap price for the people living there. But also in modern world countries these cars might become a good opportunity for people who don’t care about having a nice car to compensate something or simply don’t have the money and can’t afford a “better” car. Better in terms of failure probability and maintenance cost is anyway a highly doubtable point even for expensive cars. So yes, I think a maintainable “open source” car would be even better than expensive high end cars that you can’t fix without special knowledge and tools.

Can this become a reality? Well, I hope so, I would buy it.

[1] http://en.wikipedia.org/wiki/CAN_bus
[2] http://www.bbc.co.uk/news/technology-20269645

Image saving and processing for high traffic sites

I think everyone has already seen code similar like this:

What’s wrong with this? Let’s assume you have an image library behind the Thumb helper. It will process the image supplied in $image. It will try to create the thumbnail, opening a file if it does not already exist at the destination and start processing the image. Once complete, it will write it out to the – as long as process takes – locked file.

The issue with this is the race condition in which another user attempts to load the same page, before the initial image resize and write process has completed. The helper will do exactly as it is told, and attempt to create the image thumbnail as it doesn’t yet exist. Consider a very busy site, and you can see that this would quickly bring a server to its knees.

This is not as much of a problem when you have just a few visitors per day, or per hour. However, if you have lots of visitors and your recently uploaded article appears on your homepage, you can be sure this thumbnail is going to take a lot of hits and the described scenario is very likely to happen.

So, when should you create thumbnails?

Usually you should know the desired image dimensions for all the images on your website. By this you’ll be able to get a list of sizes into an array. I personally prefer to put this list by model into a separate config file in APP/Config/thumb_sizes.php.

Assuming that you follow the MVC pattern, you’ll validate your uploaded form in the model. Once validated, the model performs any processing in the beforeSave() or afterSave() methods. This is an ideal location for the processing of images. You would begin by reading your configured sizes:

Show Plain Text

Run a foreach over the image sizes and create the thumbnails for your image after it was successfully uploaded.

How would you store or name your images?

How you store the images exactly is up to you but it is recommended to store them in a deeper nested folder structure like /images/users/6k/51/d3/ to get around performance issues with file system limits and slowdowns. Check my very old article about the media view and file storage for that.

To avoid the need to store all the thumbnail paths, it’s recommended to store the files in a syntax like this:

Basically you store all the processing you’ve done in the filename string. This will also allow you to delete a certain type of processed image later without much effort.

To get some additional performance you could store images that do not have to be secured inside the webroot, such as:

You should hash the whole filename using md5 or sha1 and not store it in the form like it is shown above! The filenames above were just choosen to be more clear to get a better understanding of how to store the different versions of an image.

If you now think you need to protect user avatars I think protecting them from appearing in search engines a proper entry in your robots.txt should solve this problem for you.

Of course, this is not an ideal mechanism for protected against sensitive and personal information such as medical images or data. If you need to protect images you’ll need to go the whole way through the CakePHP dispatcher and check the requested image for permissions and send it to the browser using the good old Media view. And do not store them in the webroot!

How do you generate new images?

Simple answer: Write a shell for that task.

It might take a long time to generate all the new images depending on the amount of images and changes you want to apply. It’s strongly recommended to set the priority of this process to a very low priority so that more important tasks like serving your site won’t become affected by this. As an alternative you can simply limit the process also to just a limit of using not more than X% of the processing power. You can use the tools of your OS to archive this.

Another solution is to push them off to a worker process on a separate machine, enabling a larger, flexible and scalable solution for your web application. This is something we’ll talk about in another article.

What if I have to generate images on the fly?

Well, if you really have to, you should secure them at least. People can do funny things with links like this:

Just write a tiny script that will increment the width and height in one pixel steps and the script on the server will generate a large amount of images within a very short time. It will either lock your server or fill your storage device over time.

The solution is, same as for form submissions, to secure them with a hash. This requires a helper that will generate a hash with a salt and attach it to the image url.

The controller that will process the request will need to check the params, rebuild the hash from the params (excluding the passed hash), and check the hash build in the controller against the passed hash in the url. If they don’t match somebody modified the url.

The Imagine plugin – everything in a plugin

While phpThumb is highly successful and widely used, it has its own issues and quirks. I was looking for a powerful modern alternative and found Imagine. Imagine is a PHP 5.3+ image processing library that is well tested and has a nice interface.

I’ve written a CakePHP plugin for Imagine that will do the processing for you and it also contains a helper that will build secured urls for on the fly thumbnail generation. The actual image processing happens in the model by the attached Imagine behavior.

https://github.com/burzum/cakephp-imagine-plugin

It’s fresh out of the oven and ready to use! Feel free to provide feedback on the plugin, fork it on Github and contribute through pull requests. If you are using this in your project, drop me a line and let me know!