This was a little annoying…

I came across an interesting Apache quirk the week before last, it totally make sense why it happens and I was at first a little surprised, one because no one had noticed previously and secondly because it was happening at all.

We noticed that if a url like http://bob/file.php?id=$frank went to an apache then the dollar symbol got encoded, which is perfectly normal behaviour, it sees a special character it deals with it. In our case this was being trigged by a URL redirect from http to https. Something I thought was odd which I never got to the bottom of is why did it do the re-write at all? If the http to https rewrite rule was not there it just passes it through so it is a by product of the rewrite.

This in its self is fine, other than manipulation of the url should probably be an option to turn on rather than off but I guess that depends on how popular it is. Either way this can be stopped by simply telling it to not encode the URL with the [NE] flag on the end of the rule.

The annoying element of all this is no one noticed an issue, the application is able to un-encode a URL and to work with the non encoded URL and yet still things were not quite right.

It turned out with a bit of digging that if you sent in a URL of http://bob/file.php?id=%24frank apache ended up encoding the encoded URL resulting in a URL that looked like this – http://bob/file.php?id=%2524frank

I can understand that Apache doesn’t know it’s encoded already, but considering we only send out URL’s with $ in what on earth was causing it to go horribly wrong?

A bit of digging

It turned out that some web-based email service thought the best thing they could do to all URL’s is re-endcode them for you.

For example, Hotmail:

Gmail:

Can you see the difference? try clicking on the image for a more human readable one.

Not sure why our good friends at Microsoft decided it was a good idea to change peoples URL’s, there probably is one, but I’d like to think that Gmail is as complicated as Hotmail and they seem to have found a solution.

Much time of many people was spent working out how this issue occurred, but none the less it is resolved, I do feel a bit silly for not spotting the double encoding myself but at least now I know and you know that Hotmail does URL encoding and Gmail does not.