That’s good.

“Oh no”, you think, “A utf-8 encoding problem.” That three-letter combo should be a single close-quote, like this:

That’s good.

Sometimes the problem is that your application is reading the file as a win-1252 (or cp1252, or the kinda-sorta iso-8859-1 used on the web). In this case the solution is easy: instruct your application to reopen the file as utf8.

But sometimes, your file really does say “’”, even when decoded as utf-8. How this happens is that someone took some utf8 text, pasted it into a win1252 document, and then saved the document as utf8. So now the bytes in your document are:

That[c3][a2][e2][82][ac][e2][84][a2]s good

instead of

That[e2][80][99]s good.

So how do you fix it?

I wrote a tool.

The Python code below uses Python’s codec interface to register a simple stateless encoder that turns these utf8 gremlin bytes back into pure utf8 bytes. You can use it from the command line like removeUTF8Gremlins.py infile.txt -o outfile.txt or you can use it as a library by importing it and then using the CP1252asUTF8gremlins pseudo-codec anywhere you can use a stateless codec.

#!/usr/bin/env python
# encoding: utf-8

# BSD LICENSE
# Copyright (c) 2010, Dancing Mammoth Inc
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# Redistributions in binary form must reproduce the above copyright notice, this
# list of conditions and the following disclaimer in the documentation and/or
# other materials provided with the distribution.
#
# Neither the name of Dancing Mammoth nor the names of its contributors may
# be used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

"""
removeUTF8Gremlins.py

Will recode file with utf8 gremlins to a proper utf8 file.

When used as a library, will register the codec 'CP1252asUTF8gremlins', which
provides a stateless decoder which will convert bytes with gremlins into pure
utf8 bytes.

We call a 'utf8 gremlin' a byte sequence that results when a utf8 byte is read
as a cp1252 encoding into unicode chars, and then written out as utf8.

The tell-tale sign of it is bytes that look like this in a file read as utf8.

Original: That’s good.
Bytes as utf8: That[e2][80][99]s good.
When read as CP1252: That’s good. (acute lowercase a, euro symbol, trademark symbol)
Bytes as utf8 gremlins: That[c3][a2][e2][82][ac][e2][84][a2]s good.

This utility turns "Bytes as utf8 gremlins" back into "Bytes as utf8"

Created by Francis Avila on 2010-10-27.
Copyright (c) 2010 Dancing Mammoth, Inc. All rights reserved.
"""

import sys
import getopt
import codecs
import re

help_message = '''
Fix a conversion error where a utf8 file got interpreted as a win1252 file
and then saved as utf8, producing three-character multibyte gremlins.
'''

def win1252_to_utf8_gremlin_table(mapping={}):
	if mapping:
		return mapping
	def makemapping(mapping):
		for i in range(256):
			byte = ('%02x' % i).decode('hex_codec')
			try:
				cp1252uni = byte.decode('cp1252')
			except UnicodeDecodeError:
				cp1252uni = byte.decode('iso-8859-1')

			if cp1252uni:
				realutf8 = cp1252uni.encode('utf-8')
				try:
					asuni = realutf8.decode('cp1252')
				except UnicodeDecodeError:
					asuni = realutf8.decode('iso-8859-1')
				if asuni:
					utf8gremlin = asuni.encode('utf8')
					mapping[utf8gremlin] = realutf8
	makemapping(mapping)
	return mapping

def win1252_to_utf8_gremlin_re():
	mapping = win1252_to_utf8_gremlin_table()
	rechars = []
	for k,v in mapping.items():
		if k != v:
			rechars.append(k.encode('string_escape'))
	regex = '(?:%s)' % '|'.join(rechars)
	return re.compile(regex)

def reverse_win1252_to_utf8_gremlins(bytes, errors='strict'):
	regex = win1252_to_utf8_gremlin_re()
	mapping = win1252_to_utf8_gremlin_table()
	def replace(mo):
		try:
			newchar = mapping[mo.group(0)]
		except KeyError:
			if errors=='strict':
				raise ValueError('Encountered bytes with no pure utf8 equivalent.')
			else:
				if errors=='ignore':
					newchar = ''
				elif errors=='replace':
					newchar = '?'
		return newchar
	newbytes = re.sub(regex, replace, bytes)
	return (newbytes, len(bytes))

def register_win1252_to_utf8_gremlins(encoding):
	ci = None
	if encoding == 'cp1252asutf8gremlins':
		ci = codecs.CodecInfo(None, reverse_win1252_to_utf8_gremlins, name='CP1252asUTF8gremlins')
	return ci

codecs.register(register_win1252_to_utf8_gremlins)

class Usage(Exception):
	def __init__(self, msg):
		self.msg = msg

def main(argv=None):
	if argv is None:
		argv = sys.argv
	options = {}
	try:
		try:
			opts, args = getopt.getopt(argv[1:], "ho:v", ["help", "output="])
		except getopt.error, msg:
			raise Usage(msg)

		# option processing
		for option, value in opts:
			if option == "-v":
				options['verbose'] = True
			if option in ("-h", "--help"):
				raise Usage(help_message)
			if option in ("-o", "--output"):
				options['outputfile'] = value

	except Usage, err:
		print >> sys.stderr, sys.argv[0].split("/")[-1] + ": " + str(err.msg)
		print >> sys.stderr, "\t for help use --help"
		return 2

	bytes = file(args[0], 'rb').read()
	outfp = file(options['outputfile'], 'wb') if 'outputfile' in options else sys.stdout
	bytes = bytes.decode('CP1252asUTF8gremlins')
	outfp.write(bytes)
	outfp.close()

if __name__ == "__main__":
	sys.exit(main())

The introduction of a completely new design can be aggravating for existing users, particularly when features that they’ve taken the time to learn are suddenly relocated, or removed altogether.

To avoid disorienting users, we reused the existing site format while updating the visual design and improving elements such as spacing and contrast. This way, Reason’s readers benefit from the enhanced features made possible by the new publishing system, without the inconvenience of relearning how to use the site.

For example, Reason, like many other websites, structures content with a static three column layout. The left hand column is dedicated to navigational links, masthead information and advertising. The center column contains the featured content well and links to categorized content, and the right column promotes additional highlighted content, with space for advertising.

The old version of the site was well organized with plenty of white space, but the design was flat and lacked visual interest and depth. The new design is more appealing, has a more natural visual hierarchy, and does a better job of keeping content separated without feeling unnecessarily boxy.

We sent over two design variations, making sure to reuse their currently branded blue whenever possible. Their feedback follows…

“Can I see this with a completely different color scheme?”

Hearing this might make a designer cringe, but it’s very important to make sure the client is happy.  Fortunately, the project was still in the early stages, so alterations were relatively easy to make.

After a brief consultation, we agreed to work on some new variations while keeping within the same basic palette. This would mean shifting the use of color around while adding shades and tints to create more variety. This type of consultation helps keep everyone happy by taking the middle ground. We aren’t creating designs with wildly conflicting colors, and the client doesn’t have to hear a flat out refusal on our end.

We submitted the above mockups, two of which rotated color usage in sections like the highlight column (bottom right corner), and one of which used a monochromatic scheme (third mockup from the left).

When only one color is used for a design, the overall feel can often be flat or what I’d consider “overly emotional”. A design that only uses red can make you feel too hectic or heated without a cool compliment for balance. Similarly, a design that only uses blue would make you feel too serious or cold without warm oranges or reds. When it came time to show our client the revisions, we made sure to bring this up as a possible issue.

In the end, they went with the primarily orange/gold design, giving the site a bright and light feeling while still leaving room to use the complimenting blue for their logo and headings.

Here is the map.

The benefit of using HTML, CSS and Javascript instead of Flash, is that the map will function in just about any browser without having to install additional components. Mobile browsers such as Safari on iPhone, older browsers, or browsers without Javascript enabled, can still use the map. Here is how it works.

The map consists of three layers.

The bottom layer contains the full map as a background image. This loads when the page is opened, along with the top layer which contains the maparea linked to a transparent GIF that matches the size of the map. The layers are positioned over one another with CSS.

div#usmap{
position: absolute;
top: 0px;
left: 0px;
}

div#mapareas{
position: absolute;
top: 0px;
left: 0px;
z-index: 99;
}

The middle layer is where all of the animation takes place. When a user mouses over one of the mapareas, jQuery prepends the preloaded image for an individual state and positions it in the middle layer. Some images also have a corresponding mask to maintain the illusion that the state is popping up away from the page. The individual state images were created by cutting apart the main US image, and the positioning is done with jQuery’s css method. This can be somewhat time consuming for complex maps.

$("#usmap").prepend(img);
$("#usmap").prepend(imgmask);
positionimage();

.state-copy {
position: absolute;
z-index:2;
}
.state-mask {
position: absolute;
z-index:1;
}

Then jQuery calcuates the current size and zoomed size, and executes the animation.

$(".state-copy").each(function(){
var width = $(this).width();
var height = $(this).height();

var zoomheight = height * 1.2;
var zoomwidth = width * 1.2;

var	centerheight = (zoomheight - height)/2;
var	centerwidth = (zoomwidth - width)/2;

$(this).animate({
top: '-=' + centerheight,
left: '-=' + centerwidth,
width: zoomwidth,
height: zoomheight
}, 100);

});

This method will work for just about any similar image that you want to animate.

It was a great plan, but I noticed that I occasionally got garbage lines or bogus input. Using http cli tools like wget and curl revealed nothing out of the ordinary, until I realized that those garbage lines were the uninterpreted length markers for Transfer-Encoding: chunked. PHP’s http stream handler does not decode chunked transfers.

There is a pecl function http_chunked_decode(), but it operates on strings, not streams, so I would still have to buffer the entire input first.

PHP’s streams allow you to attach a chain of stream filters to a stream to process input and output (it’s the same mechanism ob_gzhandler() uses). My plan was to create a stream filter to transparently unchunk the stream. Unfortunately, the documentation on writing your own stream filter is pretty sparse, and the examples I could find on the web were all very trivial.

After a few false starts, however, I was able to create an http stream unchunker:

/**
* A stream filter for removing the 'chunking' of a 'Transfer-Encoding: chunked'
* http response
*
* The http stream wrapper on php does not support chunked transfer
* encoding, making this filter necessary.
*
* Add to a file resource with <code>stream_filter_append($fp, 'http_unchunk_filter',
* STREAM_FILTER_READ);</code>
*
* If the wrapper metadata for $fp does not contain a <code>transfer-encoding:
* chunked</code> header, this filter passes data through unchanged.
*
* @license BSD
* @author Francis Avila
*/
// Stream filters must subclass php_user_filter
class http_unchunk_filter extends php_user_filter {
	protected $chunkremaining = 0; //bytes remaining in the current chunk
	protected $ischunked = null; //whether the stream is chunk-encoded. null=not sure yet

	// this is the meat of the filter.
	// The class must have a function with this name and prototype
	// It must return a status--one of the PSFS_* constants;
	function filter($in, $out, &$consumed, $closing) {
		if ($this->ischunked===null) {
			$this->ischunked = self::ischunked($this->stream);
		}
		// $in and $out are opaque "bucket brigade" objects which consist of a
		// sequence of opaque "buckets", which contain the actual stream data.
		// The only way to use these objects is the stream_bucket_* functions.
		// Unfortunately, there doesn't seem to be any way to access a bucket
		// without turning it into a string using stream_bucket_make_writeable(),
		// even if you want to pass the bucket along unmodified.

		// Each call to this pops a bucket from the bucket brigade and
		// converts it into an object with two properties: datalen and data.
		// This same object interface is accepted by stream_bucket_append().
		while ($bucket = stream_bucket_make_writeable($in)) {
			if (!$this->ischunked) {
				$consumed += $bucket->datalen;
				stream_bucket_append($out, $bucket);
				continue;
			}
			$outbuffer = '';
			$offset = 0;
			// Loop through the string.  For efficiency, we don't advance a character
			// at a time but try to zoom ahead to where we think the next chunk
			// boundary should be.

			// Since the stream filter divides the data into buckets arbitrarily,
			// we have to maintain state ($this->chunkremaining) across filter() calls.
			while ($offset < $bucket->datalen) {
				if ($this->chunkremaining===0) { // start of new chunk, or the start of the transfer
					$firstline = strpos($bucket->data, "\r\n", $offset);
					$chunkline = substr($bucket->data, $offset, $firstline-$offset);
					$chunklen = current(explode(';', $chunkline, 2)); // ignore MIME-like extensions
					$chunklen = trim($chunklen);
					if (!ctype_xdigit($chunklen)) {
					// There should have been a chunk length specifier here, but since
					// there are non-hex digits something must have gone wrong.
						return PSFS_ERR_FATAL;
					}
					$this->chunkremaining = hexdec($chunklen);
					// $firstline already includes $offset in it
					$offset = $firstline+2; // +2 is CRLF
					if ($this->chunkremaining===0) { //end of the transfer
						break;  // ignore possible trailing headers
					}
				}
				// get as much data as available in a single go...
				$nibble = substr($bucket->data, $offset, $this->chunkremaining);
				$nibblesize = strlen($nibble);
				$offset += $nibblesize; // ...but recognize we may not have got all of it
				if ($nibblesize === $this->chunkremaining) {
					$offset += 2; // skip over trailing CRLF
				}
				$this->chunkremaining -= $nibblesize;
				$outbuffer .= $nibble;
			}
			$consumed += $bucket->datalen;
			$bucket->data = $outbuffer;
			stream_bucket_append($out, $bucket);
		}
		return PSFS_PASS_ON;
	}

	protected static function ischunked($stream) {
		$metadata = stream_get_meta_data($stream);
		$headers = $metadata['wrapper_data'];
		return (bool) preg_grep('/^Transfer-Encoding:\s+chunked\s*$/i', $headers);
	}

	function onCreate() {
		if (isset($this->stream)) { // This is usually not defined until the first filter() call.
			$this->ischunked = self::ischunked($this->stream);
		}
	}
}

stream_filter_register('http_unchunk_filter', 'http_unchunk_filter');

What you are left with is a stream filter you can then use like so:

$fp = fopen('http://my.url', 'r');
stream_filter_append($fp, 'http_unchunk_filter', STREAM_FILTER_READ);

If the http stream has a chunked transfer encoding, the filter will automatically unchunk it. However, it ignores extended data (anything after the hex-encoded chunk-length) and trailing headers, both of which are in the http specification but hardly ever used.

I’ve been doing web development on and off for more than a decade. During college, I was the webmaster of the University of Minnesota’s computer science department, where I developed a variety of web applications and gave our department website a fresh new look (recently replaced by an even fresher, newer look). I got lots of practice with Apache, Perl, JavaScript, MySQL, and the usual alphabet soup of three- and four-letter acronyms. More recently, I’ve gotten into web development using Python and the Django framework.

After college, I worked as a journalist, blogger, and policy analyst, writing for Ars Technica, the New York Times, Slate, the Cato Institute, and various other places. Writing prose is fun, but after a few years I started to miss writing code and decided to go back to grad school.

For years, PJ has provided web design and hosting services to a number of my favorite blogs (including one I contribute to), magazines, and non-profits and I’ve always been impressed by his minimalist design sense and bulletproof hosting methodology. So I was excited when he invited me to join the team for the summer. I’m expecting to learn a lot about web development while helping clients solve their problems.

• PHP 5 (OOP skills a requirement)
• MySQL 5
• XHTML
• CSS
• JavaScript
• JQuery
• Python experience a plus.

The right candidate will fit right in with our fast-paced virtual office environment and be able to hit the ground running with our varied base of clients and growing stable of in-house initiatives. We’re preferential to applicants in or around Chicago, Washington D.C., or Reston, VA but will consider other areas for the right person. Starting salary will be commensurate with experience, talent, and knowledge of the movie Xanadu.

Interested? Feel compelled to make your mark? We’d like to hear from you! Send your resume to careers@dancingmammoth.com.

For reference, here is a screencap of Apple’s site:

Which brings me to two points:

1. When you have your own iconic brand, you really shouldn’t need to steal so shamelessly.
2. It’s 2009, Gibson. Did you really think table-based layout was still a good idea?

To help explain what we mean by color distribution, take a look at this little map we created using IWF.org as a base.

On occasion, we start layouts by grabbing all the content the client would like to see on the main page, making it black, and dumping it all into a new Photoshop file. Next, we make textual hierarchy changes by pumping up the weight and size for headings and subheadings. After some spacing and alignment changes its ready, but start collecting color only after these steps have been completed

First, we establish any existing colors which will need to be incorporated into our palette due to existing branding. In the example using IWF.org, the red from the logo needed to be added.

We needed to pay attention to an additional client request, which was that we keep the site feeling open and bright. Yellow or yellow used as a gradient is good for this.

You can add two colors to the palette by choosing one with a rich hue (aka saturation), in this case blue, then branch off by grabbing a tint of the same hue.

We also always grab something fairly neutral to use as borders, or for headings and backgrounds which require less emphasis than areas with brighter hues. For IWF.org, we chose a medium gray that is used on all borders and dividing lines.

Now the color palette is ready. Most, if not all of your color usage will be dictated by the palette. You can deviate, but only by shifting the base hue with shades and tints. Start by identifying which elements should use the same color. For example, all links within the main content area need the same color. All first-level headings need the same color as well.

Now you can experiment by using the colors in your palette in these positions. Any colors that are too light can be used for text as long as there is a background behind it with good color contrast to make everything readable.

Remember to use white as a color as well, not just for backgrounds. Any of the colors in your palette can be used for a background, which means any text on top of it can be white (depending on readability).