Musings

As this comment form does not define an accept-charset value, user agents may interpret the default for this as the character encoding used to tranmit the form itself.

That makes perfect sense, as — ultimately — the comment is to appear on the blog individual archive page, which uses the same charset encoding as the form.

Does the problem occur because the form is POSTed with the “wrong” charset (and hence could be cured with an appropriate Accept-charset header)? Or does the problem occur with the charset used internally for copy/paste?

I was under the impression that it was the latter, which is why we are remapping characters server-side.

The 0x80 thru 0x9F range is defined as control characters in iso-8859-1.

Yes, thank you for that clarification.

I should have said that everything I’ve said assumes ISO-8859-1 is the charset in use (true for the vast majority of MT blogs). There will, doubtless, be other character mis-mappings if, say, you are using UTF-8 for your blog.

But I don’t have any experience with what breaks when commenters copy 'n paste on a UTF-8 encoded blog.

As this comment form does not define an accept-charset value …

Just so no one thinks me a shirker, I added an accept-charset attribute to this form. Specifically, I set

<form  method="post" accept-charset ="<MTPublishCharset>" ... >

This has absolutely no effect on the munging of smartquotes, at least not when POSTing from Mozilla.

(If you’re a masochist, and want to know why that’s the case, read bug 228779.)

An excellent idea.

I haven’t explored the new plugin callbacks yet. This sounds like an excellent application.

But I’m not so interested in a 3.0-only plugin, if that’s at all avoidable. Fortunately, you are also the leading expert on writing plugins for 3.0 which are backwards-compatible with 2.x.

So, if you want to collaborate on this, I’d quite like to do it.

You mean “ versus “? They are both valid representations of ‘ “ ’ and neither will give you a lick of trouble.

As noted, MTStripControlChars doesn’t really work with UTF-8 encoded blogs (something is needed, even for UTF-8, to filter out invalid data, but I haven’t settled on the best approach). Or you could be having troubles with another, unrelated, feature of text filters.

In recent version of Perl, the order of execution of text filters (which look like attributes to MT tags, e.g.

<$MTCommentPreviewSubject strip_controlchars="2" remove_html="1" smarty_pants="2"$>

) is random. In some instances, this can cause problems. To force a particular order of execution, you can use the MTBlock plugin:

<MTBlock smarty_pants="2"><$MTCommentPreviewSubject strip_controlchars="2" remove_html="1"$></MTBlock>

as discussed here.

Has anyone thought about using this plugin to convert all pasted-in special characters, like non-English characters?

Any characters entered into the comment form which are not in your declared charset will be automatically converted to numeric entitities by the browser when the form is submitted. Or, at least, that’s what the browser is supposed to do (aside from this common screwup between ISO-8859-1 and Windows-1252, but aside from that …).

Both hexadecimal and decimal character references are valid (X)HTML, and neither cause problems with the W3C Validator. If you have a page with hexadecimal character references which cause problems with the W3C Validator, please provide a link.

This page has hexadecimal character references on it and it validates just fine.

Works in the comments over here.

If you want it to work in comment previews, you need to set the attribute on <MTCommentPreviewBody>

p.s. I have Greater Than and Less Than signs around the above code, but seems to mess up posting. I tried adding around it.

Since I allow a quite extensive subset of HTML in your comments, I don’t do any escaping for you. If you want “<”, you need to escape it yourself, by typing “<” and so forth.

The short answer is that, somehow, your posts are being input using the UTF-8 charset, whereas your templates declare that your blog is written in the ISO-8859-1 charset. In between, you are filtering the output through my plugin.

The purpose of my plugin is to convert Windows-1252 (which Microsoft, hence most other browser makers, pretends is ISO-8859-1) to actual ISO-8859-1. It’s neither useful, nor necessary with UTF-8.

At the cost of, perhaps, boring you to tears, let me explain what happened to the “ ’ ” character you typed.

In Unicode, the “ ’ ” character is designated as U+2019. In UTF-8, that character is represented by a string of 3 bytes, 0xE28099. (I’m using hexadecimal notation, so each byte is represented by two hexadecimal digits, ‘E2’, ‘80’ and ‘99’.)

Let’s see what happens to each of those three bytes when your run them through my plugin and display the result as ISO-8859-1.

1. ISO-8859-1 is a single-byte charset. The byte 0xE2 is the character “â”.
2. The byte 0x80 is a control character. My plugin converts it to €, which is the character “€”.
3. The byte 0x99 is also a control character, and my plugin converts it to ™, which is the character, “™”.

Now, where my plugin would have been useful is if you had been using Windows-1252 (aka, what Microsoft pretends is ISO-8859-1). In Windows-1252, “ ’ ” is the single byte, 0x92, which is yet another control character in real ISO-8859-1. My plugin would have converted it to ’, which is the real way to write “ ’ ” in ISO-8859-1.

The bottom line for you is: use the same charset to publish your blog as you used to compose it. Use either UTF-8 or ISO-8859-1, but don’t mix them.

Clear?

It certainly does work with 3.17, as you can see on this blog.

It is not helpful with UTF-8. It is really intended for use with ISO-8859-1. But it “works” anyway.