June 16, 2004

Forcing Comment Previews

I’m only about 6 months behind in writing this up.

Back when I first implemented Comment Validation, I relied on a very simple bit of social engineering to ensure that comments got run through the Validator before posting. Near the <textarea> where you enter/edit your comment, there’s a PREVIEW button, but no POST button. The POST button only appears on the Comment Preview form when you’ve previewed the comment, and it has successfully Validated.

If you then scroll down and edit the comment, you are supposed to click on PREVIEW again. But if you actually look at the code I suggested for the Comment Preview page, you quickly realize that nothing prevents you from scrolling back up and clicking on POST instead. I ran with this system for 7 months without a single invalid comment being posted. But just because nobody’s yet come around and rattled the door handle, there is no reason to leave the backdoor perpetually unlocked.

Besides, spambots and crapflooding scripts bypass the comment form entirely, and POST directly to the comment CGI script. They’re immune to social engineering. A better solution was called for.

So, 6 months ago, I created a plugin to solve the problem. Introducing MTHash.

Installation is easy:

1. The plugin’s only prerequisite is the Digest::SHA1 Perl module.
2. Put the plugin in your plugins directory.
3. Create a text file called salt.txt with some random gibberish in it, and place that, too, in your plugins directory.

The plugin offers two new MT Container Tags

<MTSHA1Hash>...</MTSHA1Hash>
Replaces its contents by an SHA1 hash of the contents.
<MTSHA1SaltHash>...</MTSHA1SaltHash>
Replaces its contents by an SHA1 hash of the contents salted with the aforementioned salt.txt file’s contents. Unlike the previous one, this can’t be pre-computed without access to the salt.txt file.

OK, so how do we use it?

1. In the Comment Preview form, we add a hidden form field
<input type="hidden" name="validated" value="<MTSHA1SaltHash><MTCommentPreviewBody convert_breaks='0' sanitize='0'><$MTEntryID$><MTCommentPreviewIP><$MTCommentPreviewAuthor$><$MTCommentPreviewEmail$><$MTCommentPreviewURL$><$MTCommentPreviewSubject$><$MTCommentPreviewTextFilter$></MTSHA1SaltHash>" />
(note that the MT tags in green, for the subject and text-filter of the comment, are special to my setup; you might need to omit them both here and below).
2. Then we modify lib/MT/App/Comments.pm
--- lib/MT/App/Comments.pm.orig Thu Jul  1 15:29:58 2004
+++ lib/MT/App/Comments.pm      Sun Aug  1 23:34:52 2004
@@ -229,6 +277,24 @@
if (!$q->param('text')) { return$app->handle_error($app->translate("Comment text is required.")); } + require Digest::SHA1; + my$sha1 = Digest::SHA1->new;
+
+    $sha1->add($q->param('text') . $q->param('entry_id') .$app->remote_ip
+               . $q->param('author') .$q->param('email') . $q->param('url') .$q->param('subject') . $q->param('convert_breaks') ); + my$salt_file = MT::ConfigMgr->instance->PluginPath .'/salt.txt';
+    my $FH; + open($FH, $salt_file) or die "cannot open file <$salt_file> ($!)"; +$sha1->addfile($FH); + close$FH;
+
+    my $digest =$sha1->b64digest . "=";
+
+    if ($q->param('validated') ne$digest) {
+    return $app->handle_error($app->translate(
+    }
+
my ($comment,$commenter) = _make_comment($app,$entry);
if (!$blog->allow_unreg_comments) { if (!$commenter) {

That’s it! If you modify your comment, now you must preview it again before posting. [Revised (8/1/2004) to be even more bulletproof. Revised (9/1/2004) for MT-3.1 compatibility.]

As a side benefit, spambots, which post directly to the Comment CGI script don’t work any more. Some of my friends who’ve been using this technique (without the Validation part) tell me it works wonders against spambots.

I wouldn’t know; I don’t get comment spam anymore (so far, only 15 spam comments in over 8 months).

Posted by distler at June 16, 2004 9:59 AM

TrackBack URL for this Entry:   http://golem.ph.utexas.edu/cgi-bin/MT-3.0/dxy-tb.fcgi/383

Re: Forcing Comment Previews

Thanks a lot for this work. I am going to implement it today, since I can’t use MT-Blacklist anymore now that I use MT3.0D.

(I have to admid that I tried it out by modifying my post in the preview window and clicking the Post button, but it works flawlessly)

Posted by: Jeroen on June 16, 2004 10:17 AM | Permalink | Reply to this

Comment Spam

Jeroen,

You might take a serious look at my series of posts on Comment Spam, linked to above. The techniques outlined there have proven remarkably effective in the 8 months I have been using them.

Posted by: Jacques Distler on June 16, 2004 10:24 AM | Permalink | PGP Sig | Reply to this

Re: Comment Spam

I did read these posts, and have taken my measures against comment spammers. However, once in a while a spammer gets through, so I am always looking on raising the bar.

Posted by: Jeroen on June 16, 2004 11:09 AM | Permalink | PGP Sig | Reply to this

Re: Forcing Comment Previews

Nice plugin, Jacques.

It’s too bad that there’s not a way to achieve this without patching the MT codebase, though. It sure would be nice if MT plugins could be as simple as dropping a single file into place.

Posted by: Scott Johnson on June 17, 2004 1:38 AM | Permalink | Reply to this

MTHash

The plugin, itself, is trivial to install. And it’s useful for for other stuff like … umh … I’m sure someone will think of something.

As to its use for forcing Comment Previews, I don’t think there’s a way to plug into MT::App::Comments->post at precisely the right place, without hacking the source code. (If I’m wrong about this, I’d be happy to learn it.)

Instead of patching the source, one could go the route of replacing the entire post method (as does MT-Blacklist). But that leads to even more headaches down the line.

Posted by: Jacques Distler on June 17, 2004 2:08 AM | Permalink | PGP Sig | Reply to this

Re: MTHash

I don’t think there’s a way to plug into MT::App::Comments->post at precisely the right place, without hacking the source code.

Not just yet. MT 3.01 or 3.1 will have hooks all over the place, and let a plugin return an error that does more than just put something in the activity log, but for now if you hook comment save all you can do is change what goes in the database.

Posted by: Phil Ringnalda on June 17, 2004 3:35 AM | Permalink | PGP Sig | Reply to this

Re: MTHash

Doesn’t the MT-Approval plugin does something similar without having to hack the MT code base.

Posted by: Zack on January 10, 2005 5:53 AM | Permalink | PGP Sig | Reply to this

Re: MT-Approval

Chad’s plugin forces un-previewed comments into a moderation queue. That’s fine as an anti-spam measure (his purpose). But I really want to force each comment to be run through the Validator (aka previewed).

So my plugin/hack works a little differently (and predates his plugin by a year).

Posted by: Jacques Distler on January 10, 2005 8:12 AM | Permalink | PGP Sig | Reply to this

Re: MT-Approval

Is that so? I thought you could disable the Post button if there was any edit of the comment during preview. Can’t the output of the validator used similarly? So the Post button is disabled if either the comment has been edited or if it’s invalid XHTML.

The reason I am considering MT-Approval instead of MT-Hash is that patching the MT code means I have to keep patching it if I upgrade MT (that will require figuring out where and what to patch). All the MT hacks I had installed are gone from my installation when I upgraded to MT 3.1. Plus since I am using MT-Blacklist which overrides the commenting function, I have to find where to modify the MT-Blacklist code for MTHash.

Could you be so kind to release MTHash as a pure plugin now that MT 3.1 is out?

Posted by: Zack on January 11, 2005 5:57 PM | Permalink | PGP Sig | Reply to this

Re: MT-Approval

How badly do you really need MT-Hash/MT-Approval?

What you really need is comment validation. The gentle persuasion involved in (re)moving the POST button is remarkably effective (with people, not 'bots). In the seven months between installing comment validation and taking this next step, I did not receive a single invalid comment.

It’s nice to know that, with my current setup, it’s impossible to submit an invalid comment. But, in practice, you may not notice any difference.

Can’t the output of the validator used similarly? So the Post button is disabled if either the comment has been edited or if it’s invalid XHTML.

That’s exactly the way my system works. Try previewing an invalid comment.

The crucial thing happens once you’ve previewed a valid comment. Now the POST button is enabled. What happens if you edit your comment and then click on the POST button?

To understand the difference between my plugin/hack and Chad’s, you can forget about all the stuff about POST buttons and comment forms.

The difference occurs on the server side, when a (previously previewed) comment is POSTed. If the hash fails to match, my hack sends you to the Comment Error page, where you are instructed to try previewing your comment again.

My recollection of the first release of Chad’s plugin was that it took the POSTed comment and dropped it in a Moderation Queue. But I decided to look at version 1.0, which is a good bit closer to what I want. It “throttles” the comment, and sends the user to the comment error page with the message

In an effort to curb malicious comment posting by abusive users, I’ve enabled a feature that requires a weblog commenter to wait a short amount of time before being able to post again. Please try to post your comment again in a short while. Thanks for your patience.

Not exactly what we want, but a big improvement.

Chad is hooking into the comment-throttle callback routine which doesn’t (currently) let you pass a custom error message back to the user.

I don’t think it’s acceptable to return such a blatantly incorrect error message to the user who added a word to their comment and then clicked the POST button.

Could you be so kind to release MTHash as a pure plugin now that MT 3.1 is out?

If 6A adds the ability to pass custom error messages back to the user from the comment-throttle callback, or if they add another callback I can use, I’ll certainly try to turn my hack into a plugin.

Plus since I am using MT-Blacklist which overrides the commenting function, I have to find where to modify the MT-Blacklist code for MTHash.

If I had sufficient chutzpah, I would do like Jay Allen and usurp the MT::App::Comments->post function with my own and then you wouldn’t need to patch anything. On the other hand, there’s probably a reason why his users will let him get away with that, whereas mine wouldn’t.

Seriously, though, I sympathize with your reluctance to patch the MT source code.

My omnibus patch file for MovableType 3.1.x runs to 511 lines (of unified diffs). I must have one of the more heavily-patched MT installations on the planet. But, for the most part, the code is mature enough that the vast majority of patches work unaltered between versions. Typing

patch &lt; ../mt314.patch

is the least tedious part of upgrading.

Posted by: Jacques Distler on January 11, 2005 10:11 PM | Permalink | PGP Sig | Reply to this
Weblog: 日々の移ろい
Excerpt: 下記のようなコメントが山のようにあった。 I enjoy your site very much! Keep up the good work! オンラインカジノ http：//casinos-jp.com コメントスパム と言うようです。 調べてみると、いろいろ対...
Tracked: August 11, 2004 9:53 AM
Weblog: 日々の移ろい
Excerpt: 下記で紹介されているコメントスパム対策を Movable Type 3.01D に適用する。 Musings: Forcing Comment Previews Going My Way：プレビューボタンのみ表示してhashを仕込むというコメントスパム対策 3.01D ...
Tracked: August 16, 2004 3:20 PM
Weblog: 33rpm
Excerpt: スパムコメント(宣伝文等のコメント)が最近多くて、 33rpmサーバを利用しているユーザからも、 対応等の要望があがってたので対応しました。(わー！わー！) で、僕個人だと対応しきれな...
Tracked: September 8, 2004 2:52 AM
Weblog: むさしのblog
Excerpt: コメントスパム対策にMTHashを導入してみた。結果は（これまでのところ）大成功。
Tracked: January 23, 2005 4:42 AM

Re: Forcing Comment Previews

FInally, I have installed MTHash. My MT patch file is also becoming large, though nowhere near yours.

If the comment preview does not have a subject or text filter, then the patch file shouldn’t have this:

$q->param('subject') .$q->param('convert_breaks')

Posted by: Zack on February 3, 2005 12:09 PM | Permalink | PGP Sig | Reply to this
Read the post Front-end and Back-end Changes
Weblog: Procrastination
Excerpt: There have been a lot of changes here recently, most of them on the back-end. Most of this work was related to having a bilingual (English and Urdu) blog along with MathML equations. This required valid XHTML 1.1 and serving...
Tracked: February 11, 2005 8:22 AM
Read the post コメントスパム対策　2 MTHash編
Weblog: 株主優待フラッシュ
Excerpt: コメントスパム対策として以前、MT-DSBLを導入しました。 　これは、Open...
Tracked: May 24, 2005 4:31 AM
Weblog: I sort my thought...
Excerpt: 前はmt-comment.cgiをリネームするという方法でスパムを弾いていたのですが、最近は5〓10件/日でコメントスパムが来るようになったんで、コメントプレビューをしないと投稿できなくすると...
Tracked: September 5, 2005 10:58 AM
Read the post Are You Being Served? -- Part II
Weblog: Upon Reflection
Excerpt: Step 3: MathML At this point you should have valid XHTML 1.1 pages served with the correct MIME type to all the good little browsers.... It is much easier to use some kind of shorthand to enter equations, and let Movable Type convert it to MathML when...
Tracked: September 30, 2005 1:02 PM
Read the post コメントスパム ( Comment Spam )
Weblog: Institute for bayani
Excerpt: コメントスパム (Comment Spam) 対策が、下記サイト にて紹介されて...
Tracked: July 6, 2006 8:18 AM
Weblog: Musings
Excerpt: Henri Sivonen bursts my bubble.
Tracked: November 25, 2006 2:34 AM
Read the post Movable Type and Unicode
Weblog: Procrastination
Excerpt: Running Movable Type natively in Unicode was not as difficult as I thought but it still required a number of patches to the code.
Tracked: November 28, 2006 1:12 PM