I'm trying to hunt down the way to create a custom error at the moment.
However, using the example I have above with the newreply_post_start hook it does not work at all, for either advanced mode or quick reply.
could you show me how you created your error message, I'm digging around for a link and looking through code to no avail.
--------------- Added [DATE]1299692058[/DATE] at [TIME]1299692058[/TIME] ---------------
update: almost got this working now

just fighting with "cannot find phrase" error on a custom phrase for the error
--------------- Added [DATE]1299693536[/DATE] at [TIME]1299693536[/TIME] ---------------
got it working!
Thanks thunder, it was actually failing because I was testing using a header.