Why would you go to all that trouble?, just don't allow the file type in the attachments manager?
That said you can display an error message like this example, if you create a custom hook (custom_template_permissions)
HTML Code:
if (is_member_of($vbulletin->userinfo, 1))
{
print_no_ permission_ custom_ template();
exit();
}
if (is_member_of($vbulletin->userinfo, 8,9,10))
{
print_no_ permission();
}
In fact the easiest would probably be to edit the newattachment_errormessage template, remove the $errormessage and put your own text in there and/or conditions
These should help too
HOOKS CALLED
init_startup
fetch_userinfo_query
fetch_musername
fetch_userinfo
cache_permissions
fetch_foruminfo
style_fetch
cache_templates
global_start
parse_templates
notices_check_start
notifications_list
global_setup_complete
newattachment_start
attachdata_start
upload_accept
error_fetch
newattachment_attach
editor_wysiwyg_compatible
newattachment_complete
TEMPLATES CALLED
newattachment
newattachment_errormessage
newattachment_keybit
PHRASE GROUPS CALLED
global
posting
prefix