Styles and Templates > Style Manager
And either click "Go" or select "Main CSS" from the drop down next to the style you want to edit.
Dotted borders are done in CSS..
For example:
HTML Code:
.classname {
background: #FFF;
border: 1px dashed #000;
}
The class is set to have a white background, and a border with a one pixel width, a dashed style (dotted border), and the hex code for whatever color you want the border to be.
Edit:
If you're wanting to add the dashed border to a class that already exists in vBulletin.. just enter the
border: 1px dashed #000; (of course change the width and color) in the "Extra CSS Attributes" box for that style.