Hiding Thread content?
 

when i set my forum options to hide thread content from guests, will it hide thread content from spiders too?

Spiders are Guests, so Yes.

There is a cloaking mod, but I wouldn't advise using it.
As Lynne said, guest and spiders are treated the same

so what is the best way of allowing spiders to view thread content ie posts but not guests/visitors?

there is none, it's called cloaking and the SE don't like it ;)