[mme42]

To make a certain forum open to guests, but they can't view thread content...

Go to your forum permissions (ACP > Forums and Moderators > Forum Permissions) and find the forum that you want to do this for, edit the "Unregistered / Not Logged In" group and set the following:

Can View Forum = Yes
Can View Thread Content = No

This will mean that they'll be able to see the titles of the threads (in the forumdisplay), but they won't be able to view what's inside of the threads. This will do this for the entire section though not just one thread. I'm not sure if you can do it for just one thread. There might be a plug-in for this, but I don't know.

As for the pop-up, if i understand you correctly, you could set up a notice in your Admincp.

Admincp > Notices > Add new notice

In the html box include a message and link to your registration page. Then tick the boxes:

-User belongs to usergroup: Unregistered / Not Logged (change from the dropdown)
-User is browsing forum: (select the forum here).