PDA

View Full Version : Administrative and Maintenance Tools - Link / URL Status Checker


bairy
12-16-2006, 10:00 PM
What does this do?
For anyone who has seen or got my Image Status Checker hack, this is exactly the same, but works on links ([url=], [url]) instead.

Basically it scans all your posts, extracts all the url tags, and scans each of the images to see if they're still valid.

The rest of this is basically the description of that hack reworded


Why?
I had a look at all the links on my site and was alarmed at how many were now gone. Since the only way you can check the links on your board is to manually read every post and click them, I decided to come up with a better way... and this is it.


How does it work?
The first part: In the AdminCP, under Maintenance and Update Counters... right at the bottom is this hack. It works by looking up every url tag, then requesting the image, and reading the http status code. So code 200 means 'image ok', 404/410 means 'image gone' etc. That then gets stored in a database table. A server has 15 seconds to reply to the request or the status is labelled as "Unknown"
The second part: The browsing element, linkstatuscheck.php (original filename huh!). This allows you to browse all the images found in the last scan using some powerful filtering (statuses to display, search, order by).


Hack features

General
Fully phrased.
Templates are grouped. Who's online handled.
Part 1 - Admin
Reads the post table, scans all the [url=] and [url] tags on demand and records the actual http status code returned.
If it gets stuck during the scan, you can restart the section it's currently doing.
If an link appears in more than one post, it's only checked once.
Start from, per page and timeout options for scanning.
Part 2 - Browser
Status codes are put into one of three descriptions for simplicity: Working, Dead, Unknown. Unknown is if the server didn't respond or similar - on the basis that a temporary timeout doesn't necessarily mean the image has gone.
In the browser, link urls are force wrapped. Unless people post using all caps, you have a low screen resolution, or the font size is big, the table should never stretch.
Filtering allows you to show just the working/dead/unknown images, and there's a search facility for a variety of fields.
Convenient link to edit the post (if a dead link is found). This works by can_moderate - edit links only appear for people who own the post, or can moderate the forum it's in.
Works by canview - if someone can't view a particular forum (e.g. staff forum) normally, they can't view the images within it.
Uses css for common stuff to reduce the size of the outputted pages.



Bad Things
It's far from a perfect hack, there are many things to do. Please be aware that I won't be doing them, but if anyone else wants a crack, feel free!

Only supports http://, not https://
Can only handle replies like: HTTP 1.x 200 as the first line.
Only supports [url] and [url=] tags. If you have HTML turned on in any forums it won't see <a href=> links.
Biggie: There's no way to update a single post or link without a full re-scan. That means if someone edits their post to update or remove a dead link, it will not change on the browser until a full re-scan is done. I did play with various update methods but most are flawed in one way or another.
No cron job.
No session variables. (People without cookies will be logged out a lot).



Footnotes
The code to the Image Status Check hack is very similar, so I pretty much copy pasted and adapted it. It is a little bigger due to handling the [url=] "option", but you may find references to images. Let me know if you do.


It will work on 3.6 and 3.5, though you'll need to remove the "executionorder=" from the .xml file to get it working on 3.5


Installation
Upload linkstatuscheck.php to your vB directory. Install the product, set overwrite to yes.


Customizing

By default it's set to only allow moderators, super-moderators and administrators to view the browser. This can be changed with the setting in AdminCP > vB Options.
The phrases all start with usc_ if you want to change them.
You can add a link to linkstatuschecker.php on the navbar (or anywhere) if you want your members to be able to view it.



Screenies
Exactly the same as the Image Status Checker, only with URL links rather than image links.
What? I'm feeling lazy.

bashy
12-17-2006, 06:30 PM
Hi

Great idea thanks, but in the Search for text area, what do i actually input there?

I cracked it now, Ya have to run the checker in the Maintenabce section in the update countser section :)
Then you use the browser part, is that correct?

Veer
12-17-2006, 06:31 PM
Thanks, :)
Its Very Useful for Me, Because My Site Have Lots n Lots of Images, Shared by Members.

Question / Suggestion: (for future update)
Is there any chance to check image / link status automatically and if a image link is dead then system auto send a PM to user who has the post with dead link?

Thanks again.

projectego
12-17-2006, 06:36 PM
Thanks, this sounds like a very helpful add-on! ;)

bashy
12-17-2006, 06:40 PM
looks like all my links are of unknown any ideas?

bairy
12-17-2006, 06:49 PM
I cracked it now, Ya have to run the checker in the Maintenabce section in the update countser section :)
Then you use the browser part, is that correct?
Yes, maintainence first, then the browser.


Question / Suggestion: (for future update)
Is there any chance to check image / link status automatically and if a image link is dead then system auto send a PM to user who has the post with dead link?
There probably won't be a future update. It was really just a hack to fill a need, rather than something I plan to develop further.
Besides, auto PMing would add a lot of strain to the server, and people can just look at the linkstatuscheck.php if they want to see which of their posts are in trouble.


looks like all my links are of unknown any ideas?
Well, assuming you know at least one of them actually works, I'd guess the server can't communicate with the remote sites. Perhaps fopen is disabled in php?

bashy
12-17-2006, 07:08 PM
allow_url_fopen is on just checked


Well, assuming you know at least one of them actually works, I'd guess the server can't communicate with the remote sites. Perhaps fopen is disabled in php?

bashy
12-17-2006, 07:09 PM
out of 3000+ links 3 are dead about 5 or 6 are working and the rest unknown!!

bairy
12-17-2006, 07:11 PM
I'm afraid I don't know then, I don't really know php's local/remote file commands all that well.
I can tell you it uses fsockopen to ping the server, then fwrite, fread to communicate.

bashy
12-17-2006, 07:14 PM
Well, i just installed ya image one as well and get the same, 99% of unknown status!
What does duplicate / dealt with mean? i get a lot of them
Also in the update counters, the 1st page seems to work well, all the rest give the duplicate / dealt

bairy
12-17-2006, 07:17 PM
If someone makes a post with a url in, and they make another post with the same url in, then there's no point pinging the site twice. So it'll ping it once, and any occurance after that will be labelled as already dealt with.
It also happens if you manually restart pages, or start scanning from an id other than -1

bashy
12-17-2006, 07:18 PM
Thats strange... cause i dont get it on the 1st page but all the rest i do!

If someone makes a post with a url in, and they make another post with the same url in, then there's no point pinging the site twice. So it'll ping it once, and any occurance after that will be labelled as already dealt with.
It also happens if you manually restart pages, or start scanning from an id other than -1

Bounce
12-17-2006, 07:20 PM
Working fine here , thanks :)

Does the IMAGE checker work on 3.6 ?

bashy
12-17-2006, 07:24 PM
Ah well, must be my site not wanting to play ball with both checkers, thanks anyway, but have removed now, the idea was there,, thanks again :)

bairy
12-17-2006, 09:02 PM
Working fine here , thanks :)

Does the IMAGE checker work on 3.6 ?

Yes. At least, on 3.6.0. I should imagine it working in the whole 3.6 range.

UltimateOreo!
12-18-2006, 01:32 AM
Why dont you check this into beta if there are so many issues?

bchertov
12-18-2006, 06:10 AM
This looks good!

Is it possible to query to width of the image as well?

Thanks!
Barry

bairy
12-18-2006, 12:34 PM
Why dont you check this into beta if there are so many issues?
Because putting it in Beta implies that I'll be updating it at some point, which is unlikely unless a bug is found and a workaround is easy, or it's a serious bug.
This is a "I use it, it works for me, others may as well use it if it works for them" hack.

Is it possible to query to width of the image as well?
I'm guessing you meant to reply to the image status checker, however I shall answer here.
I'm afraid I probably won't be including that. I'm not sure if vbulletin holds that info in any case. May I ask why you'd want it to, there might be another way to do it...

ymy
12-18-2006, 03:33 PM
Thanks ,

Install click

Hornstar
12-19-2006, 11:48 PM
This actually looks very handy. nice work.

yingzhou
12-20-2006, 09:29 AM
Can it work with the link for host free like rapidshare or megaupload? How can I know the file is deleted or not yet?

bairy
12-20-2006, 11:15 AM
Can it work with the link for host free like rapidshare or megaupload? How can I know the file is deleted or not yet?

Probably not, since when you go to file hosting sites, you're given a html page where you click to download the file. I would imagine the script could be made to "read" the page and follow the link, but it's not something I'll be doing.

yingzhou
12-20-2006, 11:56 AM
Thank, :D I think the you should make the mod can check both url and img link. The Mod will have 2 option to check URL or IMG link. That will be usefull.

bairy
12-27-2006, 09:08 AM
Updated to 1.01

- Browser: Corrected bug that said "you haven't scanned" even if you have. This only affects people with table prefixes.

To upgrade, reupload the /linksstatuscheck.php file. The product xml hasn't changed so there's no need for a re-import.

Terminatoronly
12-29-2006, 08:10 PM
nice hack but as u said that it search for url tag ok but in my site i dont use code tag does this hack check for it??????????

grazianno
01-03-2007, 03:23 PM
same questions,
can i search for [code] or something for another bbcode ?
thx

bairy
01-03-2007, 04:00 PM
No. [url] only

grazianno
01-12-2007, 03:11 PM
i got this error:

MySQL Error : Unknown column 'executionorder' in 'field list'

bairy
01-12-2007, 10:56 PM
3.5 instructions:
It will work on 3.6 and 3.5, though you'll need to remove the "executionorder=" from the .xml file to get it working on 3.5

Eagle Creek
02-16-2007, 03:27 AM
I like the idea, but what r u going to do about it? I mean; u can't delete thousands of posts do u?

bairy
02-16-2007, 10:22 AM
Whatever you like. The mod is designed to find the dead links for you as information. It's up to you what you do with it.
My suggestion would be to show the users the output screen and let them either update the link or delete the post.

alessai2
12-22-2007, 12:57 PM
um..can you make it check everything that start with http:// ?
cuz in my forum , all links are under [code ] tags , so they dont have [url ] tags.
thanks

F0xy
03-13-2008, 06:49 PM
um..can you make it check everything that start with http:// ?
cuz in my forum , all links are under [code ] tags , so they dont have [url ] tags.
thanks

same here, if it could do this then its exactly what i need

jedrus
03-24-2008, 09:41 PM
hmm.... when i run maintenance -> Link status checker I've got something like that:

Link 0: http://: checking...
Warning: parse_url(http://) [function.parse-url]: Unable to parse URL in /admincp/misc.php(1535) : eval()'d code on line 123
status: destination server is down / request timed out
Link 1: http://%5BURL: checking...

iBlazee
05-27-2008, 09:54 PM
Please release one for 3.7

Cagonni
08-01-2008, 04:02 PM
is it possible to make this only for rapidshare?

RobbieZ
08-13-2008, 10:52 PM
Installed.. Ok . I ran the query from my admin panel..

Where exactly in my forum do i now check these links ?? My mods are supposed to be able to do this too.. but cant find and menu of any sort..

thanx

RobbieZ
08-14-2008, 04:24 PM
Does this work for 3.7.2 ??

Terbiyes
09-03-2008, 02:43 PM
<i>thank you...</i>

compuminus
02-12-2009, 11:03 PM
As other users have posted, I am also getting the 'duplicate / dealt with" message for all images except those on the first page (when updating the counters in the admincp). This ought to be extremely easy to fix if the creator of this mod can upload a new version. In the meantime, this basically makes the script worthless without the fix. I am running vB 3.8.0 FYI.

Alfa1
02-14-2009, 01:22 AM
I would really appreciate a vb 3.8 version.

ForYou
02-17-2009, 10:15 AM
Hello ,

There is error , using Vb 3.8.1

Database error in vBulletin 3.8.1:

Invalid SQL:
UPDATE linkstatus SET status = '000' WHERE linkurl LIKE '%'http%';

MySQL Error : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'http%'' at line 1
Error Number : 1064
Request Date : Tuesday, February 17th 2009 @ 12:11:26 PM
Error Date : Tuesday, February 17th 2009 @ 12:11:56 PM
Script : http://www.domain.net/montada/admincp/misc.php?do=check_link_status&totalrows=+&startfrom=0&percycle=30&timestart=&timeout=15
Referrer : http://www.domain.net/montada/admincp/misc.php?do=check_link_status
IP Address : 82.205.124.79
Username : admin
Classname : vB_Database
MySQL Version : 4.1.22-standard

Process started 0 mins ago. If nothing appears to happen for a minute or more, click here to start this section again
0 links processed. 96,684 links remaining.

Most common statuses: 200 & 304 = Image works. 404 & 410 = Image doesn't work.

Link 0: http://: checking...
Warning: parse_url(http://) [function.parse-url]: Unable to parse URL in [path]/admincp/misc.php(1933) : eval()'d code on line 334

Please Help

SymbioteK
02-19-2009, 01:08 AM
i can;t find where to check the linsk after running the maintenance process either

artinjulio
07-17-2009, 12:18 PM
nothing sorry

Kolbi
02-07-2011, 09:43 AM
Is the possibility given that someone can update it to get it workable with vB4?