They are stored as the value of the setting, so they are in the setting table. But the settings are combined and put in the datastore, so after you update the setting table, you want to call build_options() which is in includes/adminfunctions.php.
So a string of the current banned ips will be in $vbulletin->options['banip'], and you can see how that's used in function verify_ip_ban(), which is in includes/functions.php, and once you add to the string it you'd want to do something like:
Code:
require_once(DIR.'/includes/adminfunctions.php');
$vbulletin->db->query_write("UPDATE ".TABLE_PREFIX."setting SET value = '".$vbulletin->db->escape_string($value)."' WHERE varname='banip' ");
build_options();
Edit: and now that I think about it, that could be a lot of db work to be doing if you have a lot of attempts. I don't know, maybe not. But one alternative would be to just update the datastore row directly by using a REPLACE in the query. But it's a serialize array of all the options so you'd have to get it right. Another way might be to implement your own ip ban using hook global_state_check, which is called right after verify_ip_ban(). You could probably copy the verify_ip_ban function and modify it.