Nevermind, I found it, worked perfectly.
For those who stumble upon this in the future, here’s what I did:
Created a custom member field to store the vbulletin salt, each user has a different salt (column is ‘salt’ in the vb_user table).
Imported users via XML, make sure to set the password types to md5.
Edit: /system/expressionengine/modules/member/mod.member_auth.php
find the member_login function.. added/changed these lines. not the best code, I just wanted to hammer it out fast. could be cleaned up.
$mid = $query->row('member_id');
// grab vbulletin hash
$lookup_vb = mysql_query("SELECT m_field_id_3 from exp_member_data where member_id = $mid");
if ($vbr = mysql_fetch_array($lookup_vb)) {
$vb_salt = $vbr['m_field_id_3'];
} else {
echo mysql_error();
}
$vb_password = md5(md5($this->EE->input->post('password')).$vb_salt);
if ($query->row('password') != $password)
{
// To enable backward compatibility with pMachine we'll test to see
// if the password was encrypted with MD5. If so, we will encrypt the
// password using SHA1 and update the member's info.
$password = do_hash($this->EE->input->post('password'), 'md5');
if ($query->row('password') == $password)
{
$password = do_hash($this->EE->input->post('password'));
$sql = "UPDATE exp_members
SET password = '".$password."'
WHERE member_id = '".$query->row('member_id') ."' ";
$this->EE->db->query($sql);
} elseif ($vb_password == $query->row('password')) {
// check for vbulletin hash
$password = do_hash($this->EE->input->post('password'));
$sql = "UPDATE exp_members
SET password = '".$password."'
WHERE member_id = '".$query->row('member_id') ."' ";
$this->EE->db->query($sql);
}
else
{
/** ----------------------------------------
/** Invalid password
/** ----------------------------------------*/
Hope this helps someone else down the road.