After removing everything from member list, nothing seemed to change.
I add a condition in member list, only show member > x post count. Then I index post count. I have only about 60 pages left. The longest queries which is the last page is 0.2 seconds. I am feeling much better now. The majority of member is remove from the list

.
There is nothing I can do about it.
vBulletin.org suffer the same problem too:
Code:
Showing results 17576 to 17600 of 17601
Search took 3.01 seconds.
If there are 80,000+ members, the result should be worse.
Anyway, thanks for all the support guys.