and it turns out to be a very smart hacker. here is the steps I try to locate the issue.
I use tcpdump to capture all traffic for a couple hours and do a quick analysis to see whether there are some special. like sending out credit card using SMTP with the port 25, or just use stand http post, if it https. you can sort the TLS certificate to see any special certificate which represents some evil 3rd party hosts.
Nothing found special for me. most just traffic to this site and some 3rd party API call like shipping rate calculation, Fedex integration.
then do a analysis about the request log to see any url which has huge hit from the same IP.
you can use some handy linux command to do the aggregation.
awk '{print $8$2}' requestlog|sort|uniq -c|sort -r -n|more
nothing special, most hits are from crawlers ip with Google.
feel a little big frustrated now, this must be a smart hacker. then I watch for file change within 5 minutes. if they hacker intercept the request and store somewhere, I will definitly find out where did he/she store the file.
go the root directly of the magento, I spect all files change within 5 mintues (I placed a order using a dummy credit card"
find . -mmin -5 -ls
there we go, one file called db-tab-footer_bg.gif was changed minutes ago. there should not be any chagne for gif files. this turns out to be a complicated image file. since it's all stored with encrypted data ( it must be credit card there),
now, time to locate how can they capture the data and dump to this file.
just search all php files containing db-tab-footer_bg.gif.
it's in the global config file, here is the content. essentially they intercept all post request and encrypt using his/her RSA public key and put to the gif file. nobody can decrypt it.
now time to get ride of the backdoor and do the housekeeping to locate how did they inject the backdoor on the server