[tor-relays] My VPS relay has just been hacked

[Nick Sheppard]

First, many thanks for all the advice.  I will do some serious homework.

To the question What kind of virtualisation?  It's OpenVZ. In fact it's 
the OVZ Starter option from here:

http://www.edis.at/en/server/linux-vps-solusvm-openvz/zurich

and the OS option I chose is what Edis describe as "Debian 7 amd64 
minimal (64bit)"

I've restarted the VPS for 5-10 minutes at a time, long enough to watch 
the malware at work but not long enough for it to ramp up its flood of 
output.  It seems to live here, in /boot:

root at xxxxx:/boot# ls -al
total 128
drwxr-xr-x  2 root root   4096 Oct 26 14:19 .
drwxr-xr-x 23 root root   4096 Oct 26 13:58 ..
-rwxr-xr-x  1 root root 122698 Oct 24 17:19 hxyqbutesc

and it's started in a straightforward way from /etc and /etc/init.d ; 
rc1.d, rc2.d and so on have links to /etc/init.d/hxyqbutesc :

root at xxxxx:/etc# ls -l rc1.d
total 4
1 root root  18 Sep 13 20:21 K01quotarpc -> ../init.d/quotarpc
1 root root  13 Sep 14 22:41 K01tor -> ../init.d/tor
1 root root  17 Sep 13 20:21 K04rsyslog -> ../init.d/rsyslog
1 root root 369 Aug  5 07:47 README
1 root root  18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs
1 root root  20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc
...

root at xxxxx:/etc# ls -l rc2.d
total 4
1 root root 677 Aug  5 07:47 README
1 root root  18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs
1 root root  20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc
...

and in /etc/init.d is a short script:

root at xxxxx:/etc/init.d# ls -l
total 196
-rw-r--r-- 1 root root 2427 Aug  5 07:47 README
-rwxr-xr-x 1 root root 1276 Aug  5 07:47 bootlogs
-rwxr-xr-x 1 root root 1281 Aug  5 07:47 bootmisc.sh
-rwxr-xr-x 1 root root 3816 Aug  5 07:47 checkfs.sh
-rwxr-xr-x 1 root root 1099 Aug  5 07:47 checkroot-bootclean.sh
-rwxr-xr-x 1 root root 9673 Aug  5 07:47 checkroot.sh
-rwxr-xr-x 1 root root 3033 Aug  5 07:47 cron
-rwxr-xr-x 1 root root 1329 Aug  5 07:47 halt
-rwxr-xr-x 1 root root 1423 Aug  5 07:47 hostname.sh
-rwxr-xr-x 1 root root 3880 Aug  5 07:47 hwclock.sh
-rwxr-xr-x 1 root root  317 Oct 26 13:58 hxyqbutesc
...

which looks like this:

root at xxxxx:/etc/init.d# cat hxyqbutesc
#!/bin/sh
# chkconfig: 12345 90 90
# description: hxyqbutesc
### BEGIN INIT INFO
# Provides:             hxyqbutesc
# Required-Start:
# Required-Stop:
# Default-Start:        1 2 3 4 5
# Default-Stop:
# Short-Description:    hxyqbutesc
### END INIT INFO
case $1 in
start)
         /boot/hxyqbutesc
         ;;
stop)
         ;;
*)
         /boot/hxyqbutesc
         ;;
esac

and runs the main file  /boot/hxyqbutesc , which is mainly hex numbers 
with many embedded filenames and paths.  The timestamp Oct 24 17:19 is 
an hour or two before the VPS was suspended on Friday evening (can't be 
more precise yet because I'm not sure what timezone the suspension email 
was sent from).

Thank you to the people who suggested using ps as well as top.  ps -x 
gives eg this when the malware is working:

PID TTY      STAT   TIME COMMAND
     1 ?        Ss     0:00 init [2]
     2 ?        S      0:00 [kthreadd/3277]
     3 ?        S      0:00 [khelper/3277]
  1373 ?        Ssl    0:00 who
  1406 ?        Sl     0:00 /usr/sbin/rsyslogd -c5
  1451 ?        Ss     0:00 /usr/sbin/cron
  1478 ?        Ss     0:00 /usr/sbin/sshd
  1857 ?        Ss     0:00 vzctl: pts/0
  1858 pts/0    Ss     0:00 -bash
  3474 ?        Ss     0:00 ls -la
  3477 ?        Ss     0:00 cat resolv.conf
  3478 ?        Ss     0:00 route -n
  3479 ?        Ss     0:00 ifconfig eth0
  3481 ?        Ss     0:00 su
  3489 ?        Ss     0:00 sleep 1
  3492 ?        Ss     0:00 ifconfig
  3493 ?        Ss     0:00 sh
  3494 ?        Ss     0:00 grep "A"
  3496 ?        Ss     0:00 pwd
  3497 pts/0    R+     0:00 ps -x

For process 1373 top would give "hxyqbutesc" instead of "who" (this line 
doesn't change), and the ten lines for 3474 - 3496 would be the two 
changing blocks of ten-letter random names.  Assuming the glimpses of 
command names here can be taken at face value, something very sinister 
is clearly going on ...  other commands that flash into view are "who, 
whoami, uptime, bash, id, cd /etc, netstat -antop, top, echo "find", 
gnome-terminal".


As for how it got in, most people seem to suspect an attack through ssh. 
  But when I ssh'd in yesterday to start investigating, I was careful to 
note the "Last Login" message, and it gave my own genuine last login, 
three weeks ago from my home IP.  Can I take this at face value to mean 
that ssh has not been used between my own two logins?  Can "Last Login" 
be falsified?

Some suggest an exploit using out-of-date software.  Well, I last did an 
apt-get dist-upgrade 29 days ago, (after the bash fix), and tor and 
obfsproxy (included in the dist-upgrade) were the only things I 
installed over the "minimal" Debian 7.6 - so I was reasonably up-to-date.

And some suggested that the malware might have come through the Solus 
control panel (which of course has root access - I ssh in by requesting 
a temporary (time-limited) console from Solus, and it gives me a 
one-time strong password to use for that).  Is this plausible?  perhaps 
the next step is to send all this to Edis support and see if they 
noticed anything unusual around 17:19 last Friday?

Thanks again for everybody's help,

Nick Sheppard