Hey admins - URGENT.

Discussion in 'General chat' started by dgriffith, Oct 23, 2008.

  1. murray

    murray Junior Member

    Joined:
    Jun 2, 2004
    Messages:
    460
    Likes Received:
    0
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    hey all

    fixed the compromised server scripts with the clean.php script that was posted here by nickggr. he has truly saved the day! thanks dude! :D

    m.
     
  2. Re: Hey admins - URGENT.

    I had the exact same problem with one of our web servers. What concerns me is that we are running absolutely *zero* cms, blogs, or anything else like that is open or closed source. It is all 100% custom code.

    I'm thinking there must have been an exploit on PHP, apache , or something else. I'm looking through my logs and finding nothing strange or out of place. I'm really concered as to how that extra content was being added to my content. Our server is with LT, but out of the dozen servers with LT, this was the only one. Any leads would help a lot.
     
  3. dylanz

    dylanz Junior Member

    Joined:
    Nov 8, 2007
    Messages:
    122
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    Justin, what services do you have open ? Maybe diagnose with a nmap variety (nmap –vv –PE –sS –sU your_ip_here) ? How are you updating your systems ? Are you connecting via straight FTP, or SFTP ?

    Thanks for posting more about the issue. Knowing that any you're running a 100% custom application layer chalks quite a few things off of the list. It could indeed be a targeted PHP / Apache exploit... what PHP / Apache versions are you running ? That will help narrow it down more.

    I would still "highly stress" that your machine has been compromised by someone with root privs... which is less than good. If you're brave, use the clean.php script provided and go about your daily business... but, know that there could be backdoors installed, which jeopardize your data, and your users.
     
  4. Re: Hey admins - URGENT.

    We have the following installed:

    PHP Version 5.1.6
    Apache Version 2.2.3

    Here are the ports open:

    Discovered open port 80/tcp on xxx.xxx.xxx.xxx
    Discovered open port 21/tcp on xxx.xxx.xxx.xxx
    Discovered open port 111/tcp on xxx.xxx.xxx.xxx
    Discovered open port 638/tcp on xxx.xxx.xxx.xxx

    We do use FTP to connect to our servers (trust me, I know, we're changing that, just the boss liked using FTP).

    I'm not really sure whats up with port 638. Maybe thats a backdoor?
     
  5. dylanz

    dylanz Junior Member

    Joined:
    Nov 8, 2007
    Messages:
    122
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    I would close port 111/tcp (unless you're using it internally, then block it externally only), and would also close port 638/tcp, unless you know that you're using it (as I'm not sure what mcns-sec is). I would also bind FTP (21/tcp) to a different port if you can. Also, I highly suggest ditching FTP for SFTP or SCP.

    But yeah, port 111/tcp (SUN RPC) being open publicly is probably a bad thing. You may want to check to see what ports your current OS has open by default... that way you can work backwards. It could very well be possible that those ports were opened by the intruder, and they would be able to do remote manipulations to your machine through them moving forward.

    Murray, can you post your port information as well ? Would be interested to see if there was a correlation.
     
  6. Re: Hey admins - URGENT.

    I'm also on Layered and I'm running Movable Type, but the database was not affected. Only my php files were touched, and yes, I'm almost positive it was root. MT builds out files as nobody and I'm using it to build my php includes (header and footer). When I went into the module directory, I noticed that they were owned by root. When I deleted them and rebuilt them with MT, they were owned by nobody again. We're dealing with a root attack here.
     
  7. Re: Hey admins - URGENT.

    One other thing... the attack seems to have been automated, because they hit all of my php files, but all of my index files w/o extensions remained untouched.
     
  8. dylanz

    dylanz Junior Member

    Joined:
    Nov 8, 2007
    Messages:
    122
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    Thanks for the post theplasticmind.

    Soooo... this is bizarre. I've checked google/technorati, and this forum seems to have the most active posts about this exploit thus far. You'd think some of the technical forums that also reference it would be on top, but... permaculture for the win !

    It is definitely a root attack, and it is definitely automated. No doubt about both of those facts.

    How did the attack happen ? That's a great question, and still a mystery. It has happened across a bunch of random hosting providers, a bunch of random application layers... but, still have yet to hear from anybody else as to the architecture they are running. So... please post your arch specs, ie: PHP / Apache / etc versions, uname -a, open ports, etc.

    Root attacks are no joke. If at best, it was some script kiddie who wrote some little script to go and infect PHP files, so he could then point the end users at his other exploit, which could infect their host computers (getting the most reach). At worst, it could be that, as well as completely compromising the target server, opening ports, and basically owning the machine for future debauchery.

    I'm not a system administrator by any means (I'm a web application developer), but have worked with tons of extremely bright admins in Fortune 500 companies, and there are specific protocols you take in these situations. Only once was a root exploit deployed, and the team of admins had tabs on where it came from, what data was transferred to external sources (it was part of a db dump, unfortunately, and some data was compromised... ie: the attacker had a target :cry:), etc, etc. If they couldn't narrow it down, it's time to reinstall the OS across the clusters, re-instantiate users, load the backups, etc.

    But yeah, since this is the hub of this specific exploit at the moment, any application/arch specific details would help narrow it down.
     
  9. jmbullis

    jmbullis Guest

    Re: Hey admins - URGENT.

    Hey Guys,

    I just wanted to give you an update on where I am at. Apparently this issue has been around for a few days now. I think it originated recently as going to prevedvsem123.cn. In the last few days the url changed a few times and we happened to catch the bug when it was going to hu1-hu1.cn. When I Googled prevedvsem123.cn I found a lot of other people being affected by this. Every one of them ties back to LT.

    I got a call from someone at LT and they are apparently aware of the situation. They informed me that PHP had been compromised and that sshd had been rewritten so that anytime I updated the root password it would notify an email address at @ymail. After looking back, I can see that this issue started affecting my server around the 10th and we just now received the injection.

    At this point, per the request of LT, and the agreement of our server consultants we are disconnecting the hard drives, and reloading the os. I figured there was something else we could have done, but based on the extent of the damage there is no telling what all has been changed. I am convinced this is a good idea since this is related to a very similar attack which occurred in September of 2007.

    Right now I am just waiting on the reloaded server to come back online and then I will be proceeding to clean the old data and porting it over. I wish I was a super server guy like you guys but this nub just got pwned....
     
  10. fr34k123

    fr34k123 Guest

    Re: Hey admins - URGENT.

    Hi guys,
    I first noticed this on our website yesterday evening - obviously my first port of call was to email our server hosts. Judging by the emails ive had from my host this is very wide spread - what my hosts said:

    It is not PHP version related as we have seen this today hit all different versions of PHP

    It is a root brute force attack targeting Shell access

    I downloaded the clean script as supplied by dylanz (correct me if i'm wrong) and got my server hosts to look over it - here's what they said:

    That script is not a complete fix as it will only clean certain code -
    it is not a fix all

    What these thugs have done is write a series of ever-changing scripts
    that then go and pull new versions every few days - we have been reading
    them through carefully today for our own sites

    The only way to do that completely - especially in your CMS type of site
    - is to use a script that FINDS the locations and then go and delete it
    - anything else is a false hope

    The only way i could see me fixing this problem is to edit each separate file manually - remove the malicious code inserted then re-upload the clean edited file - unless i misunderstood my host's advice - here's what they said this morning:

    We have clients that have been affected by this ever-growing shs rootkit attack so we are familiar but this is outside the scope of support
    I will say that removing the files will only remove them until the hidden worm re-installs them so you have to get to the root problem or you will just be doing this every few days.

    Sorry if all this sound a little noob - but i'm just so unfamiliar with these sorts of problems and am at a real loss as to how i can rectify the situation and put my site back to normal.

    Any advice anyone could offer a noob is greatly apreciated
     
  11. milifestyle

    milifestyle New Member

    Joined:
    Feb 15, 2008
    Messages:
    1,573
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    This may explain why i've been booted out as soon as i got to the index page. Just saw a flash of the site and explorer closed.

    I'm here now so i'm guessing its fixed.

    I recieved an email from a friend today who mentioned a guy with an email addres starting with "SIMON_25_" who was apparently hacking servers, not sure if its related.
     
  12. Re: Hey admins - URGENT.

    Considering I have a very large customer base that was affected, I brought in the help of some developer friends and we worked out a really comprehensive cleanup script that's perl-based. Here's the code:

    https://pastie.textmate.org/299801

    You'll need to do the following from the command line logged in as root:

    1. Copy/paste the code above into a file called cleanhack.pl and set its permissions to 700. (chmod 700 cleanhack.pl)

    2. Type: find START_PATH \( -name '*.html' -o -name '*.php' -o -name '*.htm' -o -name '*.tpl' \) -exec /PATH/TO/cleanhack.pl {} \;

    Where START_PATH is the root directory you want to start with. For example: find /home/ \( -name ... This script is recursive.

    3. The script will parse through all of your files and clean out the malicious code and it can be run multiple times (in case you get dropped from your server), giving you feedback from the command line that looks like this:

    /home/sample/user/test_file.php: Infected. Now cleaning...OK

    Hope this helps!!!
     
  13. dylanz

    dylanz Junior Member

    Joined:
    Nov 8, 2007
    Messages:
    122
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    Is anyone effected by this not being hosted by LT ?

    Regardless, I'd recommend everyone read jmbullis's post twice :)
    Simply running a script to clean out the random javascript injections is not going to fix the problem.

    Consider the machines completely @#$%^'ed, wipe the machine, reinstall the OS, and load your backups.
    Really, if you care about your users and your data, you will schedule some down time and do this sooner than later.

    Personally, I don't want my fellow PC friends (and myself!) getting hit by anything moving forward.
     
  14. Re: Hey admins - URGENT.

    We're reloading our OS as we speak. Unfortunately its terrible timing for our clients, but nothing else we can really do.
     
  15. Khuffie

    Khuffie New Member

    Joined:
    Oct 23, 2008
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    I'm on BurstNET
     
  16. dylanz

    dylanz Junior Member

    Joined:
    Nov 8, 2007
    Messages:
    122
    Likes Received:
    1
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    Yikes, so not hosting provider specific either. I'd recommend doing what Justin is doing (OS reinstall), and dropping FTP for SFTP or SCP. Also, if possible... only log onto the machine via SSH. Application updates via git/svn/hg/cvs/etc via SSH or SSL.

    Man, what a pain :(
     
  17. fr34k123

    fr34k123 Guest

    Re: Hey admins - URGENT.

    Hi guys,

    I know for sure this particular attack isn't hosting provider related - I'm not hosted by LT and my VPS got hit along with a few other of their servers. My hosting provider also made me aware of a different well-known hosting company that had over 2,000 of their servers hit by this hack.

    Seems like it's spreading - so hosts beware - it's certainly time to look at some prevention methods.
     
  18. Delphy

    Delphy New Member

    Joined:
    Oct 24, 2008
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    0
    Re: Hey admins - URGENT.

    Hey All,

    I too had this infection and also wrote a script to clean up all the variations I've seen (3 of them so far). I also wrote up a quick guide to help you secure your server. Since the first attacks about 5 or 6 days ago I haven't had a single re-infection afterwards.

    My stuff is at https://www.delphster.net/scripts/

    Regards
    Delphy
     
  19. Re: Hey admins - URGENT.

    I just got this email:

    Dear Layered Tech Customer ~

    As a result of a routine internal security analysis, a vulnerability was detected which allowed certain communications between the Layered Tech help desk and clients to be vulnerable to interception. While normal help desk communications are not a source of concern, occasionally LT clients submit unencrypted passwords via e-mail or the help desk ticketing system which could result in unauthorized system access by 3rd parties.

    As a result, we strongly advise all customers to take proactive measures and change user and system credentials.

    Given the overall industry rise in security issues, it is best to err on the side of caution and maintain robust security procedures. Layered Tech also recommends the following security practices:


    1) Always change passwords after sharing them via e-mail, or upon receipt of new system login details.
    2) Ensure that you have a defined interval for password changes (every 30, 60, or 90 days)
    3) Disable/remove non-essential applications, services, and user accounts
    4) Set regular maintenance intervals to update core applications and kernels, to address known security issues
    5) Change default ports for administration and remote access to non-standard so they are not easily identifiable



    We value your business, and will continue working diligently to safeguard against any future vulnerabilities. Please note that SSL is now required to access the LT help desk system. Clients who are unable to gain access to the system should contact our Client Services team.

    Should further information become available following our extensive security review and analysis, we will update you.

    Thank you,

    Layered Tech, Client Services
    [email protected]
    (866) 584-6784 or (972) 398-7000
     
  20. jmbullis

    jmbullis Guest

    Latest Update:

    We decided to go the route of the reload, however, based on our RAID configuration they were not able to slave the disks to our server co-located at LT. After requesting that the original drives be reinstalled to gather data for a backup, they were unable to get the drives to boot. At this point we are travelling to the location about 4 hours away to retrieve the server and the drives and will be trying to put it back together over the weekend. I will be trying to employ some of your scripts to clean this mess up.

    After some major confusion with the data center we were able to contact the Manager back who called me before. Our CEO got involved and was able to communicate with a VP of LT. It must be serious if a VP is getting involved. It greatly saddens me that this has happened because like many of the other victims we were hosting quite a few customers who are now very unhappy.

    Along with the cleaning efforts and the data recovery we are scrambling to move our backed up files to new shared hosting servers. All I know is that it is going to be a long weekend and an even longer week ahead. I don’t blame LT as I know they are probably having an exponentially stressful time than I am, I just wish that was addressed quicker with better suggestions to recover.

    Again, only a nub who hates getting pwned.
     

Share This Page

-->