Shellshock fixing

Shellshock Bash Vulnerability has been keeping me busy last few days.  Shellshock is a nasty bug that causes problems. Because ePanorama.net is running on Linux server, I has to do the necessary steps to keep the site safe.

First step was to analyze the problem: Does my system have vulnerable bas version? The answer is yes to pretty much every Linux system around (bug was introduced over 20 years ago) unless they have been specifically update for this. So I am pretty sure I have the problem. Running tests told I had the problem.

Does it pose serious danger is next question? There are several attack surfaces. The SSH attack surface is not very dangerous in my case, because using it needs the attacker to have login credentials for the system. There are very few login credentials in use and they have already quite much rights, so user trying to use to bug to get more rights is not a serious thread to worry much about. The more serious is HTTP attack surface, where anyone could include bash commands to HTTP headers… This site still uses some services implemented with CGI-BIN interface and bash (some of them put originally on-line 15 years). This attack vector needed something to be done quickly. I verified with CVE-2014-6271/CVE-2014-7169 test page that I really has the problem.

The alternatives to solve the problem are fixing the bash to a working version and/or using some form of protection that blocks dangerous request from getting into the system. So I first checked if the problem could be easily solved out just by running Linux update. In this case this did not work that easily.

So I had to try the other option, which was to block the potentially dangerous traffic. Red Hat article Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271, CVE-2014-7169) article give instructions how to block potentially dangerous traffic using mod_security and IPtables. I started with the simple IPtables trick, it worked but caused some collateral damage blocking something that should get through. That was the first cure, the benefit to damages ratio was acceptable.

This give some time to work on real fix, which was to update bash. To do that I needed to figure out the best way to get the new bash version and make sure things work well with it. I ended up compiling it from the source code similarly to methods described at https://shellshocker.net/. Then some testing with the new bash version. The services turned to work well with it. Then some manual hacking to get it replace all the ways the old bash could be called. And then testing that there is no issues with new bash. Things worked well. So now I can again run the system without that problematic firewall rule to block potentially dangerous HTTP requests.

Things are working well again. The plus-side of this whole story is that I have learned more on bash, IPtables and mod_security. Things have worked pretty well despite all the work going on (not any serious problems). On the weekend there was also a short service outage due Amazon’s massive messy EC2 reboot.

8 Comments

  1. Tomi Engdahl says:

    After the Shellshock fixing there has been series of security updates on the server.
    There are improved firewalling etc..
    Hopefully the changes have not caused too much collateral damage.
    If you see problems that something that did work earlier does not work anymore, please send feedback.

    Reply
  2. Tomi Engdahl says:

    One tip related to bash updating (new bash to replace old in running system without reboot):

    Replacing a running executable in linux
    http://stackoverflow.com/questions/1712033/replacing-a-running-executable-in-linux

    5 down vote

    In linux, you can safely replace the running executable while the process is running. As long as the process is running, it that instance will continue to use the “old” code. All new calls to the application will use the “new” code. So, simply restarting the application (or the device if necessary) will use the new copy.

    It’s OK to replace the executable while the program is running IF you rm (unlink) it first.
    This is not what happens with cp, so don’t use that. Either mv the file, or, to really be sure, rm it and then put the new one in the same place.

    this sequence will work fine:

    Copy new executable to a local file.
    Verify local file.
    unlink() existing executable.
    rename() new executable to the correct name.

    The application will keep running after the unlink() – the kernel won’t release the underlying data until all executing copies are finished.

    http://askubuntu.com/questions/44339/how-does-updating-running-application-binaries-during-an-upgrade-work says:

    17 down vote accepted

    Linux (and other UNIXes) draws a distinction between the name of a file (the link), the file itself (often identified with the inode), and open handles to the file. When you go to delete a file, you call the unlink() call – this erases the link to the file (you could also use rename() to overwrite it with a different inode). However, if open handles to the file (or other links – files can have multiple hardlinks) remain, the inode remains, and so does the file content, until all links and handles go away.

    So running programs using the library or whatever keep a handle to the old version (often implicitly through a memory mapping), so it stays on disk. It just doesn’t have a filename anymore, and will be cleaned up when all programs using it shut down (or on the next reboot, during the filesystem check or journal replay).

    Unlike Windows, you can delete or replace an open file; to give a simplified explanation, new requests for the files open the new file, existing handles use the file that existed when they were created. In other words, in Linux you can have files/file versions that still exist, although there’s no longer a pointer to them in the directory structure; those cease existing there is no pointer to them at all (closed and all).

    This is used not only in distro upgrades, but happens on every package upgrade (the dist-upgrade just adds a couple more automagic steps to that process).

    Many Linux processes continue to work after the packages they come from have been upgraded – but some don’t.

    Reply
  3. Tomi Engdahl says:

    Guard yourself against Shell Shock: Net security experts
    http://www.thehindubusinessline.com/features/smartbuy/tech-news/guard-yourself-against-shell-shock-net-security-experts/article6481210.ece

    eScan launches online diagnostic tool
    Hyderabad, Oct 8:

    Hackers are always at it—inventing new viruses and tricks to gain access to your data. The latest one ‘Shell Shock’ is going to cause immense losses if you are not vigilant enough, warn security experts.

    “All the Unix-based operating systems such as Linux and Mac OS X are likely to be affected with the Bash Shell Shock vulnerability,”

    The anti-virus solutions firm has launched an online tool to identify servers affected by the latest vulnerability.

    Internet security solutions company Symantec has asked users to adopt sandboxing and process access control as another layer of security.

    “The restricted sandbox ensures even trusted programmes perform required functions, thereby blocking any potential harm that an unknown program could cause,” the company said in an advisory on the Shell Shock threat.

    It also asked the organisations to implement file and system tamper prevention allowing the system administrator to customise and lock down configurations, settings and files.

    Reply
  4. Tomi Engdahl says:

    > I am an at home user of Ubuntu 14.04. I have bash, dash and busybox installed on my Ubuntu desktop.

    You have bash installed in your box. It will be vulnerable bash if you have not updated it last well (do test to see).
    You need to have bash 4.3.27 or newer (if such has come up).
    Let’s assume you have vulnerable bash. If anyone can use that to hit your computer depends on what you use it for.
    The main attack vectors are:
    Web services – if you run web server and use CGI-scripts your are in danger because anyone connecting to web server can run commands
    SSH – users that have credentials to system (user account and password) might be able to run command at higher than their own right
    OpenVPN server – if you run OpenVPN server with password authentication anyone can potentially run commands on your system
    DHCP – if someone compromises your DHCP server (usually your broadband router) they can run command on your system

    If you are not running web server or OpenVPN server, no worries on them.
    SSH vulnerability is an issue mainly on multi-user systems where there are many different people using the same computer with different user levels, if you are the only user of your desktop then I would not worry much…
    DHCP vulnerability could be issue after if someone hacks to your DHCP server or has physical access to your LAN to put in a rogue DHCP server in…

    > Also, if I had a router hooked up to my computer that uses busybox and I had a firewall on that same router, would a hacker be able to compromise me that way?

    The busybox as itself would not be vulnerable. If someone can hack in your router some way, they might or might not be able (depends on router design) to use it to make the DHCP attack.. Not very probable though.

    > I want to know what to do to protect myself against this thing beyond what I am already doing, which is I am installing security updates every time Ubuntu tells me there is one available. Please let me know!

    Install the security updates and verify the bash version you already have. That’s the best start.
    This shellshock is a serious issue if you have an Internet facing public server – you need to worry (I spend some nights on this issue on this web site to keep it safe… first quick fix with firewall rules, then bash update and taking modsecurity to use, then fixing some issues that modsecurity caused)
    If you have normal desktop Linux workstation behind the firewall that you keep up-to-date it is a less of an issue.

    Reply
  5. Tomi Engdahl says:

    Solaris fix-it firm offers free BASH patch for legacy Oracle kit
    Sets up camp on moral high ground as lawsuit rages on
    http://www.theregister.co.uk/2014/10/16/solaris_fix_it_firm_beats_oracle_bash_patch/

    A Solaris fix-it-firm being sued by Oracle over copyrighted code says it has stepped in to defend customers not protected by Larry Ellison’s firm from BASH attacks.

    Terix has released a BASH fix for Solaris on SPARC and x86 that it claims goes further than Oracle’s own recent BASH patch.

    BASH, vulnerability CCE-2014-7169, lets hackers execute code remotely on Solaris systems.

    The Terix patch works for Solaris versions 6 and 7, in addition to 8, 9 and 10, with the code released to all under the GNU General Public License. A version for Solaris 11 is under development and will be released “shortly”, Terix promised.

    A 26 September Bash patch from Oracle covers recent versions of Solaris – namely 8, 9, 10 and 11.

    Reply
  6. Tomi Engdahl says:

    How IKEA Patched Shellshock
    http://linux.slashdot.org/story/15/06/29/2136208/how-ikea-patched-shellshock

    Magnus Glantz, IT manager at IKEA, revealed that the Swedish furniture retailer has more than 3,500 Red Hat Enterprise Linux servers. With Shellshock, every single one of those servers needed to be patched to limit the risk of exploitation. So how did IKEA patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating “That’s it, thanks for coming.” On a more serious note, he said that it took approximately two and half hours to upgrade their infrastructure to defend against Shellshock.

    Ikea Patched for Shellshock by Methodically Upgrading All Servers
    http://www.eweek.com/security/ikea-patched-for-shellshock-by-methodically-upgrading-all-servers.html

    It took about 2.5 hours to test, deploy and upgrade Ikea’s entire IT infrastructure to defend against Shellshock. Here’s how Ikea did it so quickly.

    Glantz explained that Ikea has more than 3,500 Red Hat Enterprise Linux (RHEL) servers deployed in Sweden and around the world. With Shellshock, every single one of those servers needed to be patched and updated to limit the risk of exploitation. So how did Ikea patch all those servers?

    Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating “That’s it, thanks for coming,” as the audience erupted into boisterous applause.

    On a more serious note, Glantz said that it took approximately 2.5 hours to test, deploy and upgrade Ikea’s entire IT infrastructure to defend against Shellshock. The key to Ikea’s ability to quickly upgrade all its servers is having a consistent approach to system-management across its infrastructure, he said.

    To audience applause and laughter, Glantz visually displayed the system-management approach with a graphic instruction manual that showed the parts in a manner similar to how a typical Ikea furniture assembly pamphlet looks.

    “One does not patch random servers,” Glantz said.

    Glantz explained that the first step in the assembly of his IT infrastructure is to have a well-defined Standard Operating Environment (SOE). The SOE includes a definition of the hardware platforms used as well as the Linux and application software that is installed.

    It’s critical to enforce a system-management process that keeps servers and application software on the latest versions, Glantz said. He warned that if an enterprise doesn’t enforce that mandate, inevitably, the majority of systems will be running older versions and it will be more difficult to scale, manage and patch.

    Ikea uses the Red Hat Satellite server-management technology to track and manage its Linux servers in a standardized manner.

    One of the potential challenges of constantly updating servers is the risk that applications break when new server operating system software is loaded. Glantz, however, isn’t worried and noted that RHEL offers the promise of Application Binary Interface (ABI) compatibility across updates.

    Reply
  7. Tomi Engdahl says:

    Mozilla Launches Secure Open Source Fund
    http://www.securityweek.com/mozilla-launches-secure-open-source-fund

    In an effort to help secure open source software, Mozilla this week announced the creation of Secure Open Source (“SOS”) Fund, which kicks off with an initial $500,000 grant.

    The SOS Fund was created to support security auditing, remediation, and verification for open source software projects, in an attempt to prevent major vulnerabilities from slipping into them, as Heartbleed and Shellshock have in the past. According to Mozilla, there hasn’t been adequate support for securing open source software until now, and the new initiative aims at changing that.

    The Fund is part of the Mozilla Open Source Support program (MOSS), and the initial $500,000 funding should cover audits of a series of widely-used open source libraries and programs, Mozilla said. However, Mozilla challenges the millions of organizations out there that leverage open source software to join the initiative and provide additional financial support.

    “Open source software is used by millions of businesses and thousands of educational and government institutions for critical applications and services. From Google and Microsoft to the United Nations, open source code is now tightly woven into the fabric of the software that powers the world. Indeed, much of the Internet – including the network infrastructure that supports it – runs using open source technologies,” Chris Riley, Head of Public Policy, Mozilla, notes in a blog post.

    Reply
  8. Tomi Engdahl says:

    Shellshock Attacks Still Cheap and Easy: IBM
    http://www.securityweek.com/shellshock-attacks-still-cheap-and-easy-ibm

    Two and a half years after being discovered, the Shellshock vulnerability continues to be abused in attacks, and for a good reason: it is a very cheap and easy attack, IBM says.

    Discovered in September 2014, Shellshock is a vulnerability found within the bourne-again shell (BASH), the default command shell in almost each and every Linux and Unix system at the time. An attacker able to abuse the security flaw could execute commands with super-user privileges remotely.

    Tracked as CVE-2014-6271, the issue was found to affect a great deal of devices, including Web servers and Internet-of-Things (IoT) devices such as DVRs, printers, automotive entertainment systems, routers and even manufacturing systems. Mac OS X systems were also impacted.

    With many applications relying on BASH, an attacker could exploit the vulnerability by sending a command sequence to the web server to be interpreted with the BASH.

    In July 2015, researchers warned that Shellshock was still being abused, and the attacks continue nearly two years later. Many vulnerable devices haven’t been patched to this day, and attackers are enticed to continue hitting those targets.

    “Attackers need only a server, basic programming skills and access to malware to carry out this type of attack. The level of knowledge and effort required is quite low. Fraudsters can simply launch attacks against hundreds of different IP addresses per minute and wait to hit a vulnerable server by chance,” IBM’s Joerg Stephan explains.

    To carry out a Shellshock attack, an attacker only needs to spend around $5 a month, Stephan says. For just over $30, an attacker could target around 1 million servers within a six-month period, which could translate into 100,000 victims, as roughly 10% of all servers remain unpatched, IBM says.

    To show just how simple it would be to come up with the necessary code, IBM’s researcher published some basic Python code that can do the trick.

    A bash script would download a bot from the server, save it to a certain path, make the file executable and run it, and could also include a line to execute the bot after each reboot, for persistence.

    Reply

Leave a Comment

Your email address will not be published. Required fields are marked *

*

*