GHOST security bug in Linux

As I expected the flow of security bugs continues this year. Now first serious open source bug disclosed this year following last year’s Heartbleed bug in OpenSSL, the Shellshock bug in Bash and the POODLE bug related to the the SSL v3 fall back issue.

The GHOST vulnerability is a serious weakness in the Linux glibc library. Glibc or the GNU C Library is the most common and an open-source form of the C Standard Library, and part of Linux. The GHOST vulnerability allows attackers to remotely take complete control of the victim system without having any prior knowledge of system credentials. CVE-2015-0235 has been assigned to this issue.

GHOST has been traced back to a buffer overflow flaw in the the __nss_hostname_digits_dots() function of glibc, otherwise known as GNU C Library, a core part of nearly all Linux systems, according to Qualys’ Amol Sarwate. CVE 2015-023: Nasty bug in glibc gethostby* functions leads to possible wealth of remote code execution opportunities. The GHOST vulnerability is a serious weakness in the Linux glibc library as it allows attackers to remotely take complete control of the victim system without having any prior knowledge of system credentials. GHOST means attackers could take over Linux servers using something as innocent looking as an email. This comes on the heel of other major vulnerabilities with silly names such as Shellshock, Heartbleed, and POODLE.

It seems that the the bug has existed for a little more than 14 years (the first vulnerable version of the GNU C Library affected by this is glibc-2.2, released on November 10, 2000). Or actually it existed for 12.5 years, it was patched in 2013 (glibc 2.17 and before vulnerable). The issue has not been widely fixed because stable-distributions didn’t want to update glibc when there were no known security issues. Not there is. And update storm is starting.

More details can be found in this advisory or in the YouTube interview below.

Many Linux distributions are potentially vulnerable to GHOST and should be patched. The easiest way to test if your servers are vulnerable to GHOST is to check the version of glibc that is in use with the following command: ldd --version

If the version of glibc is older than 2.18, your system is vulnerable to GHOST and should be updated. If you are using 2.18 or later, you are safe from the vulnerability.

The vulnerability was been already fixed on most newer versions of Linux distributions. However, it remains a threat to users of stable and older Long Term Support (LTS) releases where the bug remains. With Linux, the old adage of “if it’s not broke, don’t fix it” generally applies, especially for businesses.

If you are using an affected distro, don’t panic. Simply update your system as patches should be available now. For major current Linux distributions where there is update package available and the systems can be booted, this would be just another update process (for example for RHEL). Update and restart is likely easier than trying to figure out every service that calls glibc. It is highly recommended that you update and reboot all of your affected Linux servers. The easiest way to fix the GHOST vulnerability is to use your default package manager to update the version of glibc.

In case you have an older distribution where there  is no ready update packet or you are running a critical system that should not be stopped and needs heavy testing process before update can be accepted for use, things are harder for you. If you have a vulnerable embedded Linux device you might not be able to get update any time soon, or ever. Just imagine all the network devices built since 2000 that will never be updated… Most probably don’t run glibc, but there will be many that do.

Thankfully, Qualsys hasn’t released a working version of the hack yet. They’re waiting until half of all Linux servers are updated, and then are releasing it to force the hand of the remaining half. That seems punitive and reckless to put the straggling updaters at risk, but that’s how they roll in INFOSEC



  1. Tomi Engdahl says:

    Highly critical “Ghost” allowing code execution affects most Linux systems
    New bug haunting Linux could spark “a lot of collateral damage on the Internet.”

    “A lot of collateral damage on the Internet”

    The glibc is the most common code library used by Linux. It contains standard functions that programs written in the C and C++ languages use to carry out common tasks. The vulnerability also affects Linux programs written in Python, Ruby, and most other languages because they also rely on glibc. As a result, most Linux systems should be presumed vulnerable unless they run an alternative to glibc or use a glibc version that contains the update from two years ago. The specter of so many systems being susceptible to an exploit with such severe consequences is prompting concern among many security professionals.

    Besides Exim, other Linux components or apps that are potentially vulnerable to Ghost include MySQL servers, Secure Shell servers, form submission apps, and other types of mail servers. Update: In a later post, Qualys researchers enumerated apps they believed were not vulnerable. The list included Apache, Cups, Dovecot, GnuPG, isc-dhcp, lighttpd, mariadb/mysql, nfs-utils, nginx, nodejs, openldap, openssh, postfix, proftpd, pure-ftpd, rsyslog, samba, sendmail, sysklogd, syslog-ng, tcp_wrappers, vsftpd, and xinetd.

    “If [researchers] were able to remotely exploit a pretty modern version of Exim with full exploit mitigations, that’s pretty severe,”

    The bug affects virtually all Linux-based software that performs domain name resolution. As result, it most likely can be exploited not only against servers but also client applications. Word of the vulnerability appears to have caught developers of the Ubuntu, Debian, and Red Hat distributions of Linux off guard.

  2. Tomi Engdahl says:

    Bug 15014 – (CVE-2015-0235) gethostbyname_r() returns EINVAL (22) instead of ERANGE (34) (CVE-2015-0235)

  3. Tomi Engdahl says:

    Qualys Security Advisory CVE-2015-0235
    GHOST: glibc gethostbyname buffer overflow

    - We identified a number of factors that mitigate the impact of this
    bug. In particular, we discovered that it was fixed on May 21, 2013
    (between the releases of glibc-2.17 and glibc-2.18). Unfortunately, it
    was not recognized as a security threat; as a result, most stable and
    long-term-support distributions were left exposed (and still are):
    Debian 7 (wheezy), Red Hat Enterprise Linux 6 & 7, CentOS 6 & 7,
    Ubuntu 12.04, for example.

  4. Tomi Engdahl says:

    Fixing CVE-2015-0235

    Just like the recent OpenSSL heartbleed bug, this will be an annoying one to fix. The update is in the glibc package, but that’s a set of libraries that are being used by a lot of running services. After the update, each of these services needs to be restarted …

    To find all the services that rely on the glibc libraries, run the following command. It will list all open files (lsof) and find the files that refer to the glibc libraries.

    $ lsof | grep libc | awk ‘{print $1}’ | sort | uniq

    The updates are now available for RHEL 5, 6 and 7 as well as CentOS 5, 6 and 7 (all architectures).


  5. Tomi Engdahl says:

    Possible attack vectors

    The gethostbyname() call is probably among the most used ones on a server. That means any kind of DNS resolve can be used to trigger the CVE. The only catch is, you need to control whatever DNS is being resolved.

    That could mean;

    Mailservers using reverse DNS lookups on connecting IPs (DNS Blacklisting, SPF checks, …)
    Form submits that allow user content which results in a DNS lookup, think URLs, WordPress XML-RPC pingbacks, …
    MySQL servers doing authentication checks based on hostnames (in MySQL privileges)
    SSH servers that perform DNS lookups for authentication allow/deny rules

    For a more in-depth look, including code examples, have a look at the Qualys mailing list entry which covers the situation more in-depth.

    Any kind of DNS lookup can potentially trigger this. The only “positive” thing is that the exploit doesn’t immediately escalate privileges, you’re still the same user that ran the command. But there are ways of doing privilege escalation of course. And non-privileged users are still valuable assets for DDoS attacks, making server inventories, …


  6. Tomi Engdahl says:

    procmail (a SUID-root and SGID-mail binary) is vulnerable through its
    “comsat/biff” feature:


  7. Tomi Engdahl says:

    Serious Network Function Vulnerability Found In Glibc


    The libc -> glibc switch was so much fun, that I think we should do it again in reverse!

    ” – We identified a number of factors that mitigate the impact of this bug. In particular, we discovered that it was fixed on May 21, 2013 (between the releases of glibc-2.17 and glibc-2.18). Unfortunately, it was not recognized as a security threat; as a result, most stable and long-term-support distributions were left exposed (and still are): Debian 7 (wheezy), Red Hat Enterprise Linux 6 & 7, CentOS 6 & 7, Ubuntu 12.04, for example. ”

    So it’s actually already been fixed. All that’s needed here is for some distributions to push the fix out.

    “But .. but now it has a CVE number and everything – so it must be scary”

    Written by somebody who clearly neither manages a large amount of hosts exposed to the Internet nor manages multiple environments in which there are some new hosts that are luckily patched along side other older hosts that have to run *slightly* older releases of distro’s for one reason or another.

    According to directions side-thread, glibc versions prior to 2.19 are vulnerable. Checking my machines, Slackware-current and Lubuntu-14.10 are fine. Only my poor tiny Raspberry Pis are vulnerable (2.13). But they run slowly enough I can watch the gethostbyname() lookups myself :)

    I don’t get it. Proprietary software has all sorts of serious vulnerabilities. Why is it that when a vulnerability is found in FOSS, you people all come out and mock it while ignoring all the incompetence of proprietary software?

    FOSS *is* more secure, and that’s true even with the occasional vulnerability. You’re extremely illogical to point to some vulnerabilities and conclude that it isn’t more secure. How many vulnerabilities are not known about because no one can look at the source code?

    In case you’re unaware, “bugs are shallow” doesn’t mean they don’t exist.

    The difference between a deep bug and a shallow bug is is what happens after you notice a problem. A shallow bug is right there, at the surface.

    A deep bug is one where you look at function foo(), which creates an instance of class Bar, which is subclassed from IEParser, which calls friend class HTML4Lexer, which has function TagAtrribute() – but TagAtrribute() returns the correct value, so how the heck is it wrong in Bar? Then when you found out WHY it’s wrong, you can’t come up with any way of fixing it without rewriting the HTML specification.

    Heartbleed is actually a great example. Many people looked at it right away and within an hour or so there was a patch available.

    So long as we’re writing in C, this kind of thing (buffer overflows in particular) will probably continue.

    (Lest I start a flame-war: C is awesome in its way, but more than almost any other language, it really does make it easy to miss things like this.)

    If you think managed languages will prevent you from leaving security vulnerabilities, you are either not writing significant server software, or your software has vulnerabilities.

    The hardest security problems to solve aren’t the overflows, it’s the features given to users.

    There are so many examples of exploits in managed systems that it’s a display of ignorance to claim otherwise. .Net is especially bad in this regard

  8. Tomi Engdahl says:

    I ain’t afraid of no GHOST – securo-bods
    Serious – but it’s no Heartbleed

    GHOST (CVE-2015-0235), which can be triggered by the gethostbyname functions, impacts many systems built on Linux, starting with glibc-2.2 released in November 10, 2000. A proof-of-concept exploit developed by Qualys might be used against the Exim mail server, which is the default mail transfer agent on Debian Linux systems.

    During testing, however, Qualys discovered the majority of other widely used applications and servers are not vulnerable even though they use the affected function. Technologies left unscathed include Apache, mariadb/mysql, nfs-utils, openldap, openSSH, postfix, pure-ftpd, Samba, Sendmail, tcp_wrappers and others.

    “GHOST poses a remote code execution risk that makes it incredibly easy for an attacker to exploit a machine,” warned Wolfgang Kandek, chief technical officer for Qualys. “For example, an attacker could send a simple email on a Linux-based system and automatically get complete access to that machine.”

    “This is a serious vulnerability because it’s high impact when exploited and is very widespread, since there are many public-facing Linux systems that are vulnerable and easily accessible by cyberattackers,” Wysopal explained. “For example, attackers can now use this vulnerability to remotely install cyber-espionage malware or turn machines into botnet “zombies” that execute DDoS attacks on demand.”

    Moore agreed with other experts that Ghost – although worthy of immediate triage – was nowhere near as serious as the infamous Heartbleed OpenSSL security vulnerability.

    “To be clear, this is NOT the end of the Internet as we know it, nor is it another Heartbleed. In a general sense, it’s not likely to be an easy bug to exploit. One easily-exploitable case identified so far is the Exim mail server. An attacker could abuse this vulnerability to execute arbitrary commands on an unpatched server.”

    “Still, it could potentially be nasty if exploited so we strongly recommend immediate patching and rebooting. Without a reboot, services using the old library will not be restarted,” Moore concluded.

  9. Tomi Engdahl says:

    This is a funny analysis of GHOST.
    But Technical Analysis of ghost ? Lol.

    Technical analysis of Qualys’ GHOST

    PPS. On a more serious note, check out this message to get a sense of the risk your server may be at. Either way, it’s smart to upgrade.

  10. Tomi Engdahl says:

    Moore agreed with other experts that Ghost – although worthy of immediate triage – was nowhere near as serious as the infamous Heartbleed OpenSSL security vulnerability.

    “To be clear, this is NOT the end of the Internet as we know it, nor is it another Heartbleed. In a general sense, it’s not likely to be an easy bug to exploit. One easily-exploitable case identified so far is the Exim mail server. An attacker could abuse this vulnerability to execute arbitrary commands on an unpatched server.”

    “Still, it could potentially be nasty if exploited so we strongly recommend immediate patching and rebooting. Without a reboot, services using the old library will not be restarted,” Moore concluded.


  11. Tomi Engdahl says:

    Busting the Ghost Security Vulnerability Haunting Linux Systems

    A bug is haunting Linux systems.

    The Ghost vulnerability recently revealed by researchers at Qualys has triggered comparisons to Shellshock, but some experts say that both the impact and how organizations should approach patching it is different.

    “Applying a patch to bash and rolling out a newer version to me seems a lot easier,” said Jon Passki, lead security researcher at Coverity. “None of its dependencies are touched, so the fix can be very specific. As a sysadmin or someone in security operations, I’d rather have Shellshock than Ghost.”

    Ghost is a buffer overflow issue in the Linux glibc library. If the bug is exploited, it could enable an attacker to take control of a targeted system.

    “Glibc (libc) is a core library for many packages and for the host operating system,”

    he short of it is upgrading from one version of glibc to another isn’t possible until the main operating system is upgraded. Then you get into third party packages. They often write and compile against certain versions of libc. Again, they’re in the same boat. For example, maybe there’s a bug in libc that prevents their application from working, so they’ll compile to an older version

    “Libc might backport the fix because of all the aforementioned issues,” he added. “That would still require, in some cases, applications to be recompiled with the now patched backported libc…So there will be a lag in when those new versions are available.”

    Patching glibc is a little different than a library like OpenSSL due to kernel and build tool dependencies

    According to Qualys, the first vulnerable version of the GNU C Library affected by this is glibc-2.2, which was released on Nov. 10, 2000. There are a number of factors that mitigate the bug however; for example, the issue was actually fixed on May 21, 2013, between the releases of glibc-2.17 and glibc-2.18. However, most stable and long-term-support distributions were left exposed including Debian 7 (wheezy), Red Hat Enterprise Linux 6 and 7, CentOS 6 and 7 and Ubuntu 12.04.

    The release of a fix in 2013 means that many newer Linux operating systems were never at risk

    “Secondly, not all applications are at equal risk,” Kinger added. “Exploitation is very difficult as an attacker only has a small amount of initial exploit code that can be used: 4 or 8 bytes (depending on whether the system is a 32- or 64-bit system). Additional code must be written to an address referenced by a pointer which the attacker can modify. As a result, many apps are not at risk. So far, we are not aware of any potential web attack vectors, which reduces the attack surface considerably.”

    The GHOST.c utility included in the original advisory can quickly tell you whether or not the local glibc has been patched, said Moore.

    Qualys Security Advisory CVE-2015-0235
    GHOST: glibc gethostbyname buffer overflow

    procmail (a SUID-root and SGID-mail binary) is vulnerable through its
    “comsat/biff” feature:

    Conclusion: more than 3 dots is impossible,
    and neither ping nor arping is vulnerable.

    The Exim mail server is exploitable remotely if configured to perform
    extra security checks on the HELO and EHLO commands

    pppd (yet another SUID-root binary) calls gethostbyname() if a
    preliminary call to inet_addr() (a simple wrapper around inet_aton())
    fails. “The inet_addr() function converts the Internet host address cp
    from IPv4 numbers-and-dots notation into binary data in network byte
    order. If the input is invalid, INADDR_NONE (usually -1) is returned.
    Use of this function is problematic because -1 is a valid address
    (” A failure for inet_addr(), but a success for
    inet_aton(), and consequently a path to the buffer overflow.

  12. Tomi Engdahl says:

    GHOST: which services are vulnerable, ssh, web server?

    I don’t buy the OpenSSH “mitigation” linked to – Reverse DNS in OpenSSH or OpenSSH+libwrap is not exploitable.)
    Because of common programming idioms, and the very specific code paths to the vulnerable code, many programs are not vulnerable. To be clear, there is a problem affecting a handful of DNS related functions in glibc(<2.18), and DNS is an excellent way to get attacker controlled data into a system, though it's not a great way to get arbitrary data in. But, unlike some recent high profile issues, simply using the library or using those functions does not automatically make a program vulnerable.
    glibc has had that function since 1996 (pre v2.0), so gethostbyname() won't be called by OpenSSH.
    In the general case, since conforming DNS data cannot contain problematic long/malformed IP addresses, reverse look up and forward/reverse checks are not an attack vector. Where hostnames/dotted-quad addresses are part of a higher protocol (e.g. the SMTP EHLO name, as used exploiting Exim) then you might have something. For OpenSSH this would suggest host names in port-forwarding. Close, but no cigar, even if OpenSSH did use gethostbyname() directly

  13. Tomi Engdahl says:

    Proof-of-concept WordPress Ghost attack appears

    Hackers could exploit the Ghost Linux bug using WordPress-based attacks, according to researchers at Trustwave, which has produced a proof-of-concept cyber attack to prove its claim.

    Ziv Mador, vice president of security research at Trustwave, told V3 that the proof-of-concept will work on all vulnerable Linux systems, and showcases how hackers could exploit Ghost.

    “The proof-of-concept code can be used to check whether a remote web server is vulnerable to Ghost. It works by sending an XML request to the XML-RPC Pingback functionality of WordPress which includes a long URL,” he explained.

    “The code works on patched and unpatched versions but they will respond in a different way thus allowing the researcher or administrator to determine whether the server is patched or not.”

    t is currently unclear whether Ghost is being actively exploited, although Qualys believes that hackers could bypass many traditional defences.

    “During our testing, we developed a proof-of-concept in which we send a specially created email to a mail server and can get a remote shell to the Linux machine,” read the advisory.

    “This bypasses all existing protections (like ASLR, PIE and NX) on both 32-bit and 64-bit systems.”

  14. Tomi Engdahl says:

    Critical “GHOST” Vulnerability Released

    Update as soon as possible!

    This is a very critical vulnerability and should be treated as such. If you have a dedicated server (or VPS) running Linux, you have to make sure you update it right away. We know for a fact that Centos/RHEL/Fedora 5,6,7 as vulnerable, as well as some Ubuntu versions.

  15. Tomi Engdahl says:

    Critical ‘Ghost’ Linux flaw can be exploited through WordPress, other PHP apps

    However, researchers from website security research firm Sucuri said Wednesday that they have good reasons to believe the flaw can also be exploited through Web applications written in PHP that use gethostbyname() function wrappers. This has the potentially to significantly expand the attack vectors.

    One clear example of such a PHP application is WordPress, which uses a function called wp_http_validate_url() to validate the URLs of pingback posts.

    “It does so by using gethostbyname(), so an attacker could leverage this vector to insert a malicious URL that would trigger a buffer overflow bug, server-side, potentially allowing him to gain privileges on the server,” Sucuri senior vulnerability researcher Marc-Alexandre Montpas said in a blog post.

    Then Thursday, security researchers from Trustwave SpiderLabs created a proof-of-concept script to trigger the glibc buffer overflow though the WordPress pingback feature.

    “This PoC allows users to remotely verify if a target web server is vulnerable to the CVE however it does not demonstrate exploitability,” they said in a blog post.

    GHOST gethostbyname() heap overflow in glibc (CVE-2015-0235)

    Install glibc Patches
    And don’t forget to reboot!
    Disable XML-RPC

    It is possible to disable the XML-RPC process altogether if you do not want to use it. There are even plugins that will disable it.

    Disable Pingback Requests
    You may also disable the pingback feature by adding the following to your functions.php file

    WAF Protections
    By using a WAF, you can identify initial pingback XML requests on your WordPress site and look for attacks.

    Monitor Your Logs
    When attackers are attempting to exploit this vulnerability against your web servers, there will most likely be error messages (segmentation faults, etc…) that will indicate a problem. Organizations should be vigilant in monitoring their logs and following up on an anomalous errors.

  16. Tomi Engdahl says:

    Cisco says GHOST is more Casper than Sleepy Hollow
    Borg exorcised GHOST years ago when it sent IPv4 to the nether realms

    Cisco has put forward at least a partial response to 2015′s first branded bug, GHOST, saying that in The Borg’s world, the glibc vulnerability is probably of relatively low severity.

    That would, at least, explain why it’s not being hunted with quite the urgency of something like Heartbleed in 2014: right now, Cisco’s advisory states that it hasn’t confirmed the vulnerability status of any individual products.

    “The superseding function is getaddrinfo() which … is not affected by this buffer overflow”.

    Cisco says its intrusion prevention system and next generation firewall both include rules that would block attempts to exploit GHOST, and the company will issue an advisory if any of its products turn out to be vulnerable and need patching.

  17. Tomi Engdahl says:

    What the GHOST tells us about free software vulnerability management

    GHOST itself is a Heap Overflow in the name resolution function of the Glibc.

    Technically GHOST is a heap overflow, which is a very common bug in C programming. C is inherently prone to these kinds of memory corruption errors and there are essentially two things here to move forwards: Improve the use of exploit mitigation techniques like ASLR and create new ones (levee is an interesting project, watch this 31C3 talk). And if possible move away from C altogether and develop core components in memory safe languages (I have high hopes for the Mozilla Servo project, watch this talk).

    GHOST was discovered three times

    But the thing I want to elaborate here is something different about GHOST: It turns out that it has been discovered independently three times. It was already fixed in 2013 in the Glibc Code itself. The commit message didn’t indicate that it was a security vulnerability. Then in early 2014 developers at Google found it again using Address Sanitizer (which – by the way – tells you that all software developers should use Address Sanitizer more often to test their software). Google fixed it in Chrome OS and explicitly called it an overflow and a vulnerability. And then recently Qualys found it again and made it public.

    Now you may wonder why a vulnerability fixed in 2013 made headlines in 2015. The reason is that it widely wasn’t fixed because it wasn’t publicly known that it was serious. I don’t think there was any malicious intent. The original Glibc fix was probably done without anyone noticing that it is serious and the Google devs may have thought that the fix is already public, so they don’t need to make any noise about it. But we can clearly see that something doesn’t work here. Which brings us to a discussion how the Linux and free software world in general and vulnerability management in particular work.

    The “Never touch a running system” principle

    Quite early when I came in contact with computers I heard the phrase “Never touch a running system”. This may have been a reasonable approach to IT systems back then when computers usually weren’t connected to any networks and when remote exploits weren’t a thing, but it certainly isn’t a good idea today in a world where almost every computer is part of the Internet. Because once new security vulnerabilities become public you should change your system and fix them. However that doesn’t change the fact that many people still operate like that.

    A number of Linux distributions provide “stable” or “Long Time Support” versions.

    These systems are delivered with an implicit promise: We will take care of security and if you update regularly you’ll have a system that doesn’t change much, but that will be secure against know threats. Now the interesting question is: How well do these systems deliver on that promise and how hard is that?

    Vulnerability management is chaotic and fragile

    LTS and stable distributions are there for a reason

    The big question is of course what to do about it. OpenBSD developer Ted Unangst wrote a blog post yesterday titled Long term support considered harmful, I suggest you read it. He argues that we should get rid of long term support completely and urge users to upgrade more often. OpenBSD has a 6 month release cycle and supports two releases, so one version gets supported for one year.

    Given what I wrote before you may think that I agree with him, but I don’t. While I personally always avoided to use too old systems – I ‘m usually using Gentoo which doesn’t have any snapshot releases at all and does rolling releases – I can see the value in long term support releases. There are a lot of systems out there – connected to the Internet – that are never updated. Taking away the option to install systems and let them run with relatively little maintenance overhead over several years will probably result in more systems never receiving any security updates. With all its imperfectness running a Debian Squeeze with the latest updates is certainly better than running an operating system from 2011 that stopped getting security fixes in 2012.

    Improving the information flow

    I don’t think there is a silver bullet solution, but I think there are things we can do to improve the situation. What could be done is to coordinate and share the work. Debian, Red Hat and other distributions with stable/LTS versions could agree that their next versions are based on a specific Glibc version and they collaboratively work on providing patch sets to fix all the vulnerabilities in it. This already somehow happens with upstream projects providing long term support versions, the Linux kernel does that for example. Doing that at scale would require vast organizational changes in the Linux distributions. They would have to agree on a roughly common timescale to start their stable versions.


Leave a Comment

Your email address will not be published. Required fields are marked *