In this two-part series, Brien Posey will explain how to identify and prevent buffer overflow attacks on home-grown applications. Part one below details how hackers use buffer overflows to take control of your programs. Part two will teach you how to protect applications from attack.
Buffer overflows are a favorite exploit for hackers. The vast majority of Microsoft's available patches fix unchecked buffer problems -- but what about applications developed in-house? They are just as susceptible as commercial applications to buffer overflow attack. It is therefore critical that you understand how they work and perform vulnerability testing on your home-grown applications prior to deployment.
What is a buffer overflow attack?
A buffer overflow is an exploit that takes advantage of a program that is waiting on a user's input. There are two main types of buffer overflow attacks: stack based and heap based. Heap based attacks flood the memory space reserved for a program, but the difficulty involved with performing such an attack makes them rare. Stack-based buffer overflows are by far the most common and are the type I will focus on in this tip.
In a stack-based buffer overrun, the program being exploited uses a memory object known as a stack to store user input. Normally, the stack is empty until the program requires user input. At that point, the program writes a return memory address to the stack and then the user's input is placed on top of it. When the stack is processed, the user's input gets sent to the return address specified by the program.
However, a stack does not have an infinite potential size. The programmer who develops the code must reserve a specific amount of space for the stack. If the user's input is longer than the amount of space reserved for it within the stack, then the stack will overflow. This in itself isn't a huge problem, but it becomes a huge security hole when combined with malicious input.
For example, suppose a program is waiting for a user to enter his or her name. Rather than enter the name, the hacker would enter an executable command that exceeds the stack size. The command is usually something short. In a Linux environment, for instance, the command is typically EXEC("sh"), which tells the system to open a command prompt window, known as a root shell in Linux circles.
Yet overflowing the buffer with an executable command doesn't mean that the command will be executed. The attacker must then specify a return address that points to the malicious command. The program partially crashes because the stack overflowed. It then tries to recover by going to the return address, but the return address has been changed to point to the command specified by the hacker. Of course this means that the hacker must know the address where the malicious command will reside. To get around needing the actual address, the malicious command is often padded on both sides by NOP instructions, a type of pointer. Padding on both sides is a technique used when the exact memory range is unknown. Therefore, if the address the hacker specifies falls anywhere within the padding, the malicious command will be executed.
The last part of the equation is the executable program's permissions. As you know, most modern operating systems have some sort of mechanism to control the access level of the user who's currently logged on. Executable programs typically require a higher level of permissions than the user who's currently logged on, and are therefore run either in kernel mode or with permissions inherited from a service account. When a stack overflow attack runs the command found at the new return address, the program thinks it is still running. This means that the command prompt window that has been opened is running with the same set of permissions as the application that was compromised. Generally speaking, this often means that the attacker will gain full control of the operating system.
Click for part 2 of this two-part series to learn how to protect your applications against buffer overflow attacks.
More Information from SearchWindowsSecurity.com
Reader Feedback
Member Dave S. writes: I figure there must be two kinds of people in the world, those who want to know the technical details of how buffer overflows work and those who are content with the knowledge that buffer overflows, along with many other types of malicious code attacks, are a serious problem you need to protect yourself against by patching, installing personal firewalls, using anti-virus software, etc.
However, Mr. Posey's description of the mechanics of a buffer overflow is fraught with so many misconceptions and outright errors that the article might well fall into the little understood category of "anti-knowledge." Granted, describing a technical concept like buffer overflows, even the fairly straightforward stack-based type, is difficult to do in a few paragraphs. It might be best to let those who really want to know this type of information to seek out more detailed, and accurate, works.
For example, Mr. Posey states: "The programmer who develops the code must reserve a specific amount of space for the stack." The stack is an area of memory allocated by the operating system to each running process and it is dynamic in size.
Another example: "The malicious command is often padded on both sides by NOP instructions, a type of pointer." NOPs are not pointers; they are simply assembly code instructions that do nothing. Their legitimate use is generally to provide a delay in execution. In malicious code they do help to increase the target zone for redirecting code execution to a point before the "malicious command." Padding with NOPs after the command serves no purpose.
An even more basic misstatement: "In a stack-based buffer overrun, the program being exploited uses a memory object known as a stack to store user input. Normally, the stack is empty until the program requires user input." The stack is used by a process as temporary storage during the normal activity of calling a function, even in applications that use no user input.
Author Response
I have to disagree. I admit that I'm not a professional programmer, but not all programming languages are created equally, and not all programmers use the same programming techniques. In programming, there are lots of different ways that you can write a program to end up with the same end result. There are situations in which the programmer specifies a size for a stack.
For more technical details regarding this tip, refer to Hacking Exposed book series, which helped develop my research.
