Buffer overflows can be prevented by GS cookies

Buffer overflows have plagued Windows users for years, but by using a compile time technology known as GS cookies, you can prevent them from damaging your Windows shop.

Hacking Exposed Windows
By Joel Scambray

Have a look inside the third edition of Hacking Exposed Windows : Microsoft Windows Security Secrets and Solutions by Joel Scambray, with this excerpt from chapter 12, "Windows security features and tools."

One of the most common, if not the most common, security-impacting implementation flaws in software is the buffer overflow. Even though people have been publicly exploiting buffer overflows since as early as 1988 when the Morris worm hit, they remain extremely prevalent in software that is being written today. Over time, the software industry and those who write operating systems have taken steps to minimize the exploitability of these conditions. In this section, we discuss the mitigations provided by the compiler used to build Windows Vista and Windows Server 2008. Before we get into the mitigations, we briefly discuss the buffer overflow condition so that the purpose of these mitigations is clear.

An overview of overflows

A buffer overflow is a generic term used to describe a condition that is the result of attempting to store more information at a memory location than the allocated space allows. For example, if a developer is writing an application that reads a series of names from a file, she might assume that the longest a name will ever be is 25 characters. To be safe, she allocates enough space to account for names that are up to 50 characters long and begins reading them in. If the file contains a name that is longer than 50 characters, a buffer overflow occurs. If an unfriendly person is able to influence the names that enter this file, he may be able to alter the program's execution by surgically replacing values in portions of memory that are adjacent to the buffer used to store the acquired name.

Hacking Exposed Windows

The finer points of User Account Control (UAC) in Windows Vista

Windows Resource Protection (WRP) protects critical system resources

When an application needs to store information in memory, such as names, it has a couple of options for where to put it: the heap or the stack. A buffer overflow can occur in either of these locations, but for now we will focus on stack-based overflows. The stack, which is used to control execution, comprises a series of stack frames. A stack frame is placed on the stack each time a function is called and removed each time a function returns. A stack frame, as created by the original Visual Studio 2003 compiler on the x86 platform, uses the layout shown in Figure 12-10, starting with the highest memory location first.

When a stack overflow occurs, it starts moving up this stack, taking out other local variables, exception handler structures, the frame pointer, return address and arguments passed to the function itself. Attackers take advantage of this behavior by overwriting these frame components with useful values. In the coming sections, we will discuss the following security features provided by the VS2003 and VS2005 compiler that help reduce the probability of an attacker successfully exploiting overflow conditions:

• GS cookies
• SafeSEH
• Stack layout changes
• Address space layout randomization (ASLR)

GS cookies

GS is a compile time technology that aims to prevent the exploitation of stack-based buffer overflows on the Windows platform. GS achieves this by placing a random value, or cookie, on the stack between local variables and the return address, as shown in Figure 12-11.

Figure 12-10

Standard stack frame
generated by Visual
Studio 2003
Figure 12-11
Stack frame with GS cookie

This concept is not unique to the Windows world. In fact, Linux distributions have had similar solutions for quite some time in the form of StackGuard and ProPolice. If a stack-based buffer overflows enough for an attacker to control the return address or frame pointer, the cookie has also been overwritten. Therefore, before the function returns, this cookie value can be verified to ensure such an overflow has not occurred. If the cookie value does not match the original value, an error dialog is presented to the user and the process is terminated.

Under the hood of GS

When a native application starts up, the first function that is typically executed is one of the C RunTime (CRT) entry points such as mainCRTStartup. The first action taken by these functions is to call __security_init_cookie, which is responsible for initializing the cookie that will eventually end up in every qualified function's stack frame. I say "qualified" because a number of scenarios produce a cookieless stack frame:

• The optimization (O) option is not enabled.
• The function does not contain a stack buffer.
• The function is decorated with __declspec(naked).
• The function has a variable argument list ("...").
• The function begins with inline assembly code.
• The compiler determines that the function's variables are used only in ways that are less likely to be exploitable.

Actually, previous research by Ollie Whitehouse of Symantec has uncovered another scenario that results in a cookieless frame: a stack buffer that is smaller than five bytes. However, as of VS2005 SP1, developers have the option to add additional checks to GS by adding the strict_gs_check(on) pragma to their code. This causes the compiler to place security cookies in places that it otherwise would not, such as buffers smaller than five bytes and buffers allocated for integer arrays.

The primary goal of __security_init_cookie is to generate a nondeterministic value for the security cookie. To accomplish this, a number of environmental values are captured, including these:

• System Time
• Current Process ID
• Current Thread ID
• Static value in the PE
• Current Tick Count
• Performance Counters

This can be observed by disassembling the __security_init_cookie function, as shown in the following code listing:

0:000 uf __security_init_cookie
97 00403fac 55 push ebp
97 00403fad 8bec mov ebp,esp
97 00403faf 83ec10 sub esp,10h
117 00403fb2 a110104200 mov eax,dword ptr [overflow!__security_cookie

170 00403fe0 50 push eax
170 00403fe1 ff1598524200 call dword ptr
[overflow!_imp__GetSystemTimeAsFileTime (00425298)]
175 00403fe7 8b75fc mov esi,dword ptr [ebp-4]
175 00403fea 3375f8 xor esi,dword ptr [ebp-8]
178 00403fed ff1594524200 call dword ptr
[overflow!_imp__GetCurrentProcessId (00425294)]
178 00403ff3 33f0 xor esi,eax
179 00403ff5 ff1574524200 call dword ptr
[overflow!_imp__GetCurrentThreadId (00425274)]
179 00403ffb 33f0 xor esi,eax
180 00403ffd ff1590524200 call dword ptr [overflow!_imp__GetTickCount]
180 00404003 33f0 xor esi,eax
182 00404005 8d45f0 lea eax,[ebp-10h]
182 00404008 50 push eax
182 00404009 ff158c524200 call dword ptr
[overflow!_imp__QueryPerformanceCounter (0042528c)]
182 0040400f 8b45f4 mov eax,dword ptr [ebp-0Ch]
182 00404012 3345f0 xor eax,dword ptr [ebp-10h]
187 00404015 33f0 xor esi,eax

Throughout this listing, the value of the security cookie is stored in the esi register, while the result of each function call is stored in the eax register. Between each call, you can see that these values are XORed against the current cookie value, thus creating a fairly nondeterministic security cookie.

NOTE: While we're on the topic of nondeterministic cookie values, Matt Miller recently wrote an article on uninformed.org that reflects his initial research on the determinism of GS cookies. His research has shown that due to the accessibility of entropy sources used to generate the GS cookie, local attackers are able to increase their probability of calculating a process's cookie value. However, at the time of this writing, Miller's research does not represent an immediate threat to the efficacy of GS cookies, but it's a start.

Once the cookie has been initialized, the application operates normally until a qualified function has been invoked. In these instances, the function prologue has been modified by the compiler to insert the cookie into the stack frame before the return address and frame pointer. This can be observed in the following code listing:

0:000 uf foo
21 00401040 55 push ebp
21 00401041 8bec mov ebp,esp
21 00401043 83ec24 sub esp,24h
21 00401046 a110104200 mov eax,dword ptr [overflow!__security_cookie]
21 0040104b 33c5 xor eax,ebp
21 0040104d 8945fc mov dword ptr [ebp-4],eax

In this listing, the first three instructions represent a typical function prologue. The next three instructions represent modifications made by the Visual Studio compiler with /GS enabled. The fourth instruction loads the previously initialized value of __security_ cookie in to the eax register. This value is then XORed against the current frame pointer (EBP) as seen in the fifth instruction. Finally, this value is placed in the stack frame, as seen in the final instruction.

Before this function returns, it must ensure that the version of the cookie currently in the stack frame matches the value stored in the previously initialized version, __security_cookie. To accomplish this, the function's epilogue has been modified with the following instructions:

28 00401071 8b4dfc mov ecx,dword ptr [ebp-4]
28 00401074 33cd xor ecx,ebp
28 00401076 e86d020000 call overflow!__security_check_cookie

In this listing, the first instruction loads the stack frame's version of the cookie into the ecx register. This value is then XORed against the frame pointer, as seen in the second instruction. On Vista, this provides additional entropy due to ASLR. Finally, the __security_check_cookie is called, which compares the value contained in ecx against the original value in __security_cookie.

All in all, cookies are fairly effective at preventing the exploitation of stack-based overflows on both Windows and non-Windows platforms. However, intricacies exist within Windows that prevent GS alone from putting an end to the prevalent exploitation of stack-based buffer overflows.

This was last published in July 2008

Dig Deeper on Network intrusion detection and prevention and malware removal



Find more PRO+ content and other member only offers, here.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.