Oracle Corp. is going through a rough patch at the moment, largely because of rough patches. The Redwood Shores, Calif.-based database giant, which once advertised its products as "unbreakable," is now one of a number of vendors being criticized for its slow and often ineffective software patch process.
But code is complex, claim Oracle, Microsoft and other vendors, making patch production anything but an exact science. Still business partners and customers demand more, insisting that the processes vendors use to develop and distribute security fixes is still rough around the edges.
The vendor perspective
Oracle's most recent patch troubles started last month when it released several dozen security patches, but was criticized by UK-based security consultant David Litchfield and others, who complained that Oracle acts too slowly on bug reports and left a number of software holes unplugged.
Few deny that software code is complex, which is why the largest vendors say it often takes several months to patch a flaw. Security professionals close to the patch development processes at Microsoft, for example, said the company has made tremendous strides recently.
Each flaw reported to Microsoft triggers a chain of activity that begins with an e-mail acknowledgement from the Microsoft Security Response Center, and ends with the release of a rigorously tested update by the company.
"Our commitment to our customers is that they are going to get a real response from a real person within 24 hours," said Stephen Toulouse, security program manager at the Microsoft Security Response Center (MSRC), regarding his group's email@example.com e-mail hotline for reporting vulnerabilities.
The size of the core MSRC team constantly fluctuates, but it consists of about 12-24 people at any one time, said Toulouse. "But we may have as many as 100 people working on a single update," he said, "depending on its complexity, and how many applications are affected."
The anatomy of a patch
Here's how the patch process works at Microsoft: First, MSRC team members read and respond to bug reports, usually within a day. The MSRC then makes a judgment call about the vulnerability, based in part on the severity of the damage that might result if it is exploited.
During emergencies, the MSRC is given enormous latitude and power to commandeer the software giant's vast resources. "In the event of a criminal attack," said Toulouse, "we own the Microsoft worldwide process."
Typically, however, the next step is a call from the Microsoft security response team to the business division that produced the flawed product. Each of Microsoft's product development teams has a liaison who works directly with Toulouse and his colleagues.
A meeting between the security and development staff is scheduled with the initial call, and (in the case of a low level threat) usually takes place within a week, said Toulouse.
The next step -- actually writing the patch -- is the easy part. "That happens very quickly," Toulouse said. But making sure the patch is itself sound is another story entirely.
"Testing is the long pole of the process," said Toulouse. "We cannot afford to issue an update that introduces a new problem, such as breaking the product's compatibility (with other products)."
Microsoft tests most of its patch code in-house. It also shares patched code with some of its partners who test it in their labs.
Both Microsoft and Oracle trumpet their regular patch release cycles as a sign of their commitment to users. Microsoft has its monthly "patch Tuesday" release. Oracle has its quarterly updates.
However, third-parties are complicating the process by turning the hunt for bugs in big vendors' products into a public competition. Ego and simple competitiveness are prompting security pros to directly broadcast their discoveries to online discussion groups, rather than telling the vendors first.
"There is a certain notoriety to be gained from being the first to discover a vulnerability, you can't deny that," said security consultant Kevin Richards, international vice president of chapter relations for the Information Systems Security Association.
"If you have a consulting business, such a discovery can translate into more billable hours."
The customer's needs
Security professionals, meanwhile, are questioning whether quarterly or even monthly patches from vendors are coming quickly enough to help maintain their defense posture.
"As a security person, I want to know who's affected and what's my exposure," said Richards, whose clients have included the U.S. Department of Defense, and the companies Briggs and Stratton and Master Lock.
Regular patch cycles are a good thing, said Richards, but companies like Oracle and Microsoft must also be prepared to act quickly to patch more severe threats.
"I'm not sure we want to know every time there is a hiccup," said Richards. "But if it's something like Cisco's [IOS], which can affect 98% of the routers out there, I want to know right away."
Oracle said in an e-mail to SearchSecurity.com that it believes its quarterly security patches, announced a year ago, "efficiently" addresses most vulnerabilities. It also strives to fix the most serious issues first, and updates its main code base to ensure future software releases are secure.
That all sounds good, said a critic of Oracle's recent patch management performance, but it is far from the truth.
Steve Manzuik, Security Product Manager at eEye Digital Security, which makes vulnerability management software, is one of many who claim that it takes Oracle months to reply to bug reports -- in that much time, Oracle is choosing to simply ignore its obligations to users.
"They (Oracle) are trying to say they have a plan," said Manzuik. "It sounds really cool. But they are not actually living up to that."
This article originally appeared on SearchSecurity.com.