Microsoft Service Pack Testing
For many, the unannounced release of Service Pack 2 for Windows Server 2003 came as a surprise and caused some administrators to write some ad-hoc installation-blocking scripts because they had not properly tested the service pack. However, the surprise does not have to be a major issue for IT departments and with proper planning this panic can be avoided for future service packs.
Service packs are released by Microsoft to fix a large amount of issues that may not be considered high-priority or security-related within Windows or other Microsoft applications (Office, Internet Explorer, etc) while at the same time bundling all of those security patches and hotfixes that were released since the previous service pack. Thus, the service packs tend to be quite large and bloated, creating unpredictable test results for different IT groups.
Service packs also have a notorious history for breaking applications that were not part of Microsoft’s tested applications. This causes a huge risk to business continuity as many mission-critical applications may have odd behaviors or even be crippled entirely by a service pack installation, as was evident from numerous examples from Windows XP Service Pack 2. Additionally, binary diffing is an advancing trend used by many security researchers. Researchers are testing service packs looking for Silently Fixed Vulnerabilities. Utilizing many different binary diffing tools, researchers have the potential to find internally-fixed vulnerabilities that may not be back-ported to previous service packs, thus becoming zero-day vulnerabilities for the majority of hosts that have not installed the latest service pack. This threat adds an element of urgency to the patch process and can justify administrators into patching their hosts a bit faster.
Eventually all machines must update to the latest service pack level since Microsoft’s support is based upon the latest service pack level. This is a problem that must be tackled quickly and efficiently. Normally, standard patch precedence works for most small patches; but will not translate well when it comes to service packs due to unpredictable complications that may arise. When it comes to service pack analysis, different groups will have varying roles and responsibilities.
Developers
For both in-house and production development teams, it is critical that developers test their code against the latest service pack available. Development teams don’t want to have this testing occur under someone else timeline once the service pack is released, so it’s important that development teams utilize beta periods to test their applications to allow maximum exposure to the latest service pack, even if it may be half a year away from public release. Typically, Microsoft will grant a few months of beta period prior to the public release of a service pack - which allows for testing on the parts of the development teams. During the beta periods, developers should pay close attention to the service pack release notes to identify potential conflicts or changed functionality with any APIs the development team may be using.
Once the service pack has been released, Microsoft will usually grant a few more months for a “grace period” where the service pack does not automatically install on hosts with Auto-Update. Development teams should have already concluded their main testing phases at this point and should just be ensuring that the public service pack did not make any unexpected changes to their released applications when compared to the Beta release. Once this is verified, the developers can feel confident that their application is approved for the new service pack.
Administrators
Administrators have the most daunting role when it comes to service pack testing. They want to make sure all of their hosts are up-to-date with the latest patches, while minimizing any issues that may come up and break internal applications or configurations. Administrators are looking for the obvious signs of disruption: abnormal program exists, new error codes, access denied dialogs, etc. However, the only way for administrators to know if the service pack is safe for the host is trial-by-fire which is done by installing the released service pack on their production hosts. Therefore administrators must work quickly and effectively once the service pack is released publicly thereby minimizing the impact on continuity while still keeping their hosts up-to-date.
To buy administrators some time during the testing phase, host-based security products can help mitigate any risk that may occur from silently fixed vulnerabilities. A well versed host-based security product has the power to delay patch timelines substantially to allow more internal testing prior to rollout. Also, to help from the accidental installation of the service pack prior to testing, Windows/Microsoft Update can be disabled from a GPO. This is standard practice for many large enterprise networks, but should be adopted by any network where service pack/patch testing is necessary because of custom line-of-business applications. This gives administrators as much control over the deployment of patches as possible; minimizing the risk of a failed patch installed by a user going to Windows Update. Administrators will thank themselves for these policies once a service pack is released and its installation-priority can be delayed until they can finish their analysis of the service pack within their own reasonable schedule while still maintaining the integrity and continuity of their network.
Once the service pack is released publicly, production environments should immediately start testing the final release against their pre-existing servers and workstations. Because many of these hosts are mission-critical, virtualization technologies can be utilized to avoid any disruption in business continuity during this testing phase. Virtualization is a constantly-emerging technology that allows administrators and developers alike to create a virtual host for testing or production purposes, typically using freely distributed tools. For networks where virtualization is commonly used for servers (I personally see this as a good solution for a variety of reasons), testing is easily accomplished by creating a ‘clone’ of the host and testing on a separate machine. For physical hosts, VMware® has a free tool available that allows for the conversion from Physical to Virtual (P2V) so that administrators can test the physical systems resiliency to the service pack. This tool will create an exact duplicate of the physical host and allow an administrator to create a duplicate LAN network for testing without ever touching the production subnet. This is a huge benefit for any administrator that may not want to take Microsoft’s word on faith alone.
Once the effects of the service pack installation have been normalized, administrators should build the patch-deployment timeline for the entire network, starting with the least critical hosts to potentially identify pitfalls that may not have been found in the previous testing period. Once this process is agreed upon in change management, the testing and planning phase has been completed and standard deployment practices can now proceed.
Once the service pack has passed through the enterprise developers and administrators, CTO’s and CSO’s now have the opportunity to approve the service pack release throughout their environment’s servers and workstations. Only after having well-educated teams properly analyze the service can CSO’s and CTO’s make this educated decision and secure their environment.
Source: Andre Protas, Research Engineer, eEye Digital Security |
eEye CTO Marc Maiffret to demonstrate live hacking of Microsoft Vista at SANS
The SANS San Diego Event will be taking place from March 29th – April 6th 2007. SANS is one of the largest sources for information security training and certification in the world. On April 4th from 8:00-9:00 PM (Elizabeth A), eEye Digital Security Co-Founder and CTO, Marc Maiffret will be will be demonstrating a live hacking of Windows Vista system with Office 2007 and discussing the topic of “It’s More Than a Microsoft World”. Long regarded as a security expert and thought leader in vulnerability assessment and endpoint security, Marc Maiffret leads the efforts of eEye’s world renowned Research Team and speaks regularly on the state of security across the globe. Full Article eEye Exhibiting at DoDIIS Worldwide Conference 2007
Over the past several years, the Defense Intelligence Agency CIO-sponsored DoDIIS Worldwide Conference has become the Intelligence Community’s premier IT conference. This year’s event, to be held in Chicago, will highlight cutting-edge capabilities currently deployed and explore innovative solutions designed to support the changing needs of the warfighter. Full Article eEye Digital Security Increases Presence in Asia-Pacific Market
eEye Digital Security®, a leading developer of endpoint security and vulnerability management software solutions, today announced their increased presence in Asia-Pacific with the appointment of Gun Suk Ling as eEye’s Vice President, Sales, Asia-Pacific. Full Article |
Tech Talk
News & Articles
Announcements
Reader Q&A
Etcetera