IT Intel White Paper Virtualizing High Security Servers in a Private Cloud. Contents Business Challenge growth and spikes in demand This results. in low physical server utilization and, Executive Overview 1 Intel IT operates a worldwide. limits the ability to quickly provision new,computing environment that supports. server capacity In addition a conventional, Business Challenge 2 more than 90 000 Intel employees. approach requires the manual collection of, Intel IT s Cloud Computing and includes approximately 90 000. configuration purchasing history and other,Strategy 2 servers About 20 percent of our. information which complicates capacity, Security A Delicate servers are used to provide a broad. planning to support new IT initiatives, Balancing Act 3 range of services to Intel s employees. Creating Zones of Trust 3 customers and partners This Office For these reasons Intel is moving to a new. and Enterprise environment includes enterprise architecture based on cloud. Solution 3 computing that better meets our agility. applications for online collaboration, Virtualization Security e mail and calendaring as well as and efficiency objectives and helps us. Risk Assessment and Controls 4 better service the business groups our. large business applications such as, Implementation of the enterprise resource planning software customers To date Intel IT has virtualized. HTZ Architecture 4 more than 63 percent of its Office and. Approximately 10 percent of these Office Enterprise applications To meet our goal of. Key Results 7, and Enterprise servers are considered to virtualizing 75 percent of this environment. Lessons Learned 8, be high security and have the following we developed a level of security high enough. Conclusion and characteristics in common to confidently virtualize some of our most. Next Steps 8 security sensitive systems,Host data with a level higher than Intel. Confidential Intel Confidential data is We responded to this challenge by creating an. Acronyms 8, information that requires a signed non architecture called a High Trust Zone HTZ. disclosure agreement to view This architecture addresses specific calculated. Run mission critical Tier 1 applications that risks with controls that mitigate those risks. directly impact Intel s ability to design ship and provide an acceptable risk posture for the. order book build pay close network or secure virtualization of these systems. communicate,Intel IT s Cloud Computing,Provide authentication or encryption. services or other functions for which,Our strategy is to grow the cloud from the. virtualization causes too great an increase,inside out The private cloud we re building for. Office and Enterprise computing is based on, Though the current environment has met a highly virtualized energy efficient flexible. Intel s requirements to date the accelerating environment This approach offers many of. pace of business is driving a need to respond the benefits of a public cloud but without. more quickly to changing business demands the risks associated with hosting sensitive. IT Intel At the same time Intel IT must continually applications and data outside the Intel. The IT Intel program connects IT find ways to reduce costs environment The increased agility and more. professionals around the world with their, Conventional approaches to enterprise efficient resource utilization have realized a. peers inside our organization sharing, lessons learned methods and strategies computing constrain our ability to increase net savings of USD 9 million to date. Our goal is simple Share Intel IT best business agility and reduce costs For We expect that our private cloud will enable. practices that create business value and example traditional enterprise computing us to extend even higher levels of security. make IT a competitive advantage Visit dedicates servers to specific applications and availability to all applications without. us today at www intel com IT or contact Each server is sized to support application. your local Intel representative if you d,like to learn more. 2 www intel com IT, Virtualizing High Security Servers in a Private Cloud IT Intel White Paper. the need for costly specialized hardware separation between servers increasing the In 2011 we made significant progress in. and software This is due to high availability risk that a compromise may spread from one implementing this architecture which is based. capabilities such as automated virtual VM to others on the same physical host In on four cornerstones. machine VM restart and the availability of addition compromise of the virtualization. Trust calculation Dynamically determining, mission critical features such as Machine software s hypervisor can lead to compromise. whether a user should be granted access, Check Architecture Recovery MCA Recovery of all the hosted VMs as well as shared. to specific resources and what type of, MCA Recovery provides automatic detection physical resources such as hard drives storing. access to provide The trust calculation is, and isolation of many types of errors and application data and code. based on factors such as the user s client,recovery from these errors. We can establish controls to mitigate many device and location the type of resources. Implementing a private cloud will help us of these risks but this means deciding on requested and the security controls that. take advantage of the efficiencies of cloud limits for every level of security required As are available. technology As standards mature and we increase the use of a shared multi tenant User and data perimeters Treating users. security and costs of public clouds improve environment based on virtualization we and data as additional security perimeters. we envision using a mixture of public and anticipate that business groups will require that require protection in addition to. private clouds based on use case Currently differentiated security policies based on the protection the enterprise network. for less sensitive applications we take data classification and mission criticality boundary requires. limited advantage of public cloud services In addition our customers will want more. Balanced controls Installing a balance, For example we use several software as visibility into the secure data flow in the. of detective and corrective controls that, a service SaaS applications from cloud cloud and how business specific security. increase flexibility and recovery ability, providers including expense and time card policies are enforced Key security focus areas. while supplementing preventative controls, tools health benefit applications and social include data encryption and segregation VM. such as firewalls, media applications isolation secure VM migration virtual network. isolation and security event and access Security zones Dividing our environment. Security A Delicate monitoring Externally facing applications that into zones according to the sensitivity of. Balancing Act business partners or consumers can access the data and access controls enabling. Security is a delicate balancing act between are a particular concern because they pose a each zone to be controlled and monitored. business needs and risk As we develop and higher threat to prevent the spread of a compromise in. implement our cloud strategy the security of one zone to the other zones. Intel s data and applications remains a critical Creating Zones of Trust. focus We must maintain the security and In 2010 we began a radical five year. integrity of both corporate intellectual property redesign of our information security. and personal information regardless of where architecture with both cloud computing and. this data resides or how it is being used virtualization in mind 2 We assumed that some For our systems with high security. possibility of compromise is inevitable Our sensitivity we created an HTZ virtual. With the use of virtualization private and, new model greatly increases flexibility and environment which is the highest. public clouds create new security challenges, focuses on rapid detection of compromise trust instance of our virtualized. in areas such as resource isolation security, and survivability In particular it uses zones server environment see Figure 1. event management and data protection In, of trust that provide more flexible dynamic To populate it we identified specific. a non virtualized environment the physical, and granular controls than do traditional types of applications that are. infrastructure provides separation that is, enterprise security models conducive to virtualization from a. assumed to create a level of protection for,risk perspective as long as certain. applications and data However when using Intel s new security architecture is discussed in mitigations and controls are in place to. virtualization to consolidate multiple servers Rethinking Information Security to Improve Business. keep the risk at an acceptable level, onto a single host we give up the physical Agility Intel Corp January 2011. www intel com IT 3, IT Intel White Paper Virtualizing High Security Servers in a Private Cloud. Internal Domain High Trust Zone,All hosts see one shared build. logical unit number LUN,HTZ Landing Zone,Build Physical to Virtual. Deployment Build Physical Hosts Physical to Virtual. Server Subnet Converter Server VMs,Central Management. Intel Terminal Management Server Console,Intranet Server Access Subnet. Virtual Infrastructure Client,Data Center Utility Management. Landing Zone,Physical Hosts Logging Server VM,Bastion Host VM SIEM VM HTZ Virtual. Server Machines Security VMs Storage, HTZ High Trust Zone SIEM security information and event management. Figure 1 Intel IT s High Trust Zone HTZ architecture creates a separate landing zone for the hosts running the virtualized servers that will support. our systems with high security sensitivity, Examples of our high security sensitive systems Communication between zones is Six application controls including. applications include those that tightly restricted monitored and controlled application readiness reviews that examined. To separate zones we locate them on everything from the potential for access. Contain highly sensitive data, different physical LANs or virtual LANs overlap from outside the HTZ to security. Perform administrative or security vLANs and use different management hardening security development life cycle. functions for the rest of our environment systems for each practices authentication authorization. Are mission critical logging and so forth, Are under regulatory control Virtualization Security Five monitoring controls including. Exceptions are a small percentage of, Risk Assessment and Controls continuous monitoring for anomalous events. Intel Data Center Engineering and Intel Security or attacks on the HTZ virtual infrastructure. applications that we virtualize in total, worked together to assess the specific network VMs and applications. isolation or do not plan to virtualize because,tangible risks of virtualizing security sensitive. the residual risk would remain unacceptably Four network controls such as redundant. applications that are Tier 1 mission critical, high These include applications for which switches and network intrusion detection. restricted secret and top secret We identified, we cannot risk memory exposure or system NIDS sensors. 41 risks and ranked them by consequence,VM theft An example of virtualization. low moderate or high, in isolation is using a dedicated virtual Implementation of the. environment for a single application or After outlining the risks and associated likely HTZ Architecture. piece of an application The dedicated impacts we identified controls that would We took a three phase approach to implementing. environment reduces the risk because of reduce these risks to acceptable levels to our HTZ architecture and its required controls. shared tenancy and administration allow virtualization Through an internal risk. analysis process and a survey of industry Phase 1 Protect the Virtualization. Access to zones such as our HTZ is, sources including the Cloud Security Alliance Management Infrastructure. determined by the results of the trust, CSA the National Institute of Standards Our first step was to protect the virtualization. calculation and is controlled by policy, and Technology NIST and suppliers we management infrastructure by isolating the. enforcement points PEPs PEPs control, identified 24 total controls that could be virtualization infrastructure from the servers. communication between zones and may, grouped in four categories being virtualized protecting accounts used. include a range of controls including, firewalls application proxies intrusion Nine administrative controls such as to control virtualization securing applications. detection and prevention systems dedicated HTZ system administrator moving to the HTZ and hardening the. authentication systems and logging accounts and multi factor authentication operating systems OSs and platforms that. manage virtualization Hardening the OSs,4 www intel com IT. Virtualizing High Security Servers in a Private Cloud IT Intel White Paper. is a process that involves configuring each Protecting Accounts Used to or VM performance incidents based on the. OS with desired features and removing the Control Virtualization environment in which they reside. unnecessary features or applications This To establish a clear separation of duties. Our goal in application security testing was, first step encompassed the implementation we define specific roles for individuals and. to successfully carry over existing policies, of the first 14 of our 24 controls support groups In a large organization such. from the physical world to the virtual world,as Intel numerous support personnel by. Although there is usually no reason to change, Isolating the Virtualization Infrastructure default might have access to management. proven security practices when virtualizing, A number of virtualized servers in the servers It is critically important that we. there are exceptions One exception may be, environment support management including carefully restrict support roles permissions. to allow direct access to the VM console as, logging servers and terminal servers To and access to those who are fully trained. opposed to access through remote desktop, ensure customer virtual servers do not impact on virtual environments As an example any. technology Direct access to the console, these servers we maintain the management configuration inconsistencies introduced. requires allowing a way to provide access to, virtual servers in a separate cluster of hosts across a cluster of virtualized hosts could. the virtual infrastructure or at least parts, running an enterprise level virtualization potentially cause a multi user outage as. of it and controlling the associated risk It, product This protection by separation the VMs move dynamically across hosts. also requires managing permissions at the, extends to all areas of the environment in the cluster Such an event might also. VM level rather than at the infrastructure, including management and service accounts trigger a time consuming Sarbanes Oxley. level which can be cumbersome in a large, network architecture and OS configuration Act SOX audit if any of the VMs hosted. organization with many virtual servers, on management servers such as terminal require SOX compliance. servers We also segment live migration Many organizations have first second and Application Risk Readiness Review. production and backup traffic from one third level support for a service product or All applications and systems intended. another through a mixture of physical and infrastructure Each defined support role for the HTZ underwent a risk readiness. logical separation must have the necessary permissions for review This review process ensured that. Using an approach analogous to our physical their expected tasks For the HTZ we take applications landing in the environment did. environment we cluster virtualization hosts this a step further and restrict permissions not add additional risk to the multi tenant. into separate landing zones to maintain for tasks that personnel are not expected environment or materially change our trust. logical and physical isolation within the to perform Defining custom roles versus in the security of the HTZ environment An. greater HTZ When including virtual servers built in roles is more difficult because in many additional benefit of the review was the. as part of the management infrastructure cases permission dependencies are poorly opportunity to re examine the operational. we create a separate landing zone for the documented for management software and security of legacy applications and ensure. hosts devoted to running them No customer for incidents to be addressed expeditiously implementation of current best practices. VMs can run in this landing zone Also escalation paths must be clearly defined. The risk readiness review included the, whenever possible we dedicate resources following measures. such as the subnet and private virtual LANs Securing Applications. PVLANs to these VMs While this is a Moving to the HTZ Eliminating overlap of access from. more secure solution it is also a costly one Safely moving applications to a secure area outside of the HTZ. because it dedicates two standard hypervisor first required establishing a preproduction Evaluating network architecture to. hosts to the support of a relatively small virtualized environment for application define firewall rules and identify required. number of VMs and virtual appliances development and testing application review modifications including proxies and bastion. However it helps provide a segregation of process and selective application security hosts A bastion host is a special purpose. functionality between management and testing Our preproduction and production computer on a network that is designed. customer environments environments provide the following and configured to withstand attacks. We further harden these management virtual Segregation between development Evaluating application architecture. servers to ensure that access is restricted and testing and quality assurance QA and including security development. access to the hosts hypervisors is permitted production VMs lifecycle SDL practices authentication. only through them For example direct root or A way to easily separate service level authorization and logging. administrative access on hypervisor hosts is agreements SLAs in the environment Evaluating system security including. explicitly denied and user accounts are limited A means to better differentiate response and logging access control administrative. in number and frequently reviewed remediation times for infrastructure problems restrictions and more. www intel com IT 5, IT Intel White Paper Virtualizing High Security Servers in a Private Cloud. Phase 2 Extensive Security switches storage subsystems management. ENTERPRISE,Monitoring servers and even the VMs to complete. CONTROLS the picture We also monitor the various,Change Release Management. While performance and event monitoring,Con guration Management. Self provisioning is paramount to sustaining a robust management agents which is often. Capacity Management, Monitoring and Diagnostics infrastructure monitoring for security events problematic because these logs may be more. is equally important see Figure 2 We must be difficult to access or contain fewer details. Application,able to detect and provide alerts of anomalous. APPLICATION Additionally because not all parts of the. behavior in account usage VMs hypervisors,infrastructure collect logs with equal detail. and networks Phase 2 focused on putting,and precision it is sometimes necessary to. SOFTWARE systems in place to provide centralized logging. introduce additional technology or features,Platform Security. log event monitoring business intelligence,to an existing management plan to make. and alerts to enable us to more easily identify,sure event logging meets security needs We. and investigate unusual occurrences An,Identity and Access Management. STACK use a product that monitors proxy access,important adjunct to these operations was. Security Management,to management servers and provides more. Data Protection,performing application layer hardening to make. extensive logging capabilities It has been,sure the applications themselves were secure. HYPERVISOR proven invaluable when investigating security. In phase 2 we implemented five additional,and other events malicious or not. controls for a total of 19 of our 24 controls,Infrastructure Security. HARDWARE Additional Application Layer Hardening,Implementing Centralized Logging. As with all servers endpoint security is,Centralized logging provides a single search. STORAGE AND a requirement It is vital that the VMs be. BACKUP location for support personnel to use for,hardened as much as possible while still. troubleshooting problems and the events,providing the necessary functionality. leading up to them It also provides a single,because they are the most likely route. NETWORK location to examine and trap unusual or,for a security breach We configure our. questionable access events and events that,network access control enforcement points. may indicate circumvention of normal access,for example to allow only the expected. DATA CENTER AND, FACILITY SERVICES channels if these events can be identified. application flows and prevent malicious or,Centralized logging does not necessarily. accidental access through unnecessary but, Software as a Service SaaS mean only one log location It can mean that. Platform as a Service PaaS,open ports and protocols We use risk analysis. troubleshooting logs are maintained on a, Infrastructure as a Service IaaS and governance to define how the firewall. single logging host while security events are,will be configured for a particular application. monitored on another,Patch management and antivirus software. Figure 2 We found that security for the, cloud particularly when protecting high There are a number of products capable of must also be used and kept up to date. security systems requires a sophisticated monitoring logs and triggering events based. holistic approach using extensive security on pattern matching We found that the most Phase 3 Complex Network. monitoring granular identity and access important feature in their use was the ability Monitoring. management controls, to create rules to trigger alerts Although some To complement prevention and protection. products support virtualization out of the box capabilities in our environment we are. it is still necessary to duplicate infrastructure in implementing a diverse mix of network. the lab and attempt some penetration testing attack and intrusion detection capabilities In. at least some of the common script kiddie addition to using network intrusion monitoring. attacks identified on security web sites This to analyze and monitor all traffic coming into. helps identify patterns in the logs used to and going out of the HTZ environment we. trigger events and helps fine tune triggers are adding network traffic behavior analysis. Logging from the hypervisors is only one capabilities to establish normal traffic patterns. element of monitoring the environment and enable detection of anomalous activity. Because virtual infrastructure is holistic followed by the sending of appropriate alerts. it s important to monitor the parts For These efforts will add five more controls to. this reason we collect logs from network bring the total to 24 controls overall. 6 www intel com IT, Virtualizing High Security Servers in a Private Cloud IT Intel White Paper. To further broaden our monitoring coverage process of moving from a physical Implementation of. we use a Host based Intrusion Prevention to a virtual HTZ implementation we High Trust Zone Architecture. System HIPS and a Host based Intrusion took into account strict and formalized Medium Risk and Medium Consequence. High Risk and High Consequence, Detection System HIDS on the VMs to enable security requirements and mitigated 50. diverse and wide monitoring coverage To many risk factors through a variety of. Number of risks, reduce false positives that can waste valuable controls see Figure 3 and Table 1 By 40 declined from 44 to 20. IT time we carefully fine tune these solutions addressing the concerns of virtualizing. mission critical applications we expect,We expect the evolution of controls in the HTZ. to be able to virtualize 75 percent of,architecture to be a continuous improvement 20. Office and Enterprise applications,process We are working internally and with. suppliers to accomplish the following tasks Our key results include Number of Risks. declined from 27 to 7,Creating hardening requirements for 0. Move host level intrusion monitoring and Before Phase 1 Phase 2 Phase 3. virtualization products,prevention solutions to the hypervisor Controls. level rather than at individual VM level to Identifying critical gaps in the virtualization Figure 3 The implementation of our High Trust. improve resource efficiencies products features Zone HTZ architecture and its 24 controls has. successfully eliminated or substantially lowered,Delivering a hardened virtualization environment. Work with hypervisor vendors to introduce the consequences of 41 identified risks. able to handle our high value systems,granular monitoring and alerting capabilities. in the hypervisor Creating hardening requirements for applications. Successfully moving applications into the HTZ,Perform sophisticated correlation across. diverse sets of device and application logs to We also achieved some unexpected and. identify complex attack patterns and signals equally beneficial results including. Increased accuracy of application profile,data as a result of scrubbing applications. Key Results to identify the appropriate virtualization. environment,Our HTZ architecture has enabled us, to virtualize and move to our private Improved security for the applications as. cloud some of Intel s most security a result of the hardening required to land. sensitive applications During the in the HTZ, Table 1 High Trust Zone HTZ Architecture Risks and Controls. PHASE 1 PHASE 2 PHASE 3, Virtualization management infrastructure risks Operational risks Granular network monitoring. Granularity in administrative roles and access Control and protection over virtual machine Inter VM intra host traffic monitoring. Separation of privilege accounts VM images and control. Separation of duties Enhanced monitoring of virtualization Network behavior analysis and anomaly. management and infrastructure using a detection,Restricted access to infrastructure. compatible virtual appliance deployed on, Application security risks the hypervisor it s actively protecting. Security Development Lifecycle SDL for necessary Shared infrastructure segmentation. changes to application architecture to enable such as storage. taking advantage of virtualization benefits such as. portability and resilience,Risk readiness review of applications. Code auditing of virtualization management,software and dependencies. Enhanced security logging and monitoring,Virtualization as a technology. Overall project focus, Hypervisor integrity all controls work together to address security breaches. www intel com IT 7, Lessons Learned Conclusion and Our plans include deploying our HTZ. architecture in multiple data centers and,While this is still a work in progress we have. Next Steps migrating applications to it By deploying. already started collecting some of our learnings, from implementing our HTZ architecture We designed and are implementing applications into this environment we anticipate. an environment that meets our virtualizing all suitable high security applications. A holistic view of risk and vulnerability is requirements for virtualizing high in 2012 to reach our goal of virtualizing. essential Security for the cloud particularly security mission critical applications 75 percent of the Office and Enterprise. when protecting high security systems within our enterprise private cloud environment This private cloud will allow Intel IT. requires a sophisticated approach using to meet its agility and efficiency objectives. extensive security monitoring and granular With a three phase approach we first and provide better service to business groups. identity and access management controls use controls to isolate the virtualization. Virtualization technology is still maturing management infrastructure from the servers. Consequently supplementary tools and being virtualized and to protect the accounts. controls are necessary and hypervisors must used to manage virtualization In this first Acronyms. be treated like OSs and secured as such phase we also harden the OS and platforms CSA Cloud Security Alliance. and secure the applications we intend to move HIPS host based intrusion. Treat the hypervisor like an OS, In our second phase we establish controls for prevention system. The recognition that the hypervisor is, extensive security monitoring taking a holistic HIDS host based intrusion. another OS that must be monitored and detection system. approach that includes developing deep, protected has yet to propagate across HTZ high trust zone. logging capabilities and even monitoring the, the industry As a result we are forced to IaaS infrastructure as a service. management agents For the final phase,create makeshift controls to protect the. we add complex network monitoring that LAN local area network. hypervisor and implement much more, includes a diverse mix of network attack LUN logical unit number. extreme controls elsewhere to reduce the, and intrusion detection capabilities MCA Machine Check Architecture. potential for hypervisor compromise Recovery Recovery. We are currently piloting production level, Granular administrator controls are NIDS etwork intrusion detection. applications with Phase 1 and 2 controls We,required Defining custom roles versus system. expect completion of Phase 3 engineering in, built in roles are needed to ensure NIST National Institute of. the first quarter of 2012 Given the lengthy Standards and Technology. permission is restricted for tasks specific, timing 18 months we are taking a risk OS operating system. personnel are not expected to perform, management approach allowing for quicker PaaS platform as a service. More granular logging and monitoring is adoption of the environment by targeting. PEP policy enforcement point, required Because virtual infrastructure is evaluating applications that are more risk. PVLAN private virtual LAN, holistic and has many parts it s important to tolerant and helping application owners. QA quality assurance, monitor the various management agents For decide whether they want to participate in. a complete picture we found it was necessary SaaS software as a service. the planned deployment even though not, to collect logs from network switches storage SDL security development. all controls are in place,subsystems and management servers. SIEM security information and,event management,SLA service level agreement. For more information on Intel IT best practices SOX Sarbanes Oxley Act. visit www intel com it vLAN virtual local area network. VM virtual machine, This paper is for informational purposes only THIS DOCUMENT IS PROVIDED AS IS WITH NO WARRANTIES WHATSOEVER INCLUDING ANY WARRANTY OF. MERCHANTABILITY NONINFRINGEMENT FITNESS FOR ANY PARTICULAR PURPOSE OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL. SPECIFICATION OR SAMPLE Intel disclaims all liability including liability for infringement of any patent copyright or other intellectual property rights relating to use of. information in this specification No license express or implied by estoppel or otherwise to any intellectual property rights is granted herein. Intel and the Intel logo are trademarks of Intel Corporation in the U S and other countries. Other names and brands may be claimed as the property of others. Copyright 2012 Intel Corporation All rights reserved Printed in USA Please Recycle 0112 ER KC PDF 326191 001US.
Bumi Manusia novel (international title: This Earth of Mankind), a fictional novel by Pramoedya Ananta Toer, in both Bahasa Indonesia and English as translated by Maxwell Lane. Quantitative analysis supported with qualitative detailing are used to detect and
Case Management of High risk, High cost patients (eg, A1C>11, alcohol abuse, frequent hospitalizations) Determine what are the behavior barriers/drivers that cause patients to be high risk, then intervene, utilizing intensive case management for 3 months. Bahozhoocare model