Where Logs Hide: Logs in Virtualized Environments By Dr. Anton Chuvakin WRITTEN: 2008 DISCLAIMER: Security is a rapidly changing field of human endeavor. Threats we face literally change every day; moreover, many security professionals consider the rate of change to be accelerating. On top of that, to be able to stay in touch with such ever-changing reality, one has to evolve with the space as well. Thus, even though I hope that this document will be useful for to my readers, please keep in mind that is was possibly written years ago. Also, keep in mind that some of the URL might have gone 404, please Google around.
This paper describes log management in virtualized environments-its challenges and opportunities. We will cover the similarities and differences in logging for virtualized environments versus physical environments. Introduction to Logging A beaten maxim proclaims that ―knowledge is power,‖ but where do we get our knowledge about information technology (IT) components such as computers, networking gear, application frameworks, SOA web infrastructure and the like? The richest sources of such information that is always available but often overlooked are the logs and audit trails that are produced by these systems and applications. Through logs, audit trails and various alerts, information systems often give signs that something is amiss, or an event logged in the log files provides insight into future problems. Logs can also reveal larger weaknesses –that may affect regulatory compliance and even IT governance, and, by extension, corporate governance. However, more often than not, it’s difficult to extract information from log files and distil the data into useful and usable or actionable information. To start from the very high level, logs equal accountability. Wikipedia defines accountability as " a concept in ethics with several meanings…often used synonymously with such concepts as answerability, enforcement, responsibility, blameworthiness, liability and other terms associated with the expectation of account-giving." There are many other mechanisms for accountability in an organization, but logs are the most prevalent. And if your IT staff is not accountable, neither is your business. Unless you take logs seriously, you may be sending out the message that your organization shuns accountability. Along the same lines, logs are also immensely valuable for meeting regulatory compliance. Many recent US laws including HIPAA, GLBA, Sarbanes-Oxley (SOX) and others have requirements related to log auditing and the handling of those logs (see my papers ―Log management in the age of compliance‖ and ―Six Mistakes of Log Management‖) Let’s take a look at virtualization and what it means in terms of log collection and retention. Introduction to Virtualization
Server virtualization makes it possible to combine multiple diverse systems onto a single hardware platform, thus shrinking server, storage and networking costs, reducing power requirements (through a direct decrease in consumed energy and cooling costs), increasing utilization of existing computing resources and improving productivity. The impact is significant; Garter reports savings of up to 25 percent due to server consolidations and decreased hardware purchases. Virtualization also simplifies server provisioning, increases the average workload per server and shrinks server administration workloads, reducing the amount of required hardware purchases. Organizations save money through better hardware utilization. Simplified backup and recovery is also possible, because virtual machines can be brought back online much faster than physical machines. Virtual platforms and their management tools enable the smooth transition from a physical to a virtual environment. It all sounds good, but what happens to logs, logging and log management when IT environments are virtualized? Logging Meets Virtualization As one can guess, virtualization platforms present new sources of logs to manage. In addition to having new log information to collect and analyze, we new challenges to logging and log analysis arise, such as the potential need to review access logs collected while virtual machine images were inactive. In addition, new opportunities for log management are also present, such as ensuring new virtual images are pre-configured with central logging capabilities. There may be ways to use logs to solve new problems, such as monitoring health and uptime status of virtual platforms and application stacks. The ubiquitous nature of log management allows the development of new operational, security and compliance solutions for virtual infrastructures using the tools we already have. What stays the same? First, let’s review what stays the same. A virtual server is still a server – complete with operating system and applications, and logs that must be collected, retained (for security and compliance reasons) and analyzed, just as they do in ―physical‖ environments. The rest of IT infrastructure stays the same: Routers still route network traffic, switches perform switching, firewalls and other network security devices perform their functions on network traffic, etc. In other words, IT infrastructure with virtual platforms, hosts systems and guest systems are largely the same as those with all physical elements; with all the usual logging that needs to be managed. Similarly, networking between guest systems running on a single virtual platform resembles networking between physical machines, and needs to be monitored and audited just like on a physical network. In a virtual environment, servers are still provisioned, modified and configured by system administrators, and of course accessed and utilized by end users. Such activities create audit trails that are collected and reviewed in just the same manner as are physical environments. For example, if an MS SQL database server is running on Windows 2003 operating systems, but this Windows system itself sits atop of a Linux-based VMWare host, both Windows logs and MS SQL audit trails must be collected and analyzed for access violations, new user accounts, data access attempts or unauthorized changes to
database structures. In short, the advent of virtualization is not a reason to throw away tools that work for you in physical environments. They will continue to deliver value and help your IT and business to operate efficiently, be secure and compliant with relevant regulations, especially given the fact that the future belongs to a mix of physical and virtual environments. What changes? On the other hand, virtualization has brought a lot of new technologies (all with their own logs) as well as new problems for IT departments to solve. Such problems might not have any equivalent in the physical world, where ―a server‖ always meant ―a piece of hardware‖ plus ―an operating system‖ plus ―one or more of user applications‖ running on it—a worldview that virtualization is making obsolete. A virtual platform comprises a hardware platform, operating system and a hypervisor, or virtual machine software that enables other systems to run on top of it. Such a setup gives way to several major changes: 1) New logs include hypervisor application logs, record virtualization-specific activity logs (new guest image creation, guest operating systems startup, patch access, etc). These logs must be understood by log management tools as well as the virtual machine administrators. 2) Aggregation of servers on one hardware platform calls for stricter availability monitoring. Indeed, recovering a virtual machine image from backups might be relatively simple, but availability monitoring must still be stringent. Log management tools and possibly other monitoring tools must be deployed with real-time alerting to notify the administrators of impending fault and possible crashes or problems. 3) Stricter host platform security monitoring will help reduce the risk of breaches into the virtual infrastructure world. Extensive logging, log collection and analysis will allow thorough incident investigation. Such logs include security incident response and forensics activity across virtual farms, as well as across massive SAN arrays that house virtual machine images. 4) Management tools that enable organizations to deploy and control virtual server farms introduce their own logs and logging challenges. For example, logging the activities of server administrators means recording the provisioning, configuration and status changes of virtual machines performed via such management tools. 5) As virtual machines proliferate across an enterprise’s IT infrastructure, physical hosts are retired, an new technologies must be used to secure and manage the virtual machines. Activity such as patching, management, configuration and deployment and migration of virtual machines must be logged and monitored, just like in a physical environment. Controlling and auditing these virtualization-specific activities makes another excellent use case for logs. Beware of Rogue Virtual Machines
Finally, ―rogue‖ virtual machines pose a unique security problem. If users provision their own virtual machines and their own guest systems, tracking such activities across the organization, presents a worthy challenge – for example, if a unauthorized application, that would otherwise be banned, runs in its own virtual image, enforcing the security policy becomes harder since endpoint monitoring tools might not see through the virtualization veil. Rogue machines deployed ―in the cloud‖ via Amazon web services, for example, present the ultimate challenge of this type. If a system resides on somebody else’s virtual platform in the cloud, the chances of getting evidence of activities on such systems becomes next to impossible. Logging and Virtualization—The Good, the Bad and the Ugly At this point it should be clear that changes that IT staff must face as virtualization becomes a reality in the datacenter are indeed massive. For IT staff tasked with logging activity across the infrastructure, these changes can be good, bad or ugly: 1) They’re good because it’s easier to provision systems with centralized logging already enables. IT staff can also retrofit other systems by adding logging to the virtual image of that system. Moreover, current logging tools such as LogLogic will still work – a major good point. 2) They’re bad—or partly bad—because there are new logs to collect and analyze and new activities to track and monitor. Virtual machines must be closely watched for availability and security issues and to ensure they comply with policies and regulations. 4) They’re ―ugly‖—sometimes, because unmanaged virtual machines can pop up on the organization’s systems or even in the cloud, violating IT policies and presenting significant enforcement and investigation challenges. Logs Help Virtualization In addition to being affected by it, logging and log management can also augment virtualization projects, especially in the areas of security, compliance and manageability.. Security: Logging creates a trail of accountability for users and, especially, those privileged to access the underlying hypervisor. Tracking access to virtual machine hosts system and inactive guest images creates a trail that can be used for monitoring and auditing, as well as investigations for cybercrime or insider abuse. Perusing logs for security-relevant failures, such as missing controls, unauthorized access or unapproved changes is just as helpful in a virtual environment as it is in a physical environment. Compliance: Recent mandates such as PCI DSS and others require logging, log collection and retention, log analysis and review, and log protection. For example, logging is one of the 12 PCI requirements (Requirement 10), whether the environment is physical or virtual. Hence, logs from virtual machines must be given at least as much importance as logs from physical environments Manageability: Administrators and system operators benefit from logging, as well. Monitoring for failures and errors as well as general virtual machine health is not possible
without effective log management. Conclusion Along with all the promise and benefits of a virtual infrastructure comes significant change, requiring new ways for organizations to collect and manage logs. However, existing log management tools such as LogLogic log management appliances can still be leveraged to address these new logging challenges, and to optimize, secure and bring into compliance newly virtualized IT infrastructures.
ABOUT THE AUTHOR: This is an updated author bio, added to the paper at the time of reposting in 2009. Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in the field of log management and PCI DSS compliance. He is an author of books "Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy II", "Information Security Management Handbook" and others. Anton has published dozens of papers on log management, correlation, data analysis, PCI DSS, security management (see list www.info-secure.org) . His blog http://www.securitywarrior.org is one of the most popular in the industry. In addition, Anton teaches classes and presents at many security conferences across the world; he recently addressed audiences in United States, UK, Singapore, Spain, Russia and other countries. He works on emerging security standards and serves on the advisory boards of several security start-ups. Currently, Anton is developing his security consulting practice, focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations. Dr. Anton Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys. Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with educating the world about the importance of logging for security, compliance and operations. Before LogLogic, Anton was employed by a security vendor in a strategic product management role. Anton earned his Ph.D. degree from Stony Brook University.