Security Data Mining

The Art of Digital War - [Part 4] Understanding the Threat and Normalization

2007-01-29T10:34:00.052-05:00

Part 1 / Part 2 / Part 3 / Part 4 /

You have seen the abstract data types in the section 3 of this series. The most critical element in this is to understand the context of these atomic events, which brings in the next topic “Context based DNA – Data Normalization and Aggregation”. This modal moves away from the concept of static normalization and aggregation rules. Without data normalization in the security domain, the end user has to deal with 50,000 to 80,000 thousand different types of Alerts (this is not the number of alerts an analyst receives every second). Normalization process (across the vendors) reduces this to a manageable number (100 to 300 Normalized Alert).

These rules will be formed dynamically based on the network context, placement of the security devices, placement of important assets and the traffic flow.

Normalizing the data across the vendors is the key to build vendor neutral correlation rules. Normalization helps to understand the digital attack patterns before diving deep into actual signature produced by the devices. Without Normalization linking events across the vendors will not be possible and writing rules per vendor will leads into rule management nightmares.

Current Normalization Pyramid

Following data shows how the current pyramid works with vendor alarms.

Breaking the Pyramid

The pyramid structure was definitely a good start. However, over a period of time the digital attacks were so sophisticated and complex, the pyramid structure become very rigid. However, this doesn’t mean that normalization is not right. Normalization is the key. The question over here is can we avoid this rigid pyramid like structure and can we have something very flexible and dynamic.

Before we move onto the breaking the pyramid let us understand some of the sophisticated digital attacks and what makes them sophisticated and complex.

Analyzing the Attack by a Virus/Worms
In this section let us look at one of the sophisticated Virus “MyTob” and see its behaviors.

With the above data if you compare the behaviors of virus / worms you a see the new entrants are more sophisticated compare to the viruses of early 2000. For example, MyTob has a built in SMPT engine to send out Emails. It has a backdoor and virus author can send commands via IRC channels control the system remotely. It propagates using two completely different mechanisms (an application exploit on port 445 and using SMTP engine on port 25). Once infected its self defense mechanisms are more robust compare to the earlier ones. So, the sophistication of these worms clearly shows the need for breaking the pyramid model and come up with a more flexible and adaptive model for the data normalization.

Data Aggregation

Current set of products does aggregation based on a static aggregation rules based vendor alarms or normalized alarms. This needs to be changed and it should be based on network topology and the placement of the device. Most common fields for aggregation is based on device, device alarm, severity, source, destination, and destination port.

Take the example of a firewall placed near the border router at the perimeter. This device logs lot of Access Denied device Alarm (which is based on the firewall rule) for the inbound traffic (traffic from internet to the enterprise). Over here based on the above rule which aggregates data on device, device alarm, severity, source, destination and destination port.

However, as the firewall has already blocked the inbound connection (at the perimeter) what’s more important is to aggregate the data based on device, device alarm, and source and destination port. This will give a very good aggregation on firewall data as well as a good view on the attacker his modus of operandi.

At the same time, if the firewall allows the (inbound) connection then the data needs to be aggregated based on device, device alarm, source, source port, destination and destination port.

Following tables shows the event aggregation based on different parameters which is specific to the context of the security device and the network traffic. The table shows different rules for the Firewall (placed at the perimeter) Device Alarm Network Access Denied and Network Access Allowed.

Conclusion

The objective of this article is to show how the rigid pyramid fails handling sophisticated attacks by hackers as well as automated attacks by virus / worms. Next section follows up with an adaptive Normalization technique along with context sensitive aggregation rules.

The Art of Digital War - [Part 3] Abstract Data Types in the Security Domain

2006-09-21T05:30:00.001-04:00

Part 1 / Part 2 / Part 3 / Part 4 /

Abstract Data Types in the Security Domain

The objective of this section is to identify the key abstract data types required to handle any type of security related data to identify the Digital Intrusions / Extrusions and take necessary remediation process to mitigate those attacks.

Handling of millions of network events (generated by Routers, Firewalls, IDS/IPS etc) per day is one of the key elements of all the Security Management solutions. Other set of information collected for processing and mining the attack pattern involves OS logs, vulnerability information of an asset, network topology, Asset Database, Identity Management systems and Application Logs. So you end up having lot of different data types. One of the biggest challenges is to normalize this information across the vendors. However, before the normalization process the key element is to identify and classify the data types.

So, let me start with two fundamental data types and let us see how these data fits into all the data sources available from different vendors and creates Digital ‘Conversations’. Following are the two abstract data types.

1. Entities
2. Atomic Events

Entity[1]

An Entity is a ‘thing’ in the system with a certain properties. For example An ‘Asset’ (a machine) with a set of services (applications) running (listening on a port), or a ‘User’ with one or more roles and privileges or an ‘Application’ which is farmed out on a server farm. Attributes of an Entity usually remain static (for certain Entities static attributes also changes frequently) while state of an Entity changes frequently.

The first diagram shows the Attributes and States of an Asset (Financial Server) as an Entity. OS, Services and the CVE information is for illustration purposes only. Understanding an Asset and the services it supports is the key to figure out what needs to be protected on Network. Behind every Asset (or Services to be specific) you have very important data. Services (any application listening on a port) are the doors to the world. Any SOC (Security Operation Center) Analyst must have a clear understanding of the Key Assets and Services on the network.

The second diagram shows the Attributes and States of a User as an Entity. User, applications and CVE ID for illustration purposes only.

In the same line you can create an Application as an Entity with vulnerability profiles and network characteristics (like how does it communicates in the network and with what protocols and what kind of services it provides, impact of these protocols/services in the event of a failure etc).

The third diagram illustrates an Intruder as an Entity with a certain characteristics like Intruder Classification (Internal / External or an Automated) impact of the attacks, Attack sophistication etc. All the these characteristics with Attack behaviors and recent states of the Intruder (which includes the recent attack history, damage caused, number of incident cases results in creating a normalized Threat Score for the Intruder which is on a scale of 1-10 with 10 as the Highest Threat. Threat modeling for an Intruder is the key to identify its (Intruder’s) threat to an Enterprise.

The fourth diagram shows Virus / Worm as an Entity and its potential attributes which describes its Infection and Attack Behaviors and its ability to cause damages in the enterprise. Classification of Virus and Worms and creating a Virus Entity database helps in recognizing the similar patterns and do predictive analysis on the future Viruses and BOT threats.

So, the idea of an Entity is clear it’s identifying the ‘thing(s)’ internal / external to your enterprise with a certain set of characteristics. As I mentioned in the first part of this series, the key is Identifying your Assets (Know yourself) and identifying the Intruders (Know your Enemy). Depends on the Enterprise Network size the types Entities will wary, however the key Entities identified in this section remains same (may be with more attributes and states). Now let us look at what’s an Atomic Event and how does it ties two entities together.

Atomic Event

An Atomic Event is created when any of the two entities talks to each other. Atomic Events can’t be broken down further. For Example a Firewall Network Status Event (Connection open or Connection closed Event). Collecting millions of these Atomic Events from hundreds of security devices was a huge task in late 1990’s and early 2000. First generation of the Security Data Management Vendors concentrated on this task and did a pretty good job in collecting and centrally managing all the Atomic Events. Correlation technologies correlate these Atomic Events giving more value to the end users in identifying the internal or external threats. 90% of the Atomic Events are noise. However, these technologies need to move from Event Management to Entity Management resulting in creating environment specific context sensitive conversations.

So, the bottom line is you can’t create Atomic Events without entities. Here are the some of the examples of Atomic Events by connecting two Entities.

PIX Firewall Message
2004 21:19:23: %PIX-2-106001: Inbound TCP connection denied from 172.12.10.146/2493 to 172.169.110.21/80 flags SYN on interface outside
2004 21:19:23: %PIX-2-106001: Inbound TCP connection denied from 172.12.10.146/2494 to 172.169.110.22/80 flags SYN on interface outside
2004 21:19:23: %PIX-2-106001: Inbound TCP connection denied from 172.12.10.146/2495 to 172.169.110.23/80 flags SYN on interface outside

Conversation

Grouping of Atomic Events in a specific environment (Zones) and context creates a conversation. Read the Part 2 of this series to know more about various types of Conversations. Entities, Atomic Events, Context Sensitive Conversations will give more accurate Enterprise Security Posture to CSO / CISO.

References

[1] The Merriam Webster definition for an Entity is
independent, separate, or self-contained existence
the existence of a thing as contrasted with its attributes
something that has separate and distinct existence and objective or conceptual reality

Work in Progress ......

The Art of Digital War - [Part 1] Digital Intrusion Timeline

2006-03-23T00:05:00.001-05:00

Part 1 / Part 2 / Part 3 / Part 4 /

Digital Intrusion Time Line

The objective of this article is to identify the issues around a digital intrusion. The following diagram shows the picture of a digital intrusion time line (by an internal or external Intruder or an automated Intruder – virus / worm / bots etc) along with the Vulnerability time line and security monitoring tools with current features and future building blocks. The focus is on the fundamental problems, and it will not go into analyzing different digital attack patterns or any vulnerability analysis.

Latest CERT reports a total of 5990[1] vulnerabilities for the year 2005 an increase of 58.5% from the year 2004 and a 3402% increase from the year 1995. Usually vulnerability in an application is due to un-identified bug in the code. However there are times when backdoors written explicitly in some application to get into a users machine.

An intentional backdoor into any system is more dangerous than an accidental bug due to an oversight or bad coding practices. Huge debate gone over the recent WMF[2] (Windows Meta File) Vulnerability – Microsoft Security Advisory (912840) whether it’s an intentional backdoor or not.

“Speeding up the patch process is never going to solve the problem; it is never going to be fast enough. We need to be investing very heavily in zero-day defenses, because another zero-day will happen. There is a lot of talk about whether (the software vendor has) gotten the patch out in time, but the real conversation should be about risk removal, not risk mitigation.”

Richard Ford, associate professor of computer science, Florida Institute of Technology

“Application vulnerabilities propagate so rapidly today that the old methods of dealing with them no longer suffice. New standards like AVDL offer one of the best hopes of breaking this cycle by dramatically reducing the time between the discovery of a new vulnerability and the effective response at enterprise sites”

John Pescatore - Vice President of Security Research, Gartner

Security Threat Modeling

Security Threat Modeling is an essential process to protect the Assets (or applications). It helps the organizations to determine the correct controls and produce effective counter measures within the budget. Effective management and understanding of the vulnerabilities is required to efficiently defend attacks against those (vulnerabilities). As the number vulnerabilities increases year by year the customer needs a mechanism to identify the most critical vulnerabilities in his enterprise.

The Core of Digital Security

The three key things in digital security for the enterprise are identifying and classifying the Intruder and their attacks on the Assets and the Damage it can cause on the enterprise or the potential damage on the similar attacks in the future. Regulatory compliance and other government regulations revolve around the core or rather monitoring the health of the core.

The above image shows the Intruder attack sophistication and the incident time line which starts when the intruder finds the vulnerability in the enterprise and the actual break-in and the damage he causes by information leakage, denial of service on critical systems, and attack on other systems etc.

The Defense sections shows the 3 phases which is as follows; the Monitoring phase, Attack discovery on the assets and the Containment and the Remediation process. The key will be how efficiently we can correlate and provide relevant information back to the end user at the right time so that he/she (the analyst) can stop the attack (while in progress) before it wrecks havoc in the enterprise.

The three core areas (Intruder, Assets and Damage) will remain same today (2006) or even after 15 or 20 or 2000 years. What matters is how good we are at identifying these three key elements and build a robust Security Threat Model around it.

Intruders and their Attacks

Classification of an Intruder is critical in understanding the Threat the intruder posses. A good Security Threat Model needs to understand the strengths, weakness and the attack methodologies of any Intruder. The Intruders are classified into 3 – Internal, External and Automated (Robotic) Intruder. Classification of Intruders helps us to prioritize the incidents and focus on the relevant incident.

Assets

Security revolves around protecting the Assets (Behind every Asset there will be some applications). Asset oriented Security Monitoring will be the key in this evolution. Application infrastructure of the future will be heavily distributed in nature with SOA (Service Oriented Architecture). Protecting the business services will be the most important aspect in the service oriented world.

Asset Oriented Security Monitoring will eventually move towards applications and in the future will lead to protecting the collection of web services[3] which the applications published. Security will go down to the fabric of the distributed applications. According to Forrester the ERP[4] Market will be $24 Billion by the end of 2008. SAP[5] and Oracle the leading ERP Application providers will be moving to Service Oriented architecture by the end of 2008.

Classification of assets is important to protect the assets efficiently. Asset value will not yield this classification. For example an asset which contains blog and user forum data will be classified differently compared to assets with financial transaction databases. There will be assets which require protection while data at rest[6] as well as protection of data on the wire.

Damage caused by Incidents and its impact

The above chart and depicts the damage impact if a break in happens. Today the users do the impact manually and lot of different software applications will be used in the complete process. Streamlining this business process and using this data to further improve process will help in quick remediation and containment.

Tracking the cost of Incidents, resources required for containment and remediation, and the time spent will help in predicting the actual cost involved if the similar attacks happens in the future. This information can be used in the Security Threat Model to narrow down the attacks and vulnerabilities where the potential damage will be very high.

Digital Security - Building Blocks

The first generation of security management tools processed data from security devices like firewalls, intrusion detection systems, vulnerability scanners apart from network devices like routers and switches. Correlation technologies correlated the events across the systems. However, these systems focused more on handling the events. This model is an extension of log management systems which started of the Digital Security Management space.

The second generation Security Management tools focuses more on entities like Assets and its relevance, Network and its importance, Attacker (with classification) and threat levels, Vulnerability Severity relevant to the network. This model deviates from the first generation event based management as the focus is on the entity rather than the events.

Entity model in the second generation simplifies the process of building a Security Threat Model compared to first generation event model based Risk or Threat Scores. The CSO[7] / CISO are focused more on protecting their assets instead of worrying about how many events passed through the network.

The third generation of Security Management will move closer to where the real action in the enterprise digital world – ‘The Applications’. As per the Forrester and Gartner[8] most of the enterprise applications will move towards SOA[9] (Service Oriented Architecture) by the end of 2008-2009. Cisco already announced the Cisco AON (Application Oriented Network) Architecture where the focus is on routing the application specific traffic.

End of the day security is all about protecting the data (information or knowledge) created by the applications (Assets in the enterprise) and the applications runs 24/7.

The Fourth generation of Security Management will see the convergence of physical security with information security. As per Forrester forecast[10] Security Convergence spending for Europe and North America combined will be $11 Billion dollars in 2008 compare to $506 million in 2004.

Conclusion

The objective of this article is to highlight the core of digital security and the expectations around the core. Around 30-40 years ago we knew that the fundamentals of Atom[11] are electron, proton and neutron. As the science progressed we realized that protons and neutrons were made up of quarks[12] and discovered hundreds of sub atomic particles[13] and then finally to ‘Strings’ and the String theory[14], However, electrons, protons and neutrons still remains as fundamental particles (at atomic level).

So, let me re-instate the core (AID) again.

Assets (Know yourself)
Intruders (Know your enemy)
Damage

Do we think the above three elements will change in the year 2131[15]. The answer is a big ‘NO’.

There will never be a silver bullet which will solve all the problems. What you can do is to improve the probability of successfully defending any attack. After so much of advances in medical sciences the common cold still exists!

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.

The Art of War - Sun Tzu. Lived: 500-320 BC

[1] 2005 Vulnerability List http://www.cert.org/stats/cert_stats.html

[2] WMF Vulnerability – MS Advisory 912840 -

http://www.microsoft.com/technet/security/advisory/912840.mspx

Security Focus - Zero-day WMF flaw underscores patch problems by Robert Lemos – January 12, 2006

http://www.securityfocus.com/news/11368

[3] Forrester – Large Enterprises Pursue Strategic SOA by Randy Heffner - April 5, 2005

http://www.forrester.com/Research/Document/0,7211,36580,00.html

[4] ERP Apps – Technology and Industry Battle heats up by Paul Hamerman, R Wang – June 9, 2005

Site: http://www.forrester.com/Research/Document/0,7211,37058,00.html

[5] SAPs Big Bet To Revolutionize App by Erin Kinikin – August 3, 2004

http://www.forrester.com/Research/Document/0,7211,34739,00.html

[6] Forrester Wave – Data Encryption Solutions Q3, 2005

http://www.forrester.com/Research/Document/0,7211,36486,00.html

Application Security – http://www.appsecinc.com/

Encryption of Data at Rest - http://www.appsecinc.com/presentations/Encryption_of_Data_at_Rest.pdf

DMReview – Information Management: Encryption at Rest

http://www.dmreview.com/article_sub.cfm?articleId=1033567

[7] CSO Online - http://www.csoonline.com/research/leadership/cso_role.html

[8] Gartner – http://www.gartner.com/

Future of Enterprise Security – September 15, 2004

http://www.gartner.com/DisplayDocument?ref=g_search&id=454567

Cool Vendors in Security and Privacy – March 28, 2005

http://www.gartner.com/DisplayDocument?ref=g_search&id=475999

[9] Forrester – Your Strategic SOA Platform Vision By Randy Heffner – March 29, 2005

Site: http://www.forrester.com/Research/Document/0,7211,35951,00.html

Development Roles In The World Of Service-Oriented Architecture – January, 13, 2005

http://www.forrester.com/Research/Document/0,7211,35822,00.html

SOAP Vs REST – A Comparison – By Randy Heffner, September 13, 2004

http://www.forrester.com/Research/Document/0,7211,35361,00.html

Forrester Wave – Enterprise Service Bus Q4 2005

http://www.forrester.com/Research/Document/0,7211,36162,00.html

[10] Forrester - Trends 2005: Security Convergence Gets Real By Steve Hunt – January 11, 2005

http://www.forrester.com/Research/Document/0,7211,36137,00.html

Converged IT And Physical Security: Small But Real – By Laura Koetzle April 15, 2005

http://www.forrester.com/Research/Document/0,7211,36680,00.html

[11] CERN – The worlds largest particle physics lab - http://public.web.cern.ch/Public/Welcome.html

[12] Stanford University – Quarks Theory http://www2.slac.stanford.edu/vvc/theory/quarks.html

[13] Getting closer to the God Particle - http://arafkarsh.blogspot.com/2005_02_01_arafkarsh_archive.html

[14] String Theory - http://www.superstringtheory.com/index.html

[15] What is so peculiar about this year?

The Art of Digital War - Virus / Worm Analysis

2006-03-22T19:10:00.000-05:00

According to Richard Bejtlich (Author of Tao of Network Security Monitoring and Extrusion Detection) there are five phases of compromise (of an external attack).

Reconnaissance
Exploitation
Reinforcement
Consolidation
Pillage

Guess what: Viruses of the new era has attack models, similar to a sophisticated Hacker (or Cracker). It scans your network, exploits your vulnerable applications, creates backdoors for control, and does DoS (Denial of Service) attacks against other systems and even fights other viruses and worms to show supremacy!

Here is a brief summary of the Virus / Worm Behavior Analysis document (Adobe PDF document Size 105K) I prepared as part of my research on Virus behaviors around 8 months ago (in August 2005).

I was thinking about creating a virus database based on these attributes (my spreadsheet contains close to 250 attributes to understand the virus behavior).

Watch this space for more details… ..

The Art of Digital War - [Part 2] Alarms in Digital Intrusion

2006-03-09T00:10:00.001-05:00

Part 1 / Part 2 / Part 3 / Part 4 /

The following table shows a set of words and its frequency. I know you must be wondering about, frequency related to what? That’s the key!

Word	Frequency	Word	Frequency	Word	Frequency
The	1101	That	389	As	228
And	878	Is	334	Be	226
To	726	Not	315	Lord	218
Of	657	This	296	He	216
I	561	His	292	What	203
You	544	But	265	So	197
My	508	With	257	Him	189
A	498	For	247	Have	179
In	414	Your	242	….	……
It	414	Me	235

The above table shows the word count from Shakespeare’s Hamlet!

If we put these words in the ‘RIGHT CONTEXT’ you will get a classic in English literature.

‘To be or not to be: that’s the question’ is a famous quote from Shakespeare's Hamlet by Prince Hamlet (of Denmark) in a self conversation mode. Other Conversations include his (Hamlet’s) conversations with The Ghost (his assassinated father), his friend Horatio, The King Claudius etc. So, the words arranged in the Right Context, create the scenarios and build the conversation with various characters (Hero, Villain, Friends, Lovers etc) which results in the complete story.

Now let us look at the current set of Intrusion Detection Systems. It generates Alarms with some severity (and lots of them are false positives) and the current breed of Enterprise Security Management software’s do some basic algorithms to do the scoring which ends up similar to the data in the following table.

Alarm	Risk Score / Priority / Event Count
Alarm 100	1101
Alarm 12931	878
Alarm 14987	726
Alarm 231	657

If you compare the two set of table it doesn’t tell you exactly the story behind those words (or Alarms). Security Analysts with their experience and intuition runs through these Alarms and creates a mental map of a potentially story. However, what matters or what the industry or the Security Analysts wants, is to put these words (Alarms/Events) in the ‘RIGHT CONTEXT’!

Therefore, why don’t we look at this data set from a different point of view? i.e., instead of Security Events, Why don’t we try to see a ‘Conversation’?

For Example.

1. A financial user using his Financial Application for his routine daily job.

2. Customers accesing the Web Application

3. Inter department communications.

All these have a definitive start and end segments. However currently we look at all these transactions as events and then we do correlate based on these events.

Now why should we see events? Why can’t we see a conversation?

What is a Conversation?

A Conversation happens when a user logs into a system do certain activities and then logs off.A Conversation could be for few seconds to hours, depends upon the nature of the conversation. A Conversation could be polite or rude! So the idea is clear. A Conversation shows a set of events in itslogical order (or grouping them in its logical order).

If we look at the usual network traffic, most of raw events (Alarms generated by the security devices) can be grouped under a certain types of conversations. For Example;

1. Business Conversation
Legitimate web users or business partners accessing the Application server (Web Services).

2. Inter - Department Conversations
Understanding the applications which communicate across the department (in normal office hours)

3. Personal Conversations
Employees browsing the web pages, checking personal emails etc

4. Un-known Conversation
This type of conversations is the one which doesn’t fit into current business rules or policies.

5. Rude Conversation
An Attacker scanning a server and compromising it

6. Impolite Employee Conversation
Employees breaking the security policies

7. Harmful Robotic Conversation
Self propagating worm attack

Data doesn’t tell a story unless it is interpreted in the right way.