[4] Snort Configuration: preprocessing punk packets : preprocessor

Preprocessing Punk Packets

Snort relies on a packet-sniffing subprocess using the pcap library of functions, which it gets from either the Linux libpcap library (see Chapter 4) or the Windows WinPcap driver (see Chapter 5). Snort is a network packet sniffer, so it works by detecting well-defined patterns (signatures) as network traffic flows by. As Figure 9-1 shows, preprocessing is the first stop on the long train ride for a packet through Snort’s systems.

The signature analysis that most people associate with a network IDS comes into play with Snort’s detection systems, which are built by using the rules files explored in Chapter 8. Snort has a robust and flexible system of rules; that flexibility puts a lot of bullets in the belts of security administrators.

This low overhead and plug-in architecture gives Snort such a remarkable efficiency. You can turn on or off different elements as needed for every installation, which is one reason why Snort fits “just right” in most environments. If you don’t need certain functions, turn them off. Preprocessing is one of the shining examples of that philosophy in action.

Figure 9-1: Follow Snort’s nose as data packets travel through its many systems.

Defining preprocessing

 Remember  Preprocessing fits into this whole quilt at the lowest level: the network layer. The purpose of preprocessing is to remove as much work as possible from the detection engine and associated plug-ins (which are far more processor intensive) by removing packets that just waste Snort’s time.

 Tip  Imagine having a million tiny balls of three different colors — red, blue, and yellow — and each ball has a number printed on it. Preprocessing is like sorting the jumbled balls into three trash cans by color and then looking through the pre-sorted balls for a specific number. If you know that only red balls can represent an attack in progress, then it makes no sense to spend time carefully checking the numbers on any of the blue and yellow balls.

Understanding the benefits of preprocessing

Preprocessing performs lots of useful tasks for a well-oiled Snort IDS. It provides a clearer picture of a stream of communications between the computers, instead of just relying on a single packet without any context. When two computers are exchanging information, the “conversation” that they’re engaged in is usually called a session. Sessions have a well-defined beginning and end, as well as rules for who can speak and at what time.

Session analysis reduces both false positives and false negatives and gives the detection engine more visibility of the kind of behavior that is actually occurring. Packet inspection with preprocessing can reveal a lot about what tricks those wily hackers are employing to subvert the integrity of network communications, and possibly evade network security sensors.

 Technical Stuff   Statefulness explained

The most useful preprocessor plug-ins are the ones that involve stream reassembly and stateful inspection. Because the normal method of communication using TCP/IP can be likened to a bucket brigade (little pails of info with “to:” and “from:” addresses being carried back and forth) they’re considered a “conversation” only when viewed all together, in a stream. Stateful inspection allows Snort to understand that this stream of packets is a conversation, rather than just random, individual packets.

Because Snort’s original operation was as a packet scope, inspecting each pail being carried, none of the context the communication was ever imparted on the detection engine. In other words, some of the information was lost. You’ve probably heard of state preservation or stateful inspection more from firewall vendors than from IDS vendors. But the term means the same thing whether you’re talking about firewalls or intrusion detection. You can figure out a lot of problems simply by tracking what kinds of packets are transmitted, what flags are set, and what the different elements of the envelope (the packet header) are.

By using stateful inspection, Snort can determine whether someone is portscanning (a process that usually is part of a hacker’s initial reconnaissance while getting the lay of the land). Also, Snort can detect stealth or malformed packets, which are more likely indications of probes sent out from a would-be attacker. By comparing some of the basic building blocks of a packet across time and across different hosts, the preprocessor plug-in can get a tidy view of what would otherwise be invisible behavior. This powerful benefit is one of the many imparted by using preprocessing to round out your Snort installation.

Looking under the packet magnifying glass

When normal network communication transpires (for example, when someone is browsing for a movie showtime on Yahoo!), little bits of data go back and forth in a well-defined, choreographed dance. Some advanced packet preprocessors detect ill-formed and inconsistent packets, which are usually indicative of illicit behavior. Such network scanning tools as Nmap and Strobe allow for the manufacture of fake packets that can cause information leaks and system crashes.

Detecting anomalies

Snort implements its preprocessing functions at the decoder level after the packet has been broken out into its major fields and before the rule-processing engine does any pattern matching or rule analysis. This processing order gives you the most flexibility and keeps Snort operating at near-wire speed.

When you implement more and more front-end analysis, speed is sacrificed. In essence, the more examination you do, the slower the entire system, but, the better the results. One of the most important themes with tuning Snort — or any other IDS — is trying to find that fine line between too much noise and missed attacks: Eradicating false positives should be your first and foremost goal.

This goal makes Snort’s method of preprocessor plug-in design a well thought- out and attractive option. If you want to expend the extra resources to scour packet streams for more data, simply turn preprocessing on. If you don’t want the hassle, turn preprocessing off.

Keeping packets in a row

In this section we explore how we can setup and use the best of the preprocessors that come standard with your Snort distribution: stream4 and frag2. Without stream4 and frag2, your Snort system would be oblivious to scans against your network and attempts to bypass your IDS or firewall.

The stream4 preprocessor

The stream4 preprocessor was built to help Snort get a better view of TCP sessions by providing stateful inspection and session reassembly. According to its original design, Snort is basically stateless. The stream4 preprocessor, when enabled, allows Snort to monitor thousands of concurrent sessions and has the added flexibility of activating state management for user-defined ports. By restricting the preprocessor to only a few ports, you can save a lot of processing power, which would normally be wasted on trying to keep track of potentially millions of sessions in a large-scale network.

Also, you can set stream4 to alert you when sniffed packets aren’t part of any session at all. This event really only occurs when a hacker is trying to masquerade as one of the computers involved in a session or is trying to insert himself into the communications stream, often called a man-in-the-middle attack.

Stream4 also allows the construction of rules using the flow keyword that can identify the direction and state of the traffic. These rules are especially helpful in determining how sessions begin and end and whether an attack is inbound or outbound from your network, as well as giving some indication as to the nature of the session. (Chapter 8 shows rules building and testing, including many rules that rely on the stream4 preprocessor for a complete analysis.)

Configuring the stream4 preprocessor

 Tip  To enable stream4, open your snort.conf file with a text editor and make sure that the following line is uncommented:

preprocessor stream4: detect_scans, keepstats machine

The stream4 preprocessor has several different options, which you include as a comma-separated list, as shown in the preceding example. The following options govern what you can configure the preprocessor to look for (including portscans and state problems):

• detect_scans: The detect_scans option (which is normally set to Off if it’s not included on the configuration file) instructs the stream4 preprocessor to alert when a portscan is attempting to avoid detection by using stealth techniques. Because the regular way TCP/IP traffic works involves a three-step handshake, which many types of stealth portscanners intentionally fail, they’re snooped out by using this stream4 parameter.

• detect_state_problems: The detect_state_problems option (which defaults to Off if not explicitly set) instructs the stream4 preprocessor to analyze how the state of a flow of TCP packets is kept. This feature is intended to catch faults or failures if the state mechanism of a TCP session is somehow altered by a peer. Given how noisy this parameter can be, we don’t recommend it unless you intend on doing some heavy in-depth analysis. Although good at detecting packets that are malformed, many implementations of TCP/IP have small variations that trip this sensor.

• disable_evasion_alerts: The disable_evasion_alerts option is an advanced setting that detects special cases where an attacker tries to fool an IDS detection engine into ignoring a packet, but the packet gets to the target.

   Tip  We recommend leaving this option off (which is the default). It can generate many false positives and eat lots of processing power.

• ttl_limit: TTL means Time To Live and is a common term used when talking about packet transmissions over a network. A TTL setting can help keep tabs on how much time a packet flow takes to reach its destination. Sometimes an attacker tries to evade detection or masquerade as being somewhere else by twiddling the TTL settings with a session. You can use the ttl_limit option to alert you to a big variation in the TTL setting across a stream of traffic. This parameter is hard to tune properly, but it’s a safe bet to use 10 as a starting point as the maxi- mum TTL.

• keepstats: The keepstats parameter accumulates a set of statistical data on the connection tracking and session state analysis, which can be logged with either the machine keyword (which actually is a text file) or the "binary" keyword, which tools such as Barnyard can read.

• noinspect: To curb excessive processing on a busy Snort installation, the noinspect switch can restrict the stateful inspection to only those ports listed with the stream4_reassemble preprocessor.

• timeout: The timeout parameter sets an idle time after which stream4 stops monitoring a particular session. Basically, if Snort doesn’t see a packet as part of an active session in its table within the time specified by the timeout switch, then that session is flushed from memory. Because this option defaults to 30 seconds without even specifying it, Snort’s normal behavior is to only perform stateful analysis of traffic that has been active within a 30-second window.

• log_flushed_streams: The stream4 preprocessor works by building a session block from the little packets that comprise the entire stream. Using this method, the preprocessor can look for anomalous behavior and perform rule testing. This session block, which is kind of like a gigantic packet, can be flushed to disk for troubleshooting and further analysis. The log_flushed_streams parameter to stream4, if used, instructs the preprocessor to drop the session block in the logs when it triggers an alert.

Our advice is to experiment with the preceding options, see what works well for you, stick with it for a time, and then go back and tune as attack trends change.

Stream4 and session reassembly

Somewhat similar to session tracking (see the preceding section), Snort’s stream4 preprocessor also supports full session reassembly. By keeping a window of packets that comprise a session in memory, Snort can alert you to attacks that span multiple packets.

TCP connections can have data fragmented across a group of packets, while UDP transmissions are required to contain all the data in a single packet. Many applications that use TCP for their transfer medium are interactive in nature and often have lots of fragmentation. SSH and telnet session are notorious for splitting data in this way.

By reassembling packets, Snort’s analysis and detection engines can catch the sneakiest of attacks. For example, say that you wanted notification when someone transmits a sequence of characters containing /etc/passwd (the location of the users and groups on a Unix operating system) over a telnet session. Telnet sends a separate packet for each keystroke. So, in essence, what you’d have is thousands of little packets, with one character in each, which is horrible for a rule-matching system like Snort. Reassembly globs all of those individual packets into a giant packet to which the rules engine can analyze.

Session reassembly is set up in much the same way as the stream4 preprocessor (which we discussed in the preceding section). By adding the following line to your snort.conf configuration file, you can enable the reassembly features.

preprocessor stream4_reassemble: both ports 21 23 80 110

The following options are available in the stream4_reassemble preprocessor:

• clientonly, serveronly, both: The first parameters given to stream4_reassemble identify which sides of the connection to conduct reassembly on.

  • Clientonly refers to traffic inbound to what you’ve defined as $HOME_NET in your snort.conf file.

  • serveronly refers to traffic outbound.

  • Both means reassembly of everything.

• ports: The ports parameter directs Snort’s stream4 preprocessor to restrict its reassembly activity to just the ports identified with this switch.

  • By using the default keyword, Snort performs reassembly on ports 21 23 25 53 80 110 111 143 and 513.

  • The keyword all performs reassembly on all ports, which we don’t recommend (except for short periods of testing).

• noalerts: The noalerts parameter tells stream4 not to report on strange or problematic issues encountered during reassembly. For example, traffic manually inserted into a stream or modified packets is detected and logged, and stream4_reassemble generates an alert, unless this option has been indicated.

The preceding example of stream4_reassemble usage should be enough for most people. If you either use other protocols where you want reassembly to occur or run telnet, FTP, HTTP, or POP on nonstandard ports, change the ports listing to the appropriate application ports for your configuration.

The frag2 preprocessor

Because many varieties of devices are out there (and many can transfer data over TCP/IP), subtle implementation differences result in packets that must sometimes be reorganized and chopped into smaller packets, a process that’s called fragmentation.

 Technical Stuff  To a network-based IDS, fragmentation can result in false negatives (real attacks that are missed) because the attack actually spans several packets. The original offending data may have been in a single packet at the beginning of its journey, but through the normal process of routing, it may have been redistributed over several packets, which hides it well. Such fragmentation poses a special problem for Snort, but, thinking ahead, the Snort team has an answer: the frag2 preprocessor.

By enabling frag2 in your snort.conf file, Snort’s packet decoder uses a similar reassembly process for reconstructing fragmented packets, before submitting them to the detection engine. It takes the pieces and puts Humpty Dumpty back together again.

To enable the frag2 preprocessor, use a configuration like this:

preprocessor frag2: timeout 60

Normalizing network traffic

The decoding and normalization of certain types of network traffic is an important preprocessing chore. Pattern matching systems like Snort can fail when an attacker introduces subtle variations. These variations are perfectly acceptable and even warranted in most cases, but they can be misused by attackers.

There are many ways to encode a Web page address into a URL It’s possible for two encoded URLs to look completely different to the human eye, but are the same as far as a Web server is concerned. For example, the following two URLs are exactly the same, yet they look entirely different.

http://www.somewhere.tld/cgi-bin/form-mail.pl?execstuff
http://0/%63g%69%2d%62in/%66%6fr%6d%2d%6d%61%69l%2e%70l?%65%78%65%63%73%74uf%66

Of course, these different notations can easily trip up a Snort rule, which matches against an exact pattern. In the preceding example, consider that a rule matches the form-mail.pl string of bytes, which generates a Snort alert. The second URL would sneak by the detection engine, though it’s just as lethal as the first.

Snort’s normalization preprocessors are an excellent way to combat these open doors. The following sections cover three of these preprocessors in more depth: HTTP, telnet, and RPC (Remote Procedure Call).

http_inspect: a preprocessor for HTTP

With Snort 2.1, the http_decode preprocessor gave way to the http_inspect preprocessor. If you use Snort 2.0, use http_decode. If you use Snort 2.1 (or later), use http_inspect.

 Tip  We recommend installing and using the http_inspect preprocessor. Using http_inspect normalizes all packets containing different forms of HTTP communication into a state that Snort can easily compare and scan through its rules. A huge amount of Web traffic crosses the Net, and many attacks rely on the HTTP protocol as their transmission medium.

To configure your Snort system so that it normalizes Web traffic, you need to put a few lines in your snort.conf configuration file that look something like the following:

preprocessor http_inspect: global iis_unicode_map unicode.map 1252
preprocessor http_inspect_server: server default profile all ports { 80 }
preprocessor http_inspect_server: server 172.16.1.1 profile all ports { 80 }

The first configuration line, which contains the keyword http_inspect, is for setting up the global behavior of the preprocessor, whereas the other lines with http_inspect_server in them are for specializing the configuration for individual machines. You need to include the iis_unicode_map keyword and its argument. Without it, Snort complains and refuses to start.

The options that follow the http_inspect_server keywords include the

• Ports on which the preprocessor should operate (between the {}s).

• A series of operations that should be used when normalizing the traffic.

The options described in the next sections are some of the most useful http_inspect options. They show how HTTP preprocessing really works.

iis_unicode <yes|no>

Characters in English and Western-European languages are normally represented using the ASCII encoding system. ASCII doesn’t cut the mustard for languages that require extended character sets, so a larger encoding system called Unicode, or UTF-8, is used. By using the iis_unicode keyword to the http_inspect_server preprocessor, all non-ASCII characters are converted back into ASCII for comparison.

 Tip  The iis_unicode option is important because it alerts on attempts to hide a URL by using different encoding techniques.

double_encode <yes|no>

Double encoding is often used to preserve % in URI encoding. The percent sign is most frequently used as an escape character within a URL, giving uses the full spectrum of additional characters as part of their input. Because hackers may try to disguise their attempts using a way to obfuscate their traffic with extra %’s, this option helps trap and contain them.

iis_backslash <yes|no>

Windows and DOS systems use the back-slash character \, much like Unix systems use the / character. Web browsers, Web servers, and URLs follow the lead of the Unix world and use the forward slash as part of an encoded URL. Microsoft’s IIS Web server can treat both identically, yet Snort only matches on exact sequences of bytes. The iis_backslash parameter can catch odd sequences of slashes as part of an attacker’s behavior.

apache_whitespace

The apache_whitespace parameter helps when a muck of whitespace characters try to sneak around the detection engine. The apache_whitespace option automatically converts all tabs in a URL into spaces, before the detection engine compares the data in the URL against the Snort rules.

telnet_decode: a preprocessor for telnet sessions

In much the same way that http traffic can be preprocessed before being turned over to the detection engine for adjudication, telnet remote interactive command-line sessions can be sent through a preprocessor.

 Technical Stuff  The two peers using the telnet protocol do their housekeeping via an inline protocol that can often trip-up the Snort detection engine. The inline negotiation channel involves such flow control parameters as discussing features the client and server support and what kind of terminal should be emulated. This noise is useless for the detection engine. By using the telnet_decode preprocessor, telnet protocol chatter can be filtered out before being analyzed for security rule matches.

The following is an example of how to configure telnet_decode in your snort.conf configuration file:

preprocessor telnet_decode: 23

The trailing 23 is the port number that the preprocessor should restrict its activity to. Telnet normally runs on port 23, but if you run your telnet daemon on a different port, then change this number to that port.

 Tip  You can specify multiple ports by adding to the end of the config line, separated by spaces.

rpc_decode: a preprocessor for RPC connections

Applications that use Remote Procedure Calls (RPC connections) use a different connection method than most common network services. Servers using RPC don’t listen on published well-known ports, such as TCP port 22 for telnet or TCP port 80 for HTTP. Servers using RPC bind to a random, unreserved port and publish that they’re available to another application (the local portmapper). You may already be familiar with portmapper if you’ve seen what application is listening on a TCP and UDP ports 111 on a standard Unix computer.

 Tip  How can you snort when you don’t know where to put your nose? The answer is to decode the traffic on port 111, which is the traffic cop port. It isn’t a perfect answer, but it helps get the job done.

To configure the rpc_decode preprocessor, insert the following line into your snort.conf configuration file:

preprocessor rpc_decode: 111 32711

The numbers that follow the colon are ports to regard for decoding. Port 32711 is a port that Sun’s Solaris operating system uses for portmapper in addition to port 111.

Deciding what’s normal and what’s not

A big advantage of using a preprocessor plug-in is the detection of anomalous protocol behavior. With anomaly detection preprocessors, Snort can be extended to detect attacks using more advanced techniques than simple pattern or port matching.

Some of the best examples of this advanced analysis are well demonstrated with the portscan and bo preprocessors.

portscan, portscan2, conversation

Portscanning is a technique that hackers use to “get the lay of the land” of a network that they intend to target. If we compare digital attacks to real-world warfare ones, portscanning is analogous to sending a surveillance and reconnaissance team to catalog the buildings, access portals, and armed guards that protect an installation.

Typical Snort rules can’t see whether an individual packet is part of a probing attack. The portscan, portscan2, and conversation preprocessors operate on both

• Individual packets

• Groups of packets

What makes a portscan a portscan? Together, several factors paint a pretty good picture of the prober’s intent:

• The time over which the packets were sent.

• The number of destination ports the packets were directed toward.

• The number of destination hosts addressed by the packets.

Snort’s portscan preprocessor operates by watching for a specified number of packets, sent within a certain timeframe, directed at any of the hosts on our network. For example, portscan produces an alert if it’s programmed with a threshold that 100 different ports were addressed within 5 minutes all from the same IP address source.

The portscan preprocessor is configured in the Snort configuration file with a line like this:

preprocessor portscan: 172.16.100.0/24 10 30 /var/log/snort/portscan.log

For the specifics on those options, check the configuration flags, which conform to this setting:

preprocessor portscan: <network> <# of ports> <time period> <logfile>

The example shows that this instance of the portscan preprocessor is set up to produce alerts whenever someone is probing the 172.16.100.0 Class C network with at least 10 different ports within 30 seconds.

 Tip  A configuration line in your snort.conf file can tell the system to ignore certain networks. This line is handy when you do port scans of your own network for topology mapping purposes. To configure portscan-ignorehosts, edit your snort.conf file and input a line using the following syntax:

preprocessor portscan-ignorehosts: network/netmask

To ignore networks 192.168.4.0/24 and the host 10.10.10.66, add the following line to snort.conf:

preprocessor portscan-ignorehosts: 192.168.4.0/24 10.10.10.66/32

Sneaky stealthy snoops: advanced portscan techniques

The portscan preprocessor has an interesting advanced behavior that helps you detect obvious rogues who are trying to paint a picture of what you have online. Often hackers use specialized tools to try mapping a network by slipping “broken” or “invalid” packets past Snort. Many of those packets should never be seen within the context of normal network communications, and others shouldn’t exist at all. Snort can catch these stealth-scanning methods by using the portscan preprocessor’s ability to detect these manufactured fake packets. Table 9-1 summarizes the most common stealth portscanning techniques.

Table 9-1: Stealth Techniques Detected by Snort

Packet Type	Description
FIN	Only the FIN flag in the packet header is on.
NULL	No flags are on (which should never happen).
SYN/FIN	Only the SYN and FIN flags are present.
XMAS	Only the FIN, URG, and PSH flags have been set.

Back Orifice (bo)

 Technical Stuff  In the late 1990s, an underground hacking group named The Cult of the Dead Cow (CDC) created a remote administration application called Back Orifice (a play on Microsoft’s Back Office program suite) to control Windows operating system computers from afar. While Back Orifice can be used for legitimate system administration, it’s also a favorite backdoor program for intruders. All you must do is trick someone into installing the program, which isn’t hard because it’s small and easy to package with other programs.

 Tip  The bo preprocessor focuses on traffic associated with the Back Orifice tool, which is an incredibly strong indication of a hacker in your midst.

The bo preprocessor can detect the first few moments of a BO connection attempt across the network and alert on it appropriately. Configuring the bo preprocessor is straightforward. Simply edit your snort.conf file and add the following:

preprocessor bo

Experimental preprocessors

A handful of other preprocessors — some developed by groups outside of the Snort development organization — are out there. These experimental and third-party can

• Handle ARP spoofing. (ARP stands for Address Resolution Protocol and deals with how computers’ IP addresses are found on a local Ethernet network.)

• Look aggressively for unknown exploit code that hasn’t been identified by a signature yet.

• Analyze the other protocols found in a standard internal network.