Planet Suricata

• Repeated inspection of same app state, transactions included, and as a side effect of this we had the pattern matching engine carrying out redundant runs on already matched buffers.

• FPs resulting from re-inspection of state.

• FNs.

• FPs from cross transaction matching.  For example the following sig would FP with the old engine for the below scenario.  Assume we have a flow with 2 requests -

        alert http any any -> any any (content:"one"; http_uri; content:"two"; http_uri; sid:1;)



The corresponding commits being -

commit b0f014124dbf44829ba04ed9d090ff268f7cb0ae
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri May 3 20:34:58 2013 +0530

    Transaction engine redesigned.....

commit e71de3f98f713fd4fe6cbccf42c51e59b0fca848
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri May 3 10:03:48 2013 +0530

    Track transaction progress separately......

commit 6ebd360c225ccffab0ec65099e4f0b4882945b25
Author: Anoop Saldanha <anoopsaldanha@gmail.com>
Date:   Fri Apr 12 13:18:17 2013 +0530

    hsbd mpm and packet mpm share same mpm ctx id.....



----------------------------------------
mpm - old engine : new engine
----------------------------------------
uri - 179k : 13k
http client body - 222 : 173
http header toserver - 179k : 13k
http header toclient - 174k : 117k
http method - 143k : 10.5k
http cookie - 10.5k : 6.5k
http raw uri - 143k : 10.5k


-Default yaml-

Old engine - 40.5 seconds
New engine - 33.5 seconds
Performance increase  - 17.28%


-Modified yaml-

Old engine - 70.5
New engine - 51.5
Performance increase - 27%

This morning I added flowint lua functions for incrementing and decrementing flowints. From the commit:

Add flowint lua functions for incrementing and decrementing flowints.

First use creates the var and inits to 0. So a call:

    a = ScFlowintIncr(0)

Results in a == 1.

If the var reached UINT_MAX (2^32), it’s not further incremented. If the
var reaches 0 it’s not decremented further.

Calling ScFlowintDecr on a uninitialized var will init it to 0.

Example script:

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

This script matches the 23rd time it’s invoked on a flow.

Compared to yesterday’s flowint script and the earlier flowvar based counting script, this performs better:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2434188332   59.71  82249    795      711777      29595.35    7683.20     29809.22   
  2        2            1        0        1015328580   24.91  82249    795      154398      12344.57    3768.66     12428.27   
  3        3            1        0        626858067    15.38  82249    795      160731      7621.47     3439.91     7662.28    

The rules:

alert http any any -> any any (msg:"LUAJIT HTTP flowvar match"; luajit:lua_flowvar_cnt.lua; flow:to_server; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint match"; luajit:lua_flowint_cnt.lua; flow:to_server; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint incr match"; luajit:lua_flowint_incr_cnt.lua; flow:to_server; sid:3;)

Please comment, discuss, review etc on the oisf-devel list.

A few days ago I wrote about my Emerging Threats sponsored work to support flowvars from Lua scripts in Suricata.

Today, I updated that support. Flowvar ‘sets’ are now real time. This was needed to fix some issues where a script was invoked multiple times in single rule, which can happen with some buffers, like HTTP headers.

Also, I implemented flowint support. Flowints in Suricata are integers stored in the flow context.

Example script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowint"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowintGet(0);
    if a then
        ScFlowintSet(0, a + 1)
    else
        ScFlowintSet(0, 1)
    end 
        
    a = ScFlowintGet(0);
    if a == 23 then
        return 1
    end 
    
    return 0
end 

return 0

It does the same thing as this flowvar script:

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowvarGet(0);
    if a then
        a = tostring(tonumber(a)+1)
        ScFlowvarSet(0, a, #a)
    else
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end 
    
    if tonumber(a) == 23 then
        return 1
    end
    
    return 0
end

return 0

Only, at about half the cost:

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2392221879   70.56  82249    795      834993      29085.12    6964.14     29301.02   
  2        2            1        0        998297994    29.44  82249    795      483810      12137.51    4019.44     12216.74   

Funded by Emerging Threats, I’ve been working on giving the lua scripts access to flowvars.

Currently only “flowvars” are done, “flowints” will be next. Please review the code at:
https://github.com/inliniac/suricata/tree/dev-lua-flowvar

Pcre based flowvar capturing is done in a post-match fashion. If the rule containing the “capture” matches, the var is stored in the flow.

For lua scripting, this wasn’t what the rule writers wanted. In this case, the flowvars are stored in the flow regardless of a rule match.

The way a script can start using flowvars is by first registering which one it needs access to:

function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

More than one can be registered, e.g.:

    needs["flowvar"] = {"cnt", "somevar", "anothervar" }

The maximum is 15 per script. The order of the vars matters. As Suricata uses id’s internally, to use the vars you have to use id’s as well. The first registered var has id 0, 2nd 1 and so on:

function match(args)
    a = ScFlowvarGet(0);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(0, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end

You can also use a var:

function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"blah", "cnt"}
    return needs
end

local var_cnt = 1

function match(args)
    a = ScFlowvarGet(var_cnt);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(var_cnt, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(var_cnt, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end

Flowvars are set at the end of the rule’s inspection, so after the script has run.

When multiple stores are done from the script and/or pcre, the last match will win. So if order matters, rule priority can be used to control inspection order.

Thoughts, comments, and code review highly welcomed at the oisf-devel list.

When processing the TCP 3 way handshake (3whs), Suricata’s TCP stream engine will closely follow the setup of a TCP connection to make sure the rest of the session can be tracked and reassembled properly. Retransmissions of SYN/ACKs are silently accepted, unless they are different somehow. If the SEQ or ACK values are different they are considered wrong and events are set. The stream events rules will match on this.

I ran into some cases where not the initial SYN/ACK was used by the client, but instead a later one. Suricata however, had accepted the initial SYN/ACK. The result was that every packet from that point was rejected by the stream engine. A 67 packet pcap resulting in 64 stream events.

If people have the stream events enabled _and_ pay attention to them, a noisy session like this should certainly get their attention. However, many people disable the stream events, or choose to ignore them, so a better solution is necessary.

Analysis

In this case the curious thing is that the extra SYN/ACK(s) have different properties: the sequence number is different. As the SYN/ACKs sequence number is used as “initial sequence number” (ISN) in the “to client” direction, it’s crucial to track it correctly. Failing to do so, Suricata will loose track of the stream, causing reassembly to fail. This could lead to missed alerts.

Whats happening on the wire:

TCP SSN 1:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 101

TCP SSN 2:

-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 1001

It’s clear that in SSN 1 the client ACKs the first SYN/ACK while in SSN 2 the 2nd SYN/ACK is ACK’d. It’s likely that the first SYN/ACK was lost before it reached the client. Suricata accepts the first though, and rejects any others that are not the same.

Solution

The solution I’ve been working on is to delay judgement on the extra SYN/ACKs until Suricata sees the ACK that completes the 3whs. At that point Suricata knows what the client accepted, and which SYN/ACKs were either ignored, or never received.

Logic in pseudo code:

Normal SYN/ACK coming in:

    UpdateState(p);
    ssn->state = TCP_SYN_RECV;

Extra SYN/ACK packets:

    if (p != ssn) {
        QueueState(p);

On receiving the ACK that completes the 3whs:

    if (ssn->queue_len) {
        q = QueueFindState(p);
        if (q)
            UpdateState(q);
    }
    UpdateState(p);
    ssn->state = TCP_ESTABLISHED;

So when receiving the ACK, Suricata first searches for the proper SYN/ACK on the list. If it’s not found, the ACK will be processed normally, which means it’s checked against the original SYN/ACK. If Suricata did have a queued state, it will first apply it to the SSN. Then the ACK will be processed normally, so that is can complete the 3whs and move the state to ESTABLISHED.

Limitations

Queuing these states takes some memory, and for this reason there is a limit to the number each SSN will accept. This is configurable through a new stream option:

stream:
  max-synack-queued: 5

It defaults to 5. I’ve seen a few (valid) hits against a few terrabytes of traffic, so I think the default is reasonably safe. An event is being set if the limit is exceeded. It can be matched using a stream-event rule:

  alert tcp any any -> any any (msg:"SURICATA STREAM 3way handshake \
      excessive different SYN/ACKs"; stream-event:3whs_synack_flood; \
      sid:2210055; rev:1;)

Performance

This functionality doesn’t affect the regular “fast path” except for a small check to see if we have queued states. However, if the queue list is being used Suricata enters a slow path. Currently this involves an memory allocation per stored queue. It may be interesting to consider using pools here, although a single global pool might be ineffecient. In such a case a lock would have to be used and this might lead to contention, especially in a case where Suricata would be flooded. Per thread pools (519, 520, 521) may be best here.

IPS mode

SYN/ACKs that exceed the limit are dropped if stream.inline is enabled as is the case with all packets that are considered to be bad in some way.

Code

The code is now part of the git master through commit 4c6463f3784f533a07679589dab713096137a439. Feedback welcome through our oisf-devel list.

Not usual setup can lead to surprise

The 5th of December 2012, I’ve setup suricata in AF_PACKET IPS mode between a WiFi interface and an Ethernet interface. The result was surprising as it was leading to a crash after some time:

The issue was linked with the defrag option of AF_PACKEt fanout. I’ve proposed a patch the 7th Dec 2012 and after a discussion with David Miller and Johannes Berg, Johannes has proposed a better patch which was included in official tree. So the problem is fixed for kernel superior or equal to 3.7.

Affected kernel

Here’s the list of affected kernel:

• All kernel prior to 3.2.36
• All 3.3.x kernel
• All 3.4.x kernel prior to 3.4.25
• All 3.5.x kernel prior to 3.5.7.3
• All 3.6.x kernel prior to 3.6.11

Workaround in Suricata

If you can’t update to a not affected kernel, you can set defrag to no in af-packet configuration to avoid the issue:

af-packet:
  - interface: wlan0
    # In some fragmentation case, the hash can not be computed. If "defrag" is set
    # to yes, the kernel will do the needed defragmentation before sending the packets.
    defrag: no

Suricata 1.4.1 was recently released:
http://suricata-ids.org/2013/03/08/suricata-1-4-1-released/

I've packaged Suricata 1.4.1 and it has been tested by the following (thanks!):
Eric Ooi
David Zawdie

The new package is now available in our stable repo. You can initiate the upgrade process using the graphical Update Manager or using the following one-liner:

sudo apt-get update && sudo apt-get dist-upgrade

Install Process

The Suricata update will do the following:

• install some new dependencies (libluajit and libjansson)
• back up each of your existing suricata.yaml file(s) to suricata.yaml.bak
• update Suricata to 1.4.1

If you're running Suricata in production, then you'll need to do the following:

• apply your local customizations to the new suricata.yaml
• restart Suricata as follows:

sudo nsm_sensor_ps-restart --only-snort-alert

Upgrade Process

sudo apt-get update && sudo apt-get dist-upgrade

suricata -V

Update suricata.yaml file(s) and then run "sudo nsm_sensor_ps-restart --only-snort-alert"

Feedback
If you have any questions or problems, please use our mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

The Windows MSI installer of the Suricata 1.4.1 release is now available.

Download it here: Suricata1.4.1-1-32bit.msi

After downloading, double click the file to launch the installer. Please note that the installer is not yet signed.

If you have a previous version installed, please remove that first.

We have updated the official Ubuntu PPA to Suricata 1.4.1. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

Suricata and Netfilter can be better friend as they are doing some common work like decoding packet and maintaining flow table.

In IPS mode, Suricata is receiving raw packet from libnetfilter_queue. It has to made the parsing of this packet but this kind of thing has also been done by kernel. So it should be possible to avoid to duplicate the work.

In fact Netfilter work is limited as ipheaders srtucture are used. Patrik McHardy proposed that Netfilter offset but this is not the most costly part.

The flow discussion was more interesting because conntrack is really doing a similar work as the one done by Suricata. Using the CT extension of libnetfilter_queue, Suricata will be able to get access to all the CT information. And at a first glance, it seems it contains almost all information needed. So it should be possible to remove the flow engine from suricata. The garbage operation would not be necessary as Suricata will get information via NFCT destroy event.

Jozsef Kadlecsik proposed to use Tproxy to redirect flow and provide a “socket” stream instead of individual packet to Suricata. This would change Suricata a lot but could provide a interesting alternative mode.

The OISF development team is proud to announce Suricata 1.4.1. This is a major update over the 1.4 release, adding some exiting features, many improvements and fixing some important bugs.

Get the new release here: suricata-1.4.1.tar.gz

The most interesting new feature is the GeoIP support. Great contribution by Ignacio Sanchez. It adds “geoip” rule keyword that allows you to match on source of destination of a packet per country.

New features

• GeoIP keyword, allowing matching on Maxmind’s database, contributed by Ignacio Sanchez (#559)
• Introduce http_host and http_raw_host keywords (#733, #743)
• Add python module for interacting with unix socket (#767)
• Add new unix socket commands: fetching config, counters, basic runtime info (#764, #765)

Improvements

• Big Napatech support update by Matt Keeler
• Configurable sensor id in unified2 output, contributed by Jake Gionet (#667)
• FreeBSD IPFW fixes by Nikolay Denev
• Add “default” interface setting to capture configuration in yaml (#679)
• Make sure “snaplen” can be set by the user (#680)
• Improve HTTP URI query string normalization (#739)
• Improved error reporting in MD5 loading (#693)
• Improve reference.config parser error reporting (#737)
• Improve build info output to include all configure options (#738)

Fixes

• Segfault in TLS parsing reported by Charles Smutz (#725)
• Fix crash in teredo decoding, reported by Rmkml (#736)
• fixed UDPv4 packets without checksum being detected as invalid (#760)
• fixed DCE/SMB parsers getting confused in some fragmented cases (#764)
• parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#697)
• FN: IP-only rule ip_proto not matching for some protocols (#689)
• Fix build failure with other libhtp installs (#688)
• Fix malformed yaml loading leading to a crash (#694)
• Various Mac OS X fixes (#700, #701, #703)
• Fix for autotools on Mac OS X by Jason Ish (#704)
• Fix AF_PACKET under high load not updating stats (#706)

Special thanks

• Ignacio Sanchez
• Matt Keeler — nPulse
• Jake Gionet
• Nikolay Denev
• Jason Ish — Endace
• Jamie Strandboge
• Charles Smutz
• Rmkml

Known issues & missing features

As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is proud to announce Suricata 1.4.1. This is a major update over the 1.4 release, adding some exiting features, many improvements and fixing some important bugs.

Get the new release here: suricata-1.4.1.tar.gz

The most interesting new feature is the GeoIP support. Great contribution by Ignacio Sanchez. It adds “geoip” rule keyword that allows you to match on source of destination of a packet per country.

New features

• GeoIP keyword, allowing matching on Maxmind’s database, contributed by Ignacio Sanchez (#559)
• Introduce http_host and http_raw_host keywords (#733, #743)
• Add python module for interacting with unix socket (#767)
• Add new unix socket commands: fetching config, counters, basic runtime info (#764, #765)

Improvements

• Big Napatech support update by Matt Keeler
• Configurable sensor id in unified2 output, contributed by Jake Gionet (#667)
• FreeBSD IPFW fixes by Nikolay Denev
• Add “default” interface setting to capture configuration in yaml (#679)
• Make sure “snaplen” can be set by the user (#680)
• Improve HTTP URI query string normalization (#739)
• Improved error reporting in MD5 loading (#693)
• Improve reference.config parser error reporting (#737)
• Improve build info output to include all configure options (#738)

Fixes

• Segfault in TLS parsing reported by Charles Smutz (#725)
• Fix crash in teredo decoding, reported by Rmkml (#736)
• fixed UDPv4 packets without checksum being detected as invalid (#760)
• fixed DCE/SMB parsers getting confused in some fragmented cases (#764)
• parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#697)
• FN: IP-only rule ip_proto not matching for some protocols (#689)
• Fix build failure with other libhtp installs (#688)
• Fix malformed yaml loading leading to a crash (#694)
• Various Mac OS X fixes (#700, #701, #703)
• Fix for autotools on Mac OS X by Jason Ish (#704)
• Fix AF_PACKET under high load not updating stats (#706)

Special thanks

• Ignacio Sanchez
• Matt Keeler — nPulse
• Jake Gionet
• Nikolay Denev
• Jason Ish — Endace
• Jamie Strandboge
• Charles Smutz
• Rmkml

Known issues & missing features

As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is pleased to announce Suricata 1.3.6. This the last maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.6.tar.gz

Fixes

• fix decoder event rules not checked in all cases (#671)
• checksum detection for icmpv6 was fixed (#673)
• crash in HTTP server body inspection code fixed (#675)
• fixed a icmpv6 payload bug (#676)
• IP-only rule ip_proto not matching for some protocols was addressed (#690)
• fixed malformed yaml crashing suricata (#702)
• parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#717)
• crash in tls parser was fixed (#759)
• fixed UDPv4 packets without checksum being detected as invalid (#762)
• fixed DCE/SMB parsers getting confused in some fragmented cases (#763)

Special thanks

• Jamie Strandboge

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is pleased to announce Suricata 1.3.6. This the last maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.6.tar.gz

Fixes

• fix decoder event rules not checked in all cases (#671)
• checksum detection for icmpv6 was fixed (#673)
• crash in HTTP server body inspection code fixed (#675)
• fixed a icmpv6 payload bug (#676)
• IP-only rule ip_proto not matching for some protocols was addressed (#690)
• fixed malformed yaml crashing suricata (#702)
• parsing ipv6 address/subnet parsing in thresholding was fixed by Jamie Strandboge (#717)
• crash in tls parser was fixed (#759)
• fixed UDPv4 packets without checksum being detected as invalid (#762)
• fixed DCE/SMB parsers getting confused in some fragmented cases (#763)

Special thanks

• Jamie Strandboge

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

We are very pleased to announce that nPulse has moved to Platinum Consortium Member status of the Open Information Security Foundation (OISF).

nPulse Technologies, Inc. takes the pulse of the world’s fastest networks. For customers with extremely big pipes of 10Gbps or more, who run intelligence-driven security and network operations, nPulse solutions are open, standards-based collection platforms that integrate ultrafast flow and packet capture probes with big data analytics. Unlike traditional packet capture solutions which are proprietary, expensive, and unable to scale to today’s network core speeds, nPulse platforms capture 100% of network traffic at 20 Gbps and utilize a Big Data analytics approach to significantly reduce the time, effort, and resources required to produce actionable intelligence. For more information, visit www.npulsetech.com.

I've updated our NSM scripts to resolve the following issues:

Issue 292: Need cronjob to reload syslog-ng at midnight
Issue 295: Increase sleep value in /etc/init/securityonion.conf
Issue 296: Run snort as non-root user
Issue 297: Run snort/suricata with unique PF_RING cluster-id per interface

Thanks to the following for testing this update!

Matt Gregory
GabrielS
Heine Lysemose

Installation
The new NSM scripts package is now available in our stable repo. You can initiate the update process using the graphical Update Manager or with the following one-liner:

sudo apt-get update && sudo apt-get dist-upgrade

Reminder about MySQL Updates
As a reminder, if you are prompted to install MySQL updates, please see the following for the recommended procedure for updating MySQL:
http://code.google.com/p/security-onion/wiki/MySQLUpdates

Feedback
If you have any questions or problems, please join our mailing list and ask away!
https://code.google.com/p/security-onion/wiki/MailingLists

We are very pleased to announce that nPulse has moved to Platinum Consortium Member status of the Open Information Security Foundation (OISF).

nPulse Technologies, Inc. takes the pulse of the world’s fastest networks. For customers with extremely big pipes of 10Gbps or more, who run intelligence-driven security and network operations, nPulse solutions are open, standards-based collection platforms that integrate ultrafast flow and packet capture probes with big data analytics. Unlike traditional packet capture solutions which are proprietary, expensive, and unable to scale to today's network core speeds, nPulse platforms capture 100% of network traffic at 20 Gbps and utilize a Big Data analytics approach to significantly reduce the time, effort, and resources required to produce actionable intelligence.  For more information, visit www.npulsetech.com.

 

We are very pleased to welcome Tilera Corporation as a Gold level Consortium member of the Open Information Security Foundation (OISF)! Through this membership, Tilera will continue to focus on achieving unparalleled Suricata performance on the TILE-Gx processor family. The TILE-Gx processor family delivers industry leading performance and power efficiency (performance/watt), while providing ease-of-use with standard Linux programming.

As the leader in 64-bit manycore general purpose processors, the company is already hard at work contributing to the Consortium and has delivered the highest performance, highest density Suricata solution in the market – seeing about 40 Gbps throughput in a 1U platform. The Suricata implementation on TILE-Gx processors supports all the features of Suricata including both the IDS and IPS modes of operation.

“Tilera’s involvement with the OISF and Suricata is significant validation of Suricata as an Engine, and threading as the way forward for the industry as a whole,” said Matt Jonkman, president, OISF. “The performance benefits that Tilera has already demonstrated with Suricata and the TILE-GX processor family is thoroughly impressive and is just a taste of what is to come.”

The industry-leading performance was achieved on Tilera’s TILExtreme-Gx high density platform that packs 144 cores with four TILE-Gx36 processors in a compact 1U rack mountable device. The standard TILExtreme-Gx platform provides up to 160Gbps of Ethernet I/O and is ideal for a variety of compute and I/O intensive tasks such as Network Security (IDS/IPS, DPI, DLP), Network Monitoring, Data Forensics and Big Data processing. It is actively being deployed by several Tilera customers. Additionally, based on the performance and I/O requirements, Tilera customers have the ability to scale up or down by choosing from range of platforms ranging from half-length PCIe cards to the high density 1U chassis.

Tilera will unveil a new platform that doubles the capacity in the first quarter of 2013. Support for the TILE-Gx will also be added for the open source version of Suricata.

There will be much more exciting news to come from Tilera and the OISF in 2013. If you want to learn more about Tilera and its solutions, contact  Satish Ganesan, Director of Marketing, Networking Solutions, for Tilera. In the meantime, stay tuned to this space for the latest updates!

The Open Information Security Foundation (OISF) is a non-profit foundation organized to build a next generation IDS/IPS engine Suricata.  The OISF has formed a multi-national group of the leading software developers in the security industry.  In addition to developers and a consortium consisting of leading cyber security companies, OISF has engaged the open source security community to identify current and future IDS/IPS needs and desires.

Fedora maintainer Steve Grubb updated the Suricata package in Fedora 17 (and the upcoming 18) to 1.3.5.

If you are running Fedora, updating is as simple as:
yum update


Installing is as simple as:
yum install suricata

The Fedora package has IPS mode through NFQUEUE enabled.

We are very pleased to welcome Tilera Corporation as a Gold level Consortium member of the Open Information Security Foundation (OISF)! Through this membership, Tilera will continue to focus on achieving unparalleled Suricata performance on the TILE-Gx processor family. The TILE-Gx processor family delivers industry leading performance and power efficiency (performance/watt), while providing ease-of-use with standard Linux programming.  

 
As the leader in 64-bit manycore general purpose processors, the company is already hard at work contributing to the Consortium and has delivered the highest performance, highest density Suricata solution in the market – seeing about 40 Gbps throughput in a 1U platform. The Suricata implementation on TILE-Gx processors supports all the features of Suricata including both the IDS and IPS modes of operation.
 
"Tilera’s involvement with the OISF and Suricata is significant validation of Suricata as an Engine, and threading as the way forward for the industry as a whole," said Matt Jonkman, president, OISF. “The performance benefits that Tilera has already demonstrated with Suricata and the TILE-GX processor family is thoroughly impressive and is just a taste of what is to come.”
 
The industry-leading performance was achieved on Tilera’s TILExtreme-Gx high density platform that packs 144 cores with four TILE-Gx36 processors in a compact 1U rack mountable device. The standard TILExtreme-Gx platform provides up to 160Gbps of Ethernet I/O and is ideal for a variety of compute and I/O intensive tasks such as Network Security (IDS/IPS, DPI, DLP), Network Monitoring, Data Forensics and Big Data processing. It is actively being deployed by several Tilera customers. Additionally, based on the performance and I/O requirements, Tilera customers have the ability to scale up or down by choosing from range of platforms ranging from half-length PCIe cards to the high density 1U chassis. 
 
Tilera will unveil a new platform that doubles the capacity in the first quarter of 2013. Support for the TILE-Gx will also be added for the open source version of Suricata.
 
There will be much more exciting news to come from Tilera and the OISF in 2013. If you want to learn more about Tilera and its solutions, contact Satish Ganesan, Director of Marketing, Networking Solutions, for Tilera. In the meantime, stay tuned to this space for the latest updates!

The Open Information Security Foundation (OISF) is a non-profit foundation organized to build a next generation IDS/IPS engine Suricata.  The OISF has formed a multi-national group of the leading software developers in the security industry.  In addition to developers and a consortium consisting of leading cyber security companies, OISF has engaged the open source security community to identify current and future IDS/IPS needs and desires. 
 

The Windows MSI installer of the Suricata 1.4 release is now available.

Download it here: Suricata1.4-1-32bit.msi

After downloading, double click the file to launch the installer. Please note that the installer is not yet signed.

If you have a previous version installed, please remove that first.

The next Ubuntu release currently in development, 13.04, just received an update to Suricata 1.4.

If you are running Ubuntu 13.04, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Ubuntu package has IPS mode through NFQUEUE enabled.

The OISF also provides a PPA for this and other Ubuntu versions: instructions can be found here.

We have updated the official Ubuntu PPA to Suricata 1.4. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

Debian maintainer Pierre Chifflier updated the Suricata package in Debian Sid to 1.4.

If you are running Debian Sid, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Debian package has IPS mode through NFQUEUE enabled.

So with 1.4 out the door we have a new stable. However, we’re keeping 1.3 around for a few more months to give everyone the chance to plan updating to 1.4. Of course, we think 1.4 is a lot better than anything we released before, so we do recommend updating as soon as you can.

Continued support for 1.3 means we’ll do more releases to fix critical issues. We’ll probably include trivial fixes of smaller problems. When talking about critical issues I mean crash cases mostly. Anything else will be fixed only in 1.4 and up.

Moving forward, we’ll open our dev branch on 1.5 (or shall we jump to 2.0 already?) after the 1.4.1 or 1.4.2 release, which I hope to be doing sometime in mid-January. But we’ll see how things go.

If you have patches you need to get included, please open a pull request on github. Also, I think it’s a good idea to announce those requests on the oisf-devel list. So everyone on the list is notified and can help review and test.

About 5 months after 1.3 came out we’ve released 1.4, and we’ve been quite busy. Eric Leblond’s post here has all the stats and graphs. There are three big new features: unix socket, ip reputation and luajit. For each of these the same is true: it’s usesable now, but it’s the potential that we’re most excited about. Over the next months we’ll be extending each of those to be even more useful. We’re very much interested in ideas and feedback.

Performance obviously matters to many in the IDS world, and here too we have improved Suricata quite a bit again. We now have Suricata 1.4 running on a ISP 10gbit/s network on commodity hardware with a large ET ruleset. Of course, YMMV, but we’re definitely making a lot of progress here.

Sometimes the little things matter a lot as well. A minor new feature is that live “drop” stats are the the stats.log now:

capture.kernel_packets    | AFPacketem21              | 13640581
capture.kernel_drops      | AFPacketem21              | 442864
capture.kernel_packets    | AFPacketem22              | 7073228
capture.kernel_drops      | AFPacketem22              | 9449
capture.kernel_packets    | AFPacketem23              | 10528970
capture.kernel_drops      | AFPacketem23              | 148281
capture.kernel_packets    | AFPacketem24              | 7212584
capture.kernel_drops      | AFPacketem24              | 12643
capture.kernel_packets    | AFPacketem25              | 9763439
capture.kernel_drops      | AFPacketem25              | 17874
capture.kernel_packets    | AFPacketem26              | 10464106
capture.kernel_drops      | AFPacketem26              | 20378
capture.kernel_packets    | AFPacketem27              | 8869182
capture.kernel_drops      | AFPacketem27              | 18336
capture.kernel_packets    | AFPacketem28              | 7925045
capture.kernel_drops      | AFPacketem28              | 258168

This is supported for AF_PACKET, PF_RING and libpcap.

Last August we’ve added Suricata to github to make it easier to participate. Also, the code review tools associated with the pull requests are very useful. Github has been an unexpected success for us. At the time of writing there are 24 forks of Suricata on it, I’ve processed about 250 pull requests. The patches that have been submitted range from small fixes to full blown features, and more are on the way. I’m very grateful for these contributions and everyone’s patience with me.

Now that 1.4 is out, we’ll be taking it slow over the holidays. The team has been working like crazy, and everyone deserves a break. So the next weeks we’ll focus on further consolidation, fixing bugs that no doubt will pop up. Other than that, things will be slow. After the holidays we’ll start planning for the next milestone. Again, your ideas and contributions are very welcome!

A huge work

Suricata 1.4 has been released December 13th 2012 and it has been a huge work. The number of modifications is just impressing:

390 files changed, 25299 insertions(+), 11982 deletions(-)

The following video is using gource to display the evolution of Suricata IDS/IPS source code between version 1.3 and version 1.4. It only displays the modified files and do not show the files existing at start.

A collaborative work

A total of 11 different authors have participated to this release. The following graph generated by gitstats shows the number of lines of code by author:

Some words about activity

The activity shows that most of the work is done during week day but there is some work done on sunday: As shown the following graph, the activity as decreased during the stabilization process:

0	0	0	0	0	1	0	0	6	7	9	10	5	11	11	11	15	51	15	35	23	22	16	24	10	29	32	26	18	16	6	8
32	31	30	29	28	27	26	25	24	23	22	21	20	19	18	17	16	15	14	13	12	11	10	9	8	7	6	5	4	3	2	1

The OISF development team is proud to announce Suricata 1.4. This release is a major improvement over the previous releases with regard to performance, scalability and accuracy. Also, a number of great features have been added.

Get the new release here: suricata-1.4.tar.gz

The biggest new features of this release are the Unix Socket support, IP Reputation support and the addition of the Luajit keyword. Each of these new features are still in active development, and should be approached with some care.

The 1.4 release improves performance and scalability a lot. The IP Defrag engine was rewritten to scale better, various packet acquisition methods were improved and various parts of the detection engine were optimized further.

The configuration file has evolved but backward compatibility is provided. We thus encourage you to update your suricata configuration file. Upgrade guidance is provided here: Upgrading_Suricata_13_to_Suricata_14

New features

• Unix socket mode for batched processing of series of pcap (#571, #552) (experimental)
• Interaction with Suricata via uix socket (#571, #552) (experimental)
• IP Reputation: loading and matching (#647) (experimental)
• New keyword: "luajit" to inspect packet, payload and all HTTP buffers with a Lua script (#346) (experimental)
• Delayed detect initialization. Starts processing packets right away and loads detection engine in the background (#522)
• Support for pkt_data keyword was added (#423)
• Improved --list-keywords commandline option gives detailed info for supported keyword, including doc link (#435)
• User and group to run as can now be set in the config file
• Add stream event to match on overlaps with different data in stream reassembly (#603)
• Decoding of IPv4-in-IPv6, IPv6-in-IPv6 and Teredo tunnels (#462, #514, #480)
• Rules can be set to inspect only IPv4 or IPv6 (#494)
• Added ability to control per server HTTP parser settings in much more detail (#503)
• Make HTTP request and response body inspection sizes configurable per HTTP server config (#560)
• Filesize keyword for matching on sizes of files in HTTP (#489)
• Custom HTTP logging contributed by Ignacio Sanchez (#530)
• TLS certificate logging and fingerprint computation and keyword by Jean-Paul Roliers (#443)
• TLS certificate store to disk feature Jean-Paul Roliers (#444)
• AF_PACKET IPS support (#516)
• NFQ fail open support (#507)
• PCAP/AF_PACKET/PF_RING packet stats are now printed in stats.log (#561, #625)
• Support for Napatech cards through their 3rd generation driver was added by Matt Keeler from Npulse (#430, #619)
• Endace support improved
• New runmode for users of pcap wrappers (Myricom, PF_RING, others)

Improvements

• Add contrib directory to the dist (#567)
• Performance improvements to signatures with dsize option
• Improved rule analyzer: print fast_pattern along with the rule (#558)
• Fixes to stream engine reducing the number of events generated (#604)
• Stream.inline option new defaults to "auto", meaning enabled in IPS mode, disabled in IDS mode (#592)
• HTTP handling in OOM condition was greatly improved (#557)
• Filemagic keyword performance was improved (#585)
• Updated bundled libhtp to 0.2.11
• Build system improvements and cleanups
• Live reloads now supports HTTP rule updates better (#522)
• AF_PACKET performance improvements (#197, #415)
• Make defrag more configurable (#517, #528)
• Improve pool performance (#518)
• Improve file inspection keywords by adding a separate API (#531)
• Example threshold.config file provided (#302)

Changes since 1.4rc1

• Decoder event matching fixed (#672)
• Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#665)
• Add more events to IPv6 extension header anomolies (#678)
• Fix ICMPv6 payload and checksum calculation (#677, #674)
• Clean up flow timeout handling (#656)
• Fix a shutdown bug when using AF_PACKET under high load (#653)
• Fix TCP sessions being cleaned up to early (#652)

Credits

• Jason Ish -- Endace
• Ludovico Cavedon -- Lastline
• Last G
• Matt Keeler -- Npulse
• Chris Wakelin
• Will Metcalf
• Ivan Ristic
• Kyle Creyts
• Michael Hoffrath
• Rmkml
• Jean-Paul Roliers
• Ignacio Sanchez
• Michel Saborde
• Simon Moon
• Coverity

Known issues & missing features

As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is proud to announce Suricata 1.4. This release is a major improvement over the previous releases with regard to performance, scalability and accuracy. Also, a number of great features have been added.

Get the new release here: suricata-1.4.tar.gz

The biggest new features of this release are the Unix Socket support, IP Reputation support and the addition of the Luajit keyword. Each of these new features are still in active development, and should be approached with some care.

The 1.4 release improves performance and scalability a lot. The IP Defrag engine was rewritten to scale better, various packet acquisition methods were improved and various parts of the detection engine were optimized further.

The configuration file has evolved but backward compatibility is provided. We thus encourage you to update your suricata configuration file. Upgrade guidance is provided here: Upgrading_Suricata_13_to_Suricata_14

New features

• Unix socket mode for batched processing of series of pcap (#571, #552) (experimental)
• Interaction with Suricata via uix socket (#571, #552) (experimental)
• IP Reputation: loading and matching (#647) (experimental)
• New keyword: “luajit” to inspect packet, payload and all HTTP buffers with a Lua script (#346) (experimental)
• Delayed detect initialization. Starts processing packets right away and loads detection engine in the background (#522)
• Support for pkt_data keyword was added (#423)
• Improved –list-keywords commandline option gives detailed info for supported keyword, including doc link (#435)
• User and group to run as can now be set in the config file
• Add stream event to match on overlaps with different data in stream reassembly (#603)
• Decoding of IPv4-in-IPv6, IPv6-in-IPv6 and Teredo tunnels (#462, #514, #480)
• Rules can be set to inspect only IPv4 or IPv6 (#494)
• Added ability to control per server HTTP parser settings in much more detail (#503)
• Make HTTP request and response body inspection sizes configurable per HTTP server config (#560)
• Filesize keyword for matching on sizes of files in HTTP (#489)
• Custom HTTP logging contributed by Ignacio Sanchez (#530)
• TLS certificate logging and fingerprint computation and keyword by Jean-Paul Roliers (#443)
• TLS certificate store to disk feature Jean-Paul Roliers (#444)
• AF_PACKET IPS support (#516)
• NFQ fail open support (#507)
• PCAP/AF_PACKET/PF_RING packet stats are now printed in stats.log (#561, #625)
• Support for Napatech cards through their 3rd generation driver was added by Matt Keeler from Npulse (#430, #619)
• Endace support improved
• New runmode for users of pcap wrappers (Myricom, PF_RING, others)

Improvements

• Add contrib directory to the dist (#567)
• Performance improvements to signatures with dsize option
• Improved rule analyzer: print fast_pattern along with the rule (#558)
• Fixes to stream engine reducing the number of events generated (#604)
• Stream.inline option new defaults to “auto”, meaning enabled in IPS mode, disabled in IDS mode (#592)
• HTTP handling in OOM condition was greatly improved (#557)
• Filemagic keyword performance was improved (#585)
• Updated bundled libhtp to 0.2.11
• Build system improvements and cleanups
• Live reloads now supports HTTP rule updates better (#522)
• AF_PACKET performance improvements (#197, #415)
• Make defrag more configurable (#517, #528)
• Improve pool performance (#518)
• Improve file inspection keywords by adding a separate API (#531)
• Example threshold.config file provided (#302)

Changes since 1.4rc1

• Decoder event matching fixed (#672)
• Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#665)
• Add more events to IPv6 extension header anomolies (#678)
• Fix ICMPv6 payload and checksum calculation (#677, #674)
• Clean up flow timeout handling (#656)
• Fix a shutdown bug when using AF_PACKET under high load (#653)
• Fix TCP sessions being cleaned up to early (#652)

Credits

• Jason Ish — Endace
• Ludovico Cavedon — Lastline
• Last G
• Matt Keeler — Npulse
• Chris Wakelin
• Will Metcalf
• Ivan Ristic
• Kyle Creyts
• Michael Hoffrath
• Rmkml
• Jean-Paul Roliers
• Ignacio Sanchez
• Michel Saborde
• Simon Moon
• Coverity

Known issues & missing features

As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Lots of activity on the IPv6 front lately. There was a talk on a conference on bypassing IDS using IPv6 tricks. Also a new scan tool (Topera) claimed to scan a host while staying below the radar of an IDS was released. To start with the latter, even though Suricata doesn’t have a dedicated port scan detector, the tool’s traffic lights up like a Christmas tree. The trick it pulls is to pack a lot of duplicate DST OPTS extension headers in the IPv6 packets. These options are just fillers, the only options they use are the “pad” option. In Suricata we’ve had an event for duplicate DST OPTS headers since 1.3 and the padding only headers generate an event in 1.4. Both alerts will be very noisy, so calling this a stealth attack rather dubious.

The other thing was a talk on IPv6 evasions, where the author compared Snort and Suricata. Suricata didn’t do very well. Sadly the authors chose not to contact us. On closer inspection it turned out an old Suricata version was used. Which one wasn’t specified, but as they did mention using Security Onion, I’m assuming 1.2. In the 1.3 branch (current stable) we’ve fixed and improved IPv6 in a lot of areas. Nonetheless, while testing the various protocol tricks, we did find some bugs that are now fixed in the git masters for the 1.3 stable branch and the 1.4 development branch.

I think these developments serve as a reminder that staying current with your IDS software’s version is critical. For that reason it’s too bad that distro’s like Security Onion, Debian, Ubuntu all lag significantly. The reasons differ through. For the guys from Security Onion it’s mostly a time problem (so go help them if you can!) for Debian and Ubuntu it’s actually policy. For that reason we’re providing PPAs for Ubuntu and for Debian we’re working on getting Suricata into the “backports” repo. The only mainstream distro that does it right for us is Fedora. They just update to the latest stable as soon as it’s out.

Given the complexity of protocols like IPv6 and the new developments all over the board, I see no viable case for staying on older versions. I know it’s a hassle, but stay current. It’s important.

The Windows MSI installer of the Suricata 1.3.5 release is now available.

Download it here: Suricata1.3.5-1-32bit.msi

After downloading, double click the file to launch the installer. Please note that the installer is not yet signed.

If you have a previous version installed, please remove that first.

We have updated the official Ubuntu PPA to Suricata 1.3.5. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

Debian maintainer Pierre Chifflier updated the Suricata package in Debian Sid to 1.3.5.

If you are running Debian Sid, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Debian package has IPS mode through NFQUEUE enabled.

The OISF development team is pleased to announce Suricata 1.3.5. This a maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.5.tar.gz

Fixes

• Flow engine memory leak fixed by Ludovico Cavedon (#651)
• Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#664)
• Flow manager mutex used unintialized, fixed by Ludovico Cavedon (#654)
• Windows building in CYGWIN fixed (#630)

Credits

• Ludovico Cavedon -- Lastline

Known issues & missing features

There is talk about a possible IPv6 evasion, but since no details are available this isn't addressed yet. Due to the nature of the fixes above, we decided to release anyway. Once we get details on the evasion, we'll push out another update.

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is pleased to announce Suricata 1.3.5. This a maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.5.tar.gz

Fixes

• Flow engine memory leak fixed by Ludovico Cavedon (#651)
• Unified2 would overwrite files if file rotation happened within a second of file creation, leading to loss of events/alerts (#664)
• Flow manager mutex used unintialized, fixed by Ludovico Cavedon (#654)
• Windows building in CYGWIN fixed (#630)

Credits

• Ludovico Cavedon — Lastline

Known issues & missing features

There is talk about a possible IPv6 evasion, but since no details are available this isn’t addressed yet. Due to the nature of the fixes above, we decided to release anyway. Once we get details on the evasion, we’ll push out another update.

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Introduction

Kernel oops have been reported by some users running Suricata with AF_PACKET multiple thread capture activated. This is due to a bug I’ve introduced in AF_PACKET when fixing an other bug.

Which kernel not to use with Suricata in AF_PACKET mode

The following kernel version will surely crash if Suricata or any other program is used with AF_PACKET capture with multiple capture threads:

• Linux 3.2.30 to 3.2.33
• Linux 3.4.12 to 3.4.18
• Linux 3.5.5 to 3.5.7
• Linux 3.6.0 to 3.6.6

If only one capture thread is used there is no risk of crash. If you are running a vulnerable kernel, your configuration should looks like:

af-packet:
  - interface: eth0
    # Number of receive threads (>1 will crash with bad kernel)
    threads: 1

Some explanations

AF_PACKET capture is one of my favorite capture method on Linux with Suricata. It is mainline and it offers really good performance with latest kernel. For example, this is deployed on one of our box and run at 10Gbps speed on non-expensive hardware. This speed is achieved by using load-balanced capture. This feature allows to have multiple thread/process listening to the same interface. The load-balancing is made by the kernel. This feature has been developed by David Miller and is available since Linux 3.1.

In summer 2012, I’ve worked on adding AF_PACKET IPS mode and I’ve discovered a kernel bug which was causing a packet sending loop in the IPS code. I’ve proposed a fix af_packet: don’t emit packet on orig fanout group. The patch has reached mainline with Linux 3.6. As it was fixing a real problem it was propagated to most Linux stable branches. Some distributions, like Ubuntu, have also backported the patch to their official kernel.

But the patch was buggy and some Suricata users have reported kernel oops. I’ve fixed the bug af-packet: fix oops when socket is not present and the patch will be available in Linux 3.7. The kernel stable team has incorporated this patch in their subsequent releases so most stable branches but 3.5 don’t suffer anymore of this problem.

Note

Ubuntu Quantal has a patched kernel since at least 3.5.0-25.

I just made Suricata 1.4rc1 available with some pretty exciting features: unix socket mode and IP reputation.

Unix socket

First of all, Eric Leblond’s work on the Unix socket was merged. The unix socket work consists of two parts. The unix socket protocol implementation and a new runmode.

The protocol implementation is based on JSON messages over unix socket. Eric will be fully documenting it soon. Currently the commands are limited to shutting down and getting some basic stats. This part isn’t very exciting yet, but the groundwork for many future extensions has been laid.

The part that is exciting right now, is the unix socket runmode. That this does is start Suricata with all the rules and such, and then it waits for commands on the unix socket. Then the commands will be a pcap filename – log directory pair. This pcap will then be inspected against the rules and the logs go into the log directory supplied. As this can be easily scripted (a python script is provided), it’s a very fast way to test your pcap collections, as the overhead of starting and stopping is skipped.

This may initialy appeal mostly for those of you doing sandnetting and malware analysis, where tens of thousands of pcaps and automatically processed every hour or day, I think this could grow into a feature for a wider audience as well. For example, I could see use in Sguil or Snorby, or pretty much every event manager with full packet capture support, adding an option to scan a pcap associated with an event again. Maybe against _all_ rules, instead of the tuned set running on the live sensors. Maybe you can re-inspect old sessions against the current rules this way to find hits on attacks that were 0-days at the time, etc.

I think there could be many possibilities.

IP Reputation

A slightly more polished version of the code I discussed here is now available in this release. It’s one of those things where it will be very interesting to see how people will put it to use.

Matt Jonkman just wrote some of his ideas to the Emerging Threats mailing list: one of the ideas Matt wrote about is to amend weak rules with reputation data. So if you have a signature that is phrone to false positives, you probably disable it currently. But what if you combine it with reputation data? If the weak rule fires on a sketchy ip, it may be a more reliable alert.

We’ll see how this plays out.

1.4 final

We’re hoping that if nothing big happens, we can do a mid-December 1.4 final release. So please consider running this new release. It’s running very stable on quite a number of places, ISP networks, Lab networks, home networks, sandnetting networks, etc. But we need much more testing to find issues and/or gain confidence that we have found the most important issues. Thanks for helping out!

Disclaimer: this work was sponsored by Emerging Threats Pro.

One thing we’ve been talking about for many years at OISF is IP Reputation. The basic idea is that many organizations have information about specific IP-addresses. This information may be that a host is infected, acts as a spam relay or many other things. We’ve always thought it might be useful to apply this info to the IDS directly.

In the last weeks I’ve developed code to load IP reputation information into Suricata. This code is now part of the Suricata git master, so it’s available to all.

The work consisted of 3 main parts: data load, internal data structures and a rule keyword.

Data loading

The data I worked with was provided by Emerging Threats Pro. The data format is very simple. Two types of CSV files, one to define a mapping between category names and id’s and the other to define the scores for hosts in the categories.

The data formats are documented here: IP Reputation Format.

Internal Data Structures

To store the data in memory I hooked into our “Hosts” API. The Hosts API is a hash table like the Flow table that can be used to store data per host. It’s in use for Tagging and Thresholding. I added storage for IP Reputation to it.

Rule keyword

A new rule keyword to match on the reputation data was introduced: “iprep”. The keyword allows a rule to match on a specific category. Example:

alert ... (flow:to_server; iprep:src,Bot,>,10;)

This will generate an alert if the SRC IP of the host talking to a server is known to have a score of >10 in the “Bot” category.

The keyword is compatible to Suricata’s concept of “IP-only” rules. These are rules that do not inspect packet content or flow state and can thus be inspected once per flow direction instead of for each packet.

Speed

I’ve been playing with data sets of up to a million entries. Loading it takes hardly any time and I’m confident larger numbers will work just fine. The host table just needs bigger memcaps and hash sizes.

At runtime, the speed depends mostly on the rules. A pure “iprep” rule is quite expensive when not IP-only, although this is mostly due to the frequency of the checks. Such rules will be checked against large numbers of packets.

When created as a IP-only rule, things change. Such rules are checked only once per flow direction, so overhead appears to be minimal in this case.

Data

The data I used from Emerging Threats Pro is not available for free, so for those who want to test creating your own data is required right now. Matt Jonkman from Emerging Threats Pro will make a free feed available within a few weeks though. Of course you could also get the paid data from Emerging Threats Pro.

Update 29/11/2012

This feature is part of the just released 1.4rc1 version, please help us test it!

The OISF development team is proud to announce Suricata 1.4rc1, the first (and hopefully only) release candidate for the upcoming 1.4 version.

This release adds two major new features: a unix socket command mode, allowing for easy processing of large numbers of pcap files, and IP reputation. Both features are considered experimental.

Get the new release here: suricata-1.4rc1.tar.gz

New features

• Interactive unix socket mode (#571, #552)
• IP Reputation: loading and matching (#647)
• Improved --list-keywords commandline option gives detailed info for supported keyword, including doc link (#435)

Improvements

• Rule analyzer improvement wrt ipv4/ipv6, invalid rules (#494)
• User-Agent added to file log and filestore meta files (#629)
• Endace DAG supports live stats and at exit drop stats (#638)
• Add support for libhtp event "request port doesn't match tcp port" (#650)

Fixes

• Rules with negated addresses will not be considered IP-only (#599)
• Rule reloads complete much faster in low traffic conditions (#526)
• Suricata -h now displays all available options (#419)
• Luajit configure time detection was improved (#636)
• Flow manager mutex used w/o initialization (#628)
• Cygwin work around for windows shell mangling interface string (#372)
• Fix a Prelude output crash with alerts generated by rules w/o classtype or msg (#648)
• CLANG compiler build fixes (#649)
• Several fixes found by code analyzers

Credits

We'd like to thank the following people and corporations for their contributions and feedback:

• Jason Ish -- Endace
• Ludovico Cavedon -- Lastline
• Last G

Known issues & missing features

This is a "release candidate"-quality release so the stability should be good although unexpected corner cases might happen. If you encounter one, please let us know!

As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The next Ubuntu release currently in development, 13.04, just received an update to Suricata 1.3.4 and libhtp 0.2.11.

If you are running Ubuntu 13.04, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Ubuntu package has IPS mode through NFQUEUE enabled.

The OISF also provides a PPA for this and other Ubuntu versions: instructions can be found here.

Debian maintainer Pierre Chifflier updated the Suricata and libhtp packages in Debian Sid to 1.3.4 and 0.2.11.

If you are running Debian Sid, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Debian package has IPS mode through NFQUEUE enabled.

The naive approach would consider that an IDS is just taking packet and doing a lot of matching on it. In fact, this is not at all what is happening. An IDS/IPS like Suricata is in fact rebuilding the data stream and in case of known protocols it is even normalizing the data stream and providing keyword which can be used to match on specific field of a protocol.

Let’s say, we a rule to match on a HTTP request where method is GET and the URL is “/download.php”.

But what happen inside Suricata when want to do such a match ? Let’s try to visualize this matching on a fictive example packets stream. With such a stream, we could have the following process:

If you click on the image, you will get access to an interactive svg showing the alerting signatures.

A series of 7 IP packets are seen. They belong to the same flow but because of fragmentation they need to be assembled by the defrag engine. The #4 packet is invalid due to an invalid checksum. Suricata can alert on this packet if the following rule is activated:

alert ip any any -> any any (msg:"SURICATA IPv4 invalid checksum"; ipv4-csum:invalid; sid:2200073; rev:1;)

This rule is included in the provided signature file decoder-events.rules.

To match on individual TCP packet after defragmentation, one can use the following rule:

alert tcp-pkt any any -> any 80 (msg:"HTTP dl"; content:"Get /download.php"; sid:1; rev:1;)

It will try to find a per-packet match. This means that if the request is cut in two parts, there will be no detection.

At the TCP level, we’ve got three packets but one of them is invalid because of an invalid TCP windows. Suricata can alert on this by using the following rules:

alert tcp any any -> any any (msg:"SURICATA STREAM ESTABLISHED packet out of window"; stream-event:est_packet_out_of_window; sid:2210020; rev:1;)

This rule is included in the provided signature file stream-events.rules.

So at the stream level, we’ve got data made of packets #1, #2, #3 and #5. A matching rule at this level would be:

alert tcp any any -> any any (msg:"HTTP download"; flow:established,to_server; content:"Get /download.php";)

This is a case sensitive rule and this is not resistant to basic transformation on the request such as adding spaces between Get and the URI.

The data is part of an HTTP stream and it is normalized to avoid any application level manipulation that could alter the detection by signatures. In Suricata, we can use the following rule to have a match on normalized field:

alert http any any -> any any (msg:"Download"; content: "GET"; http_method; content: "/download.php"; http_uri)

The match is made when the traffic is identified as HTTP and we want the HTTP request method to be GET and the URL to match “/download.php”. We’ve got here one of the biggest advantage of Suricata, dedicated keywords and protocol recognition allow to write rule which are almost direct expression of our thinking.

I’m sure most of you are happy not to have this job cleanly done by Suricata!

The OISF development team is proud to announce Suricata 1.4beta3. This is the third beta release for the upcoming 1.4 version.

This is release has significant improvements to the packet acquisition. The Napatech capture card support has been updated by our supporter Npulse. The Pcap, PF_RING and AF_PACKET capture methods now feature live drop stats.

Get the new release here: suricata-1.4beta3.tar.gz

New features

• support for Napatech cards through their 3rd generation driver was added by Matt Keeler from Npulse (#430, #619)
• support for pkt_data keyword was added
• user and group to run as can now be set in the config file
• make HTTP request and response body inspection sizes configurable per HTTP server config (#560)
• PCAP/AF_PACKET/PF_RING packet stats are now printed in stats.log (#561, #625)
• add stream event to match on overlaps with different data in stream reassembly (#603)

Improvements

• add contrib directory to the dist (#567)
• performance improvements to signatures with dsize option
• improved rule analyzer: print fast_pattern along with the rule (#558)
• fixes to stream engine reducing the number of events generated (#604)
• stream.inline option new defaults to “auto”, meaning enabled in IPS mode, disabled in IDS mode (#592)
• HTTP handling in OOM condition was greatly improved (#557)
• filemagic keyword performance was improved (#585)
• updated bundled libhtp to 0.2.11
• build system improvements and cleanups

Fixes

• fixes and improvements to daemon mode (#624)
• fix drop rules not working correctly when thresholded (#613)
• fixed a possible FP when a regular and “chopped” fast_pattern were the same (#581)
• fix a false possitive condition in http_header (#607)
• fix inaccuracy in byte_jump keyword when using “from_beginning” option (#627)
• fixes to rule profiling (#576)
• cleanups and misc fixes (#379, #395)
• fix to SSL record parsing

Credits

We’d like to thank the following people and corporations for their contributions and feedback:

• Matt Keeler – Npulse
• Chris Wakelin
• Rmkml
• Will Metcalf
• Ivan Ristic
• Kyle Creyts
• Michael Hoffrath

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is pleased to announce Suricata 1.3.4. This is the fourth maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: suricata-1.3.4.tar.gz

Fixes

• fix crash in flow and host engines in cases of low memory or low memcap settings (#617)
• improve http handling in low memory conditions (#620)
• fix inaccuracy in byte_jump keyword when using “from_beginning” option (#626)
• fix building on OpenBSD 5.2
• update default config’s defrag settings to reflect all available options
• fixes to make check
• fix to SSL record parsing

Credits

• Rmkml
• Will Metcalf
• Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is proud to announce Suricata 1.4beta3. This is the third beta release for the upcoming 1.4 version.

This is release has significant improvements to the packet acquisition. The Napatech capture card support has been updated by our supporter Npulse. The Pcap, PF_RING and AF_PACKET capture methods now feature live drop stats.

Get the new release here: suricata-1.4beta3.tar.gz

New features

• support for Napatech cards through their 3rd generation driver was added by Matt Keeler from Npulse (#430, #619)
• support for pkt_data keyword was added
• user and group to run as can now be set in the config file
• make HTTP request and response body inspection sizes configurable per HTTP server config (#560)
• PCAP/AF_PACKET/PF_RING packet stats are now printed in stats.log (#561, #625)
• add stream event to match on overlaps with different data in stream reassembly (#603)

Improvements

• add contrib directory to the dist (#567)
• performance improvements to signatures with dsize option
• improved rule analyzer: print fast_pattern along with the rule (#558)
• fixes to stream engine reducing the number of events generated (#604)
• stream.inline option new defaults to "auto", meaning enabled in IPS mode, disabled in IDS mode (#592)
• HTTP handling in OOM condition was greatly improved (#557)
• filemagic keyword performance was improved (#585)
• updated bundled libhtp to 0.2.11
• build system improvements and cleanups

Fixes

• fixes and improvements to daemon mode (#624)
• fix drop rules not working correctly when thresholded (#613)
• fixed a possible FP when a regular and "chopped" fast_pattern were the same (#581)
• fix a false possitive condition in http_header (#607)
• fix inaccuracy in byte_jump keyword when using "from_beginning" option (#627)
• fixes to rule profiling (#576)
• cleanups and misc fixes (#379, #395)
• fix to SSL record parsing

Credits

We'd like to thank the following people and corporations for their contributions and feedback:

• Matt Keeler - Npulse
• Chris Wakelin
• Rmkml
• Will Metcalf
• Ivan Ristic
• Kyle Creyts
• Michael Hoffrath

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The OISF development team is pleased to announce Suricata 1.3.4. This is the fourth maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: suricata-1.3.4.tar.gz

Fixes

• fix crash in flow and host engines in cases of low memory or low memcap settings (#617)
• improve http handling in low memory conditions (#620)
• fix inaccuracy in byte_jump keyword when using "from_beginning" option (#626)
• fix building on OpenBSD 5.2
• update default config's defrag settings to reflect all available options
• fixes to make check
• fix to SSL record parsing

Credits

• Rmkml
• Will Metcalf
• Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Debian maintainer Pierre Chifflier updated the Suricata and libhtp packages in Debian Sid to 1.3.3 and 0.2.10.

If you are running Debian Sid, updating is as simple as:
apt-get update && apt-get upgrade

Installing is as simple as:
apt-get update && apt-get install suricata

The Debian package has IPS mode through NFQUEUE enabled.

We just released Suricata 1.3.3 which contains some important accuracy fixes. Also, it should be much more robust against out of memory conditions.

For those of you running Suricata in IPS mode, this is important as well. We found that rules that have the drop or reject actions, were not playing well with thresholding.

So upgrading is highly recommended!

Code changes are not too big, largest changes are due to some extra unittests:

 ChangeLog                           |   11 +
 libhtp/htp/dslib.c                  |    4 +-
 libhtp/htp/hooks.c                  |   31 +-
 libhtp/htp/htp_connection.c         |   34 ++-
 libhtp/htp/htp_connection_parser.c  |   25 +-
 libhtp/htp/htp_parsers.c            |    2 +-
 libhtp/htp/htp_request.c            |    4 +-
 libhtp/htp/htp_request_apache_2_2.c |   24 +-
 libhtp/htp/htp_transaction.c        |   68 +++--
 libhtp/htp/htp_util.c               |   35 ++-
 src/alert-debuglog.c                |    4 +-
 src/app-layer.c                     |    9 +-
 src/decode.h                        |    3 +-
 src/detect-detection-filter.c       |   96 ++++++
 src/detect-engine-alert.c           |   37 ++-
 src/detect-engine-hcbd.c            |    5 +
 src/detect-engine-hhd.c             |  121 +++++++-
 src/detect-engine-hsbd.c            |    5 +
 src/detect-engine-iponly.c          |    5 +-
 src/detect-engine-payload.c         |   26 ++
 src/detect-engine-threshold.c       |   15 +-
 src/detect-filemd5.c                |   24 +-
 src/detect-filestore.c              |   11 +-
 src/detect-filestore.h              |    2 +-
 src/detect-pcre.c                   |  485 +----------------------------
 src/detect-threshold.c              |  569 ++++++++++++++++++++++++++++++++++-
 src/detect.c                        |   11 +-
 src/detect.h                        |    2 +-
 src/flow-hash.c                     |   10 +-
 src/flow-timeout.c                  |   10 +-
 src/flow.c                          |    1 -
 src/flow.h                          |   14 +
 src/log-httplog.c                   |    2 +-
 src/runmodes.c                      |    2 +-
 src/source-ipfw.c                   |    1 +
 src/source-pfring.c                 |   20 +-
 src/stream-tcp-reassemble.c         |    4 +-
 src/stream-tcp.c                    |   12 +-
 src/stream.c                        |    3 +-
 src/threads.h                       |    1 +
 src/tmqh-packetpool.c               |    5 +-
 src/util-buffer.h                   |    6 +-
 src/util-debug.c                    |    2 +-
 src/util-host-os-info.c             |   32 +-
 src/util-threshold-config.c         |  210 +++++++++++++
 suricata.yaml.in                    |    6 +-
 46 files changed, 1340 insertions(+), 669 deletions(-)

The OISF development team is pleased to announce Suricata 1.3.3. This is the second maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.3.tar.gz

Fixes

• fix drop rules not working correctly when thresholded (#615)
• fix a false possitive condition in http_header (#606)
• fix extracted file corruption (#601)
• fix a false possitive condition with the pcre keyword and relative matching (#588)
• fix PF_RING set cluster problem on dma interfaces (#598)
• improve http handling in low memory conditions (#586, #587)
• fix FreeBSD inline mode crash (#612)
• suppress pcre jit warning (#579)

Credits

• Will Metcalf
• Chris Wakelin
• Kyle Creyts
• Michael Hoffrath

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal.  With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

The Dutch security site security.nl has interviewed me about the Suricata IDS project. The two part (Dutch language) article can be found here and part two here.

Thanks to Joran Polak of security.nl for giving me the opportunity to tell something about this project!

lstlisting

is a convenient way to display code when using latex. It has no definition for suricata rules language and I’ve cooked one:

\lstdefinelanguage{suricata}
{morekeywords= {alert, tcp, http, tls, ip, ipv4, ipv4, drop, pass, sid, priority, rev, classtype, threshold, metadata, reference, tag, msg, content, uricontent, pcre, ack, seq, depth, distance, within, offset, replace, nocase, fast\_pattern, rawbytes, byte\_test, byte\_jump, sameip, ip\_proto, flow, window, ftpbounce, isdataat, id, rpc, dsize, flowvar, flowint, pktvar, noalert, flowbits, stream\_size, ttl, itype, icode, tos, icmp\_id, icmp\_seq, detection\_filter, ipopts, flags, fragbits, fragoffset, gid, nfq\_set\_mark, tls.version, tls.subject, tls.issuerdn, tls.fingerprint, tls.store, http\_cookie, http\_method, urilen, http\_client\_body, http\_server\_body, http\_header, http\_raw\_header, http\_uri, http\_raw\_uri, http\_stat\_msg, http\_stat\_code, http\_user\_agent, ssh.protoversion, ssh.softwareversion, ssl\_version, ssl\_state, byte\_extract, file\_data, dce\_iface, dce\_opnum, dce\_stub\_data, asn1, filename, fileext, filestore, filemagic, filemd5, filesize, l3\_proto, luajit},
otherkeywords={ipv4-csum, tcpv4-csum, tcpv6-csum, udpv4-csum, udpv6-csum, icmpv4-csum, icmpv6-csum, decode-event, app-layer-event, engine-event, stream-event},
20
sensitive=true,
morecomment=[l]{//},
morecomment=[s]{/*}{*/},
morestring=[b]",
}

To use it, you can simply add this code at start of your tex file and you can then use it:

\begin{lstlisting}[language=suricata]
alert tcp any 21 -> any any (msg:"Overlap data";   \
  flow:to_client; dsize:>0;                        \
  stream-event:reassembly_overlap_different_data;  \
  classtype:protocol-command-decode; sid:1; rev:1;)
\end{lstlisting}

which give you the following result:

By the way, the lst of keywords has been obtained by running the till now hidden command:

suricata --list-keywords

I recently found out that Fedora includes Vuurmuur in it’s repositories. Since Suricata is also included, I figured I would do a quick write up on how to setup a Fedora IPS. While writing it turned more into a real “howto”, so I decided to submit it to Howtoforge.

It can be found here one HowtoForge.

Vuurmuur on Fedora is at the 0.7 version, which is still the current stable. It’s rather old though, and it reminds me again I need to make sure the 0.8 branch gets to a stable release soon. The Suricata included in Fedora 17 is 1.2.1, with 1.3.2 expected to land any day now.

The guide sets the user up from base Fedora install to a working IPS, but doesn’t cover any advanced topics such as rule management, event management etc. Still, I hope it’s useful to some, especially those that are intimidated by Vuurmuur’s and Suricata’s initial learning curves.

Looking forward to feedback!

Introduction

Some times ago, I’ve blogged about new IPS features in Suricata 1.1 and did not find at the time any killer application of the nfq_set_mark keyword. When using Suricata in Netfilter IPS mode, this keyword allows you to set the Netfilter mark on the packet when a rule match. This mark can be used by Netfilter or by other network subsystem to differentiate the treatment to apply to the packet.

It takes some time but I’ve found the idea. People around me know I don’t like Microsoft Office. One of the main reason, is that this is as awful as LibreOffice or OpenOffice.org but, on top of that, you’ve got pay for it. And, even if I don’t consider that people sending MS-Word file on the network should pay for that, I think they should at least benefit from a really slow bandwidth. I thus decided to implement this with Suricata and Netfilter.

If I’ve already told you that we will use the nfq_set_mark to mark the packet, one big task remains: how can I detect when someone upload a MS-Word file ? The answer is in the file extraction capabilities of Suricata. My preferred IDS/IPS is able to inspect HTTP traffic and to extract files that are uploaded or downloaded. You can then access to information about them. One of the new keyword is filemagic which return the same output as if you had run the unix file command on the file.

The setup

Now everything is in place. We just need to:

• Write a signatures to mark packet relative to upload
• Set up Suricata as IPS
• Set up Netfilter to send all HTTP traffic to Suricata

The signature is really simple. We want HTTP traffic and a transferred file which is of type “Composite Document File V2 Document”. When this is the case, we alert and set the mark to 1:

alert http any any -> any any (msg:"Microsoft Word upload"; \
          nfq_set_mark:0x1/0x1; \
          filemagic:"Composite Document File V2 Document"; \
          sid:666 ; rev:1;)

Let suppose we’ve saved this signature into word.rules. Now we can start suricata with:

suricata -q 0 -S word.rules

This is cool, we now have a single packet that will be marked. But, as we want to slow down all the connection, we need to propagate the mark. We call Netfilter to the rescue and our friend here is CONNMARK that will transfer the mark to all packets:

iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark

If we are masochistic enough to try to slow yourself down, we need to add the following line to take care of OUTPUT packets:

iptables -A OUTPUT -t mangle -j CONNMARK --restore-mark

Now that the marking is done, let’s suppose that eth0 is the interface to Internet. In that case, we can setup the Quality of Service. We create a HTB disc and create a 1kbps class for our MS-Word uploader:

tc qdisc add dev eth0 root handle 1: htb default 0
tc class add dev eth0 parent 1: classid 1:1 htb rate 1kbps ceil 1kbps

Now, we can do the link with our packet marking by assigning packet of handle 1 fw to the 1kpbs class (flowid 1:1):

tc filter add dev eth0 parent 1: protocol ip prio 1 handle 1 fw flowid 1:1

The job is almost done, last step is to send the port 80 packet to suricata:

iptables -I FORWARD -p tcp –dport 80 -j NFQUEUE
iptables -I FORWARD -p tcp –sport 80 -j NFQUEUE

If we are sadistic, we can also treat the local traffic:

iptables -I OUTPUT -p tcp –dport 80 -j NFQUEUE
iptables -I INPUT -p tcp –sport 80 -j NFQUEUE

That’s all with that setup, all MS-Word uploader share a 1kbps bandwith.

Effort should pay

Even among the MS-Word users, there is some people with brain and they will try to escape the MS-Word curse by changing the extension of their document before uploading them. Effort must pay, so we will change the setup to provide the 10kbps for uploading. To do so, we add a signature to word.rules that will detect when a MS-Word file is uploaded with a changed extension:

alert http any any -> any any (msg:"Tricky Microsoft Word upload";
                nfq_set_mark:0x2/0x2; \
                fileext:!"doc"; \
                filemagic:"Composite Document File V2 Document";
                filestore;
                sid:667; rev:1;)

End of the task will be to send packet with mark 2 on a privileged pipe:

tc class add dev eth0 parent 1: classid 1:2 htb rate 10kbps ceil 10kbps
tc filter add dev eth0 parent 1: protocol ip prio 1 handle 2 fw flowid 1:2

I’m sure a lot of you think we have been to kind and that the little cheaters must be watch over. We’ve already used the filestore keyword in their signature to put the uploaded file on disk but that is not enough.

Keep an eye on cheaters

The traffic of the cheater must be watch over. To do so, we will send all the packets they exchange inside a pcap file. We will need multiple tools to do so. The first of them will be ipset that we will use to maintain a list of bad guys. And we will use ulogd to store their traffic into a pcap file.

To create the blacklist, we will just do:

ipset create cheaters hash:ip timeout 3600
iptables -A POSTROUTING -t mangle -m mark \
    --mark 0x2/0x2 \
    -j SET --add-set cheaters src --exists

The first line creates a set named cheaters that will contains IP. Every element of the set will stay for 1 hour in the set. The second line send all source IP that send packet which got the mark 2. If the IP is already in the set, it will see its timeout reset to 3600 thanks to the –exists option.

Now we will ask Nefilter to log all traffic of IP of the cheaters set:

iptables -A PREROUTING -t raw \
    -m set --match-set cheaters src,dst \
    -j NFLOG --nflog-group 1

The last step is to use ulogd to store the traffic in a pcap file. We need to modify a standard ulogd.conf to have the following lines not commented:

plugin="/home/eric/builds/ulogd/lib/ulogd/ulogd_output_PCAP.so"
stack=log2:NFLOG,base1:BASE,pcap1:PCAP

Now, we can start ulogd:

ulogd -c ulogd.conf

A PCAP file named /var/log/ulogd.pcap will be created and will contain all the packets of cheaters.

Conclusion

It has been hard. We’ve fight a lot. We’ve coded a lot. But now, they are done! We’re able to know more about them than a Facebook admin does!

Today, a day after 1.3.2, we’ve released 1.4beta2. While 1.3.2 is an important update for those running 1.3.1 or lower, today’s release is where things get exciting. A lot of things were improved and added. Let me show some numbers first.

The 1.4beta2 release is a pretty big update over 1.4beta1 as it touches over 5k lines of code:

234 files changed, 5033 insertions(+), 3759 deletions(-)

Compared to 1.4beta2 vs yesterday’s 1.3.2 it’s clear over 11k lines of code are touched:

262 files changed, 11406 insertions(+), 5794 deletions(-)

Personally, I’ve been working on two main area’s: defrag engine and the luajit integration, and a couple of other things.

Defrag

The defrag engine was the last major subsystem that still used a Big Lock. Defrag uses so called “trackers” to track fragments belonging to a single IP packet. These trackers are stored in a hash table. 1.3 and prior used a hash that had no locking, so it relied on a Big Lock to protect it’s operations. Suricata has had fine grained hashes for flow and host tables for some time already, so it made sense to port defrag over as well.

Luajit

I’ve written about the luajit a couple of times already. While the basic functionality debuted in beta1, the code has been completely overhauled. The most important change that is user visible is the integration with the various HTTP inspection engines. This did result in a limitation though, for now you can just inspect one HTTP buffer per script.

A weird challenge with luajit is that it’s “state” needs to be in the 32 bit part of memory. The reason isn’t clear to me, but this gave us some trouble. Some users use many rules and agressive pattern matcher settings. When after this memory usage the luajit states had to be alloc’d, it failed. I’ve worked around this by allocating a bunch of states in advance, hoping they’ll end up in the proper memory. We’ll see how that will work.

Misc

I’ve also largely rewritten the optional rule profiling to perform better. Here too, a Big Lock was removed. The accounting is now first done on a per thread basis, and only merged at detection engine shut down. Another nice feature is that it will now print the profiling stats during a live rule reload as well.

Next, I’ve improved performance of the decode, stream and app layer event keywords. They were quite expensive as they were checked quite often. I’ve now added a prefilter check to the detection engine’s prefilter stage. Helps quite a bit!

Finally, I’ve been working on getting global and rule threshold play well together. This work isn’t done yet, but some real progress has been made. Work is tracker here and documentation lives here.

So all in all quite a bit of changes. Please help us test this so we can move to a stable and high performing 1.4!

The OISF development team is proud to announce Suricata 1.4beta2. This is the second beta release for the upcoming 1.4 version.

The main addition of this release is a usable lua scripting keyword for detection: luajit. This keyword allows you to run Lua scripts as part of the detection engine, allowing inspection beyond what the rule language offers. While not cheap, performance is not bad at all due to use of the luajit engine.

This release also brings major performance enhancements. We're able to get virtually packet loss free with AF_PACKET on our ISP test box with 6gbps-9gpbs of sustained traffic on commodity hardware with 7k rules.

Get the new release here: suricata-1.4beta2.tar.gz

New features

• New keyword: "luajit" to inspect packet, payload and all HTTP buffers with a Lua script (#346)
• Added ability to control per server HTTP parser settings in much more detail (#503)

Improvements

• Rewrite of IP Defrag engine to improve performance and fix locking logic (#512, #540)
• Big performance improvement in inspecting decoder, stream and app layer events (#555)
• Pool performance improvements (#541)
• Improved performance of signatures with simple pattern setups (#577)
• Bundled docs are installed upon make install (#527)
• Support for a number of global vs rule thresholds was added (#425)
• Improved rule profiling performance
• If not explicit fast_pattern is set, pick HTTP patterns over stream patterns. HTTP method, stat code and stat msg are excluded.

Fixes

• Fix compilation on architectures other than x86 and x86_64 (#572)
• Fix FP with anchored pcre combined with relative matching (#529)
• Fix engine hanging instead of exitting if the pcap device doesn't exist (#533)
• Work around for potential FP, will get properly fixed in next release (#574)
• Improve ERF handling. Thanks to Jason Ish
• Always set cluster_id in PF_RING
• IPFW: fix broken broadcast handling
• AF_PACKET kernel offset issue, IPS fix and cleanup
• Fix stream engine sometimes resending the same data to app layer
• Fix multiple issues in HTTP multipart parsing
• Fixed a lockup at shutdown with NFQ (#537)

Credits

We'd like to thank the following people and corporations for their contributions and feedback:

• Jason Ish - Endace
• Chris Wakelin
• Rmkml

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal.  With this in mind, please notice the list we have included of known items we are working on.

See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Today we released Suricata 1.3.2. Not a big update, but there are some important fixes in the stream engine, fast_pattern:chop handling, HTTP multipart parsing and the flow keyword with “nostream”.

As the diff stat output shows, it’s a rather light maintenance update over 1.3.1:

 ChangeLog                              |   12 ++
 libhtp/configure.ac                    |    2 +-
 libhtp/htp.pc.in                       |    2 +-
 libhtp/htp/htp.h                       |    2 +-
 src/app-layer-htp-file.c               |  145 ++++++++++++++++++++++++
 src/app-layer-htp.c                    |  192 ++++++++++++++++++++++++++------
 src/decode.c                           |    3 +
 src/decode.h                           |    1 +
 src/defrag.c                           |    4 +-
 src/detect-engine-content-inspection.c |    9 --
 src/detect-flow.c                      |   68 ++++++++++-
 src/source-af-packet.c                 |    9 ++
 src/source-ipfw.c                      |   13 ++-
 src/source-pfring.c                    |   28 ++---
 src/stream-tcp-reassemble.c            |    1 +
 src/util-cpu.c                         |   10 +-
 16 files changed, 435 insertions(+), 66 deletions(-)

Only the HTTP changes look big, but that is due to adding some unittests. Same for flow keyword.

Because of the fixes updating is still highly recommended. Most fixes improve detection accuracy.

Full notes at our new website: http://suricata-ids.org/2012/10/03/suricata-1-3-2-available/

The OISF development team is pleased to announce Suricata 1.3.2. This is the second maintenance release of Suricata 1.3 with some important fixes.

Because of the fixes below, upgrading is highly recommended.

Download: http://www.openinfosecfoundation.org/download/suricata-1.3.2.tar.gz

Fixes

• Fixed a possible FP when a regular and "chopped" fast_pattern were the same (#562)
• Fixed a FN condition with the flow:no_stream option (#575)
• Fix building of perf profiling code on i386 platform. By Simon Moon (#534)
• Fix multiple issues in HTTP multipart parsing
• Fix stream engine sometimes resending the same data to app layer
• Always set cluster_id in PF_RING
• Defrag: silence some potentially noisy errors/warnings
• IPFW: fix broken broadcast handling
• AF_PACKET kernel offset issue

Credits

• Simon Moon
• Rmkml

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal.  With this in mind, please notice the list we have included of known items we are working on.

See http://redmine.openinfosecfoundation.org/projects/suricata/issues for an up to date list and to report new issues. See http://redmine.openinfosecfoundation.org/projects/suricata/wiki/Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Suricata has a new website within the OISF!

http://suricata-ids.org

This site is dedicated to all things Suricata, and focuses on more detailed information as to why we hope you'll give Suricata a try in your environment.

We welcome feedback on the new Suricata site. Putting this separate site up from the OISF's main site was in response to feedback that the primary drivers to give Suricata a try weren't featured well enough on the OISF site. So we're hoping this will help those not familiar with Suricata get a quick idea what's there, what it does, and how to give it a run.

If you have a service or platform based upon or that supports Suricata please contact us directly to be added to an upcoming "Suricata Runs On" page! Being listed here is free and helps those interested in Suricata find your products and services.

Introduction

I’ve been working for the past few days on a new Suricata feature. It is available in Suricata 1.4rc1.

Suricata can now listen to a unix socket and accept commands from the user. The exchange protocol is JSON-based and the format of the message has been done to be generic and it is described in this commit message. An example script called suricatasc is provided in the source and installed automatically when updating Suricata.

Unix socket command

By setting enabled to yes under unix-command in Suricata YAML configuration file, the creation of the socket is now activated:

unix-command:
  enabled: yes
  #filename: custom.socket # use this to specify an alternate file

This provides for now a set of really exciting commands like:

• shutdown: this shutdown suricata
• iface-list: list interfaces where Suricata is sniffing packets
• iface-stat: list statistic for an interface

For example, a typical session with suricatasc will looks like:

# suricatasc 
>>> iface-list
Success: {'count': 2, 'ifaces': ['eth0', 'eth1']}
>>> iface-stat eth0
Success: {'pkts': 378, 'drop': 0, 'invalid-checksums': 0}

Useful but not really sexy for now. But the main part is the dedicated running mode.

Unix socket running mode

This mode is one of main motivation behind this code. The idea is to be able to ask to Suricata to treat different pcap files without having to restart Suricata between the files. This provides you a huge gain in time as you don’t need to wait for the signature engine to initialize.

To use this mode, start suricata with your preferred YAML file and provide the option –unix-socket as argument:

suricata -c /etc/suricata-full-sigs.yaml --unix-socket

Then, you can use the provided script suricatasc to connect to the command socket and ask for pcap treatment:

root@tiger:~# suricatasc
>>> pcap-file /home/benches/file1.pcap /tmp/file1
Success: Successfully added file to list
>>> pcap-file /home/benches/file2.pcap /tmp/file2
Success: Successfully added file to list

Yes, you can add multiple files without waiting the result: they will be sequentially processed and the generated log/alert files will be put into the directory specified as second arguments of the pcap-file command. You need to provide absolute path to the files and directory as suricata don’t know from where the script has been run.

To know how much files are waiting to get processed, you can do:

>>> pcap-file-number
Success: 3

To get the list of queued files, do:

>>> pcap-file-list
Success: {'count': 2, 'files': ['/home/benches/file1.pcap', '/home/benches/file2.pcap']}

suritasc script is just intended to be a basic tool to send commands. The protocol has been choozen to be simple so people can easily build their own scripts.

Some words about the protocol

The client connect to the socket and sends its protocol version:

{ "version": "$VERSION_ID" }

The server sends an answer:

{ "return": "OK|NOK" }

If server returns OK, the client is now allowed to send command.

The format of command is the following:

{
   "command": "$COMMAND_NAME",
   "arguments": { $KEY1: $VAL1, ..., $KEYN: $VALN }
}

For example, to ask suricata to treat a pcap, the command is something like:

{
  "command": "pcap-file",
  "arguments": { "filename": "smtp-clean.pcap", "output-dir": "/tmp/out" }
}

The server will try to execute the “command” specified with the (optional) provided “arguments”. The answer by server is the following:

{
  "return": "OK|NOK",
  "message": JSON_OBJECT or information string
}

Building it

This new features use jansson the encoding/decoding JSON and it needs thus to be installed on your system before you can do the build. On Debian (testing or sid) or Ubuntu, you can install libjansson-dev

Now, you can proceed with your normal build. For example:

./autogen.sh
./configure
make
make install

You can check if the support will be build in the message at end of configure:

Suricata Configuration:
  AF_PACKET support:                       yes
...
  Unix socket enabled:                     yes

  libnss support:                          no
  libnspr support:                         no
  libjansson support:                      yes

If this is your first install, you may want to run make install-full to get a working configuration of Suricata with Emerging Threats rules being downloaded and setup.

Now, you can just do:

suricata --unix-socket

Conclusion

This new features can be really interesting for people that are using Suricata to parse a large numbers of pcap. But the unix command may be really interesting when more commands will be available. Regarding this don’t hesitate to give me some feedbacks or ideas.