Planet Suricata

July 20, 2016

Security Onion

Suricata 3.1.1 now available for Security Onion!

Suricata 3.1.1 was recently released:
https://suricata-ids.org/2016/07/13/suricata-3-1-1-released/

I've packaged it and the following package is now available:
securityonion-suricata - 3.1.1-1ubuntu1securityonion1

This new package should resolve the following issue:

Issue 945: Suricata 3.1.1
https://github.com/Security-Onion-Solutions/security-onion/issues/945

This packages has been tested by Wes Lambert.  Thanks, Wes!

Updating
This package is now available in our stable repo.  Please see the following page for full update instructions:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak and migrate your HOME_NET and EXTERNAL_NET variables.  You'll then need to do the following:

  • re-apply any other local customizations to your snort.conf file(s)
  • update ruleset and restart Suricata as follows:
    sudo rule-update

Want to show your support for Security Onion?
Security Onion t-shirts are available in our CafePress store!
http://www.cafepress.com/securityonion/11820053

Conference
Security Onion Conference will be on Friday September 9 and registration is open!
https://securityonion.net/conference

Training
Need training?  Please see:
https://securityonionsolutions.com

Support
Need support?  Please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Support

Thanks!

by Doug Burks (noreply@blogger.com) at July 20, 2016 01:49 PM

April 25, 2016

Security Onion

Suricata 3.0.1 now available for Security Onion!

Suricata 3.0.1 was recently released:
https://suricata-ids.org/2016/04/04/suricata-3-0-1-released/

I've packaged Suricata 3.0.1 and the new package version is:
securityonion-suricata - 3.0.1-1ubuntu1securityonion1

This resolves the following issue:

Issue 896: Suricata 3.0.1
https://github.com/Security-Onion-Solutions/security-onion/issues/896

Wes Lambert and wingmanjt tested this package.  Thanks, guys!

Updating
This new package is now available in our stable repo.  Please see the following page for full update instructions:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak and migrate your HOME_NET and EXTERNAL_NET variables.  You'll then need to do the following:

  • re-apply any other local customizations to your snort.conf file(s)
  • update ruleset and restart Suricata as follows:
    sudo rule-update

Want to show your support for Security Onion?
Several folks have asked about Security Onion t-shirts and they are available in our CafePress store!
http://www.cafepress.com/securityonion/11820053

Training
Our next round of online classes is coming up in a few weeks:
http://blog.securityonion.net/2016/03/next-round-of-security-onion-online.html

Conference
Security Onion Conference will be on Friday September 9 and CFP is open!
http://blog.securityonion.net/2016/03/security-onion-conference-2016-cfp.html

Support
Need support?  Please see:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Support

Thanks!

by Doug Burks (noreply@blogger.com) at April 25, 2016 03:26 PM

April 19, 2016

Peter Manev

Suricata with afpacket - the memory of it all


Suricata IDS/IPS/NSM is a highly scalable, modular and flexible platform. There are numerous configuration options available which empower a lot.

This blog post aims to give you an overview of the ones that have an impact on the memory consumption for Suricata and how does the suricata.yaml config settings affect the memory usage of/for Suricata and the system it resides on.

One of the always relevant questions with regards to performance tuning and production deployment is  - What is the total memory consumption of Suricata? Or to be correct - what is the total memory that Suricata will consume/use and how can that be calculated and configured more precisely?

The details of the answer are very relevant since it will most certainly affect the deployment set up. The risk of not correctly setting up the configuration can lead to a RAM starvation which in turn would force the use of swap which would most likely make your particular set up not optimal (to be frank - useless).

In this blog post we will try to walk through the relevant settings in  a suricata.yaml configuration example and come up with an equation for the total memory consumption.

For this particular set up I use:
  • af-packet running mode with 16 threads configuration
  • runmode: workers
  • latest dev edition(git - 2.1dev (rev dcbbda5)) of Suricata at the time of this writing.
  • IDS mode is used in this example
  • Debian Jessie/Ubuntu LTS (the OS should not matter)

Lets dive into it...

MTU size does matter

How so?

If you look into the setting for max-pending-packets in suricata.yaml

max pending packets: 1024

that will lead to the following output into suricata.log:

.....
 (tmqh-packetpool.c:398) <Info> (PacketPoolInit) -- preallocated 1024 packets. Total memory 3321856
.....
which is  - 3244 bytes per packet per thread(pool).


The size of each packet takes in the memory is the sizeof(struct
Packet_) + DEFAULT_PACKET_SIZE. So ~1.7K plus ~1.5K or about 3.2K.
Total memory used would be:

<number_of_threads>*<(sizeof(struct Packet_) + DEFAULT_PACKET_SIZE)>*<max-pending-packets> = 16 * 65534 * 3.2K = 3.20GB.

NOTE: That much RAM will be reserved right away
NOTE: The number of threads does matter as well :)

So why is the NIC MTU important?

The MTU setting on the NIC (IDS) interface is used by af-packet as a default packet size(aka - DEFAULT_PACKET_SIZE) if no explicit default packet size is specified in the suricata.yaml:
# Preallocated size for packet. Default is 1514 which is the classical
# size for pcap on ethernet. You should adjust this value to the highest
# packet size (MTU + hardware header) on your system.
#default-packet-size: 1514
Note above the "default-packet-size" is commented/unset. In that case af-packet will use the MTU set on the NIC as a default packet size - which in this particular set up (NIC) if you do "ifconfig" is 1514.

So when you would like to play "big" and enable those 9KB jumbo frames as MTU on your NIC - without having a need for it  ....you may end up with an unwanted side effect the least :)


Defrag memory settings and consumption

defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep (higher than trackers)
  prealloc: yes
  timeout: 60

The setting above from the defrag section of the suricata.yaml will result in the following if you check your suricata.log:

defrag-hash.c:220) <Info> (DefragInitConfig) -- allocated 3670016
bytes of memory for the defrag hash... 65536 buckets of size 56
defrag-hash.c:245) <Info> (DefragInitConfig) -- preallocated 65535
defrag trackers of size 168
(defrag-hash.c:252) <Info> (DefragInitConfig) -- defrag memory usage: 14679896 bytes, maximum: 536870912

Here we have(in bytes) -
(defrag hash size * 56) + (prealloc defrag trackers * 168). In this case that would be a total of
(65536 * 56) + (65535 * 168) = 13.99MB
which is "defrag memory usage: 14679896 bytes" from the above output.

That much memory is immediately allocated/reserved.
The maximum memory usage allowed to be used by defrag will be 512MB.

NOTE: The defrag settings you configure for preallocation must sum up to be lower than the max amount allocated (defrag.memcap)


Host memory settings and consumption

host:
  hash-size: 4096
  prealloc: 10000
  memcap: 16777216

The setting above (host memory settings have effect on the ip reputation usage ) from the hosts section of the suricata.yaml will result in the following if you check your suricata.log:

(host.c:212) <Info> (HostInitConfig) -- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
(host.c:235) <Info> (HostInitConfig) -- preallocated 10000 hosts of size 136
(host.c:237) <Info> (HostInitConfig) -- host memory usage: 1622144 bytes, maximum: 1622144

Pretty simple (in bytes) -
(hash-size*64) + (prealloc_hosts * 136) =
(4096*64) + (10000 * 136) = 1622144 =1.54MB are allocated/reserved right away at start.
The maximum memory allowed is 162MB(16777216 Bytes)


Ippair memory settings and consumption


ippair:
  hash-size: 4096
  prealloc: 1000
  memcap: 16777216

The setting above (ippair memory settings have effect on the xbits usage) from the hosts section of the suricata.yaml will result in the following if you check your suricata.log:

(ippair.c:207) <Info> (IPPairInitConfig) -- allocated 262144 bytes of memory for the ippair hash... 4096 buckets of size 64
(ippair.c:230) <Info> (IPPairInitConfig) -- preallocated 1000 ippairs of size 136
(ippair.c:232) <Info> (IPPairInitConfig) -- ippair memory usage: 398144 bytes, maximum: 16777216

Pretty simple as well  (in bytes) -
(hash-size*64) + (prealloc_ippair * 136) =
(4096*64) + (1000 * 136) = 398144 =1.54MB will be allocated/reserved immediately upon start.
The maximum memory allowed is 162MB(16777216 Bytes)

Flow memory settings and consumption


flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30
  #managers: 1 # default to one flow manager
  #recyclers: 1 # default to one flow recycler thread

The setting above from the flow config section of the suricata.yaml will result in the following in your suricata.log:

[393] 7/6/2015 -- 15:37:55 - (flow.c:441) <Info> (FlowInitConfig) --
allocated 67108864 bytes of memory for the flow hash... 1048576
buckets of size 64
[393] 7/6/2015 -- 15:37:55 - (flow.c:465) <Info> (FlowInitConfig) --
preallocated 1048576 flows of size 280
[393] 7/6/2015 -- 15:37:55 - (flow.c:467) <Info> (FlowInitConfig) --
flow memory usage: 369098752 bytes, maximum: 1073741824

Here we have (in bytes) -
(flow hash * 64) + (prealloc flows * 280) which in this case would be
(1048576 * 64) + (1048576 * 280) = 344MB
The above is what is going to be immediately used/reserved at start up.
The max allowed usage will be 1024MB

A piece of advice if I may - don't ever add zeros here if you do not need to. By don't need to  - I mean if you do not  see flow emergency mode counters increasing in your stats.log.

Prealloc-sessions settings and consumption

stream:
  memcap: 32mb
  checksum-validation: no      # reject wrong csums
  prealloc-sessions: 20000
  inline: auto     

The setting above from the prealloc sessions config section of the suricata.yaml will result in the following in your suricata.log:

(stream-tcp.c:377) <Info> (StreamTcpInitConfig) -- stream "prealloc-sessions": 20000 (per thread)

This translates into bytes as follows (TcpSession structure is 192 bytes, PoolBucket is 24 bytes):
(192 + 24) * prealloc_sessions * number of threads = memory use in bytes
In our case we have - (192 + 24) * 20000 * 16 = 65.91MB. This amount will be immediately  allocated upon start up.
NOTE: The number of threads does matter as well :)

af-packet ring size memory settings and consumption


    use-mmap: yes
    # Ring size will be computed with respect to max_pending_packets and number
    # of threads. You can set manually the ring size in number of packets by setting
    # the following value. If you are using flow cluster-type and have really network
    # intensive single-flow you could want to set the ring-size independently of the number
    # of threads:
    ring-size: 2048

The setting above from the af-packet ring size config section of the suricata.yaml will result in the following in your suricata.log the ringsize setting actually controls the size of the buffer for each
ring(per thread) - buffer for af-packet:

[7636] 31/8/2015 -- 22:50:51 - (source-af-packet.c:1365) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=103 frame_size=1584 frame_nr=2060
[7636] 31/8/2015 -- 22:50:51 - (source-af-packet.c:1573) <Info> (AFPCreateSocket) -- Using interface 'eth0' via socket 7
[7636] 31/8/2015 -- 22:50:51 - (source-af-packet.c:1157) <Info> (ReceiveAFPLoop) -- Thread AFPacketeth01 using socket 7
[7637] 31/8/2015 -- 22:50:51 - (source-af-packet.c:1365) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=103 frame_size=1584 frame_nr=2060
[7637] 31/8/2015 -- 22:50:51 - (source-af-packet.c:1573) <Info> (AFPCreateSocket) -- Using interface 'eth0' via socket 8


In  general - that would mean -
<number of threads> * <ringsize> * <(sizeof(structPacket_) + DEFAULT_PACKET_SIZE)>
or in our case - 16*2048*3514=109MB
This is memory allocated/reserved immediately.

Above I say "in general". You  might wonder where does this come from:
(source-af-packet.c:1365) <Info> (AFPComputeRingParams) -- AF_PACKET RX Ring params: block_size=32768 block_nr=103 frame_size=1584 frame_nr=2060

Why 2060 frames while we have specified 2048,why block_size/frame_size and what is their relation? Full detailed description about that you can find here - https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt (thanks regit)


Stream and reassembly memory settings and consumption


stream:
 memcap: 14gb
 reassembly:
   memcap: 20gb


The setting above from the stream and reassembly config section of the
 suricata.yaml will result in the following in your suricata.log:
......
(stream-tcp.c:393) <Info> (StreamTcpInitConfig) -- stream "memcap": 15032385536
......
(stream-tcp.c:475) <Info> (StreamTcpInitConfig) -- stream.reassembly "memcap": 21474836480
......

The above is very straight forward. The stream and reassembly memcaps
are in total 14GB+20GB=34GB
This is max memory allowed. It will not be allocated immediately.

Further below in the config section we have :

     #raw: yes
     #chunk-prealloc: 250

Q: What does raw mean and what is chunk-prealloc?
A: The 'raw' stream inspection (content keywords w/o http_uri etc) uses
'chunks'. This is again a preallocated memory block that lives in a pool.

Q: So what is the size of "chunks " ?
A: 4kb/4096bytes

So in this case above we have -
250*4096 = 0.97MB
This is deducted/taken from the memory allocated by the  stream.reassembly.memcap value.

we also have prealloc segments (values in bytes):
    #randomize-chunk-range: 10
    #raw: yes
    #chunk-prealloc: 250
    #segments:
    #  - size: 4
    #    prealloc: 256
    #  - size: 16
    #    prealloc: 512
    #  - size: 112
    #    prealloc: 512
    #  - size: 248
    #    prealloc: 512
    #  - size: 512
    #    prealloc: 512
    #  - size: 768
    #    prealloc: 1024
    #  - size: 1448
    #    prealloc: 1024
    #  - size: 65535
    #    prealloc: 128
    #zero-copy-size: 128

More detailed info about the above you can find from my other blog post here - http://pevma.blogspot.se/2014/06/suricata-idsips-tcp-segment-pool-size.html

NOTE: Do not forget that these settings (segments preallocation) is deducted/taken from the memory allocated by the  stream.reassembly.memcap value.

App layer memory settings and consumption


app-layer:
 protocols:
   dns:
     # memcaps. Globally and per flow/state.
     global-memcap: 2gb
     state-memcap: 512kb
...
....
   http:
     enabled: yes
     memcap:2gb


Here we have - app-layer dns + http or in this case - 2GB + 2GB = 4GB

Other settings that affect the memory consumption


...
detect-engine:
  - profile: medium
  - custom-values:
...

Some more information:
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/High_Performance_Configuration

The more rules you load the heavier the effect of a switch in this setting will be. For example a switch from profile: medium to profile: high would be most evident if you would like to try with >10000 rules.

mpm-algo: ac

The memory algorithm is of importance of course. However ac and ac-bs are most performant with ac-bs being less mem intensive but also less performant.


Grand total generic memory consumption equation

So if we sum up all the config options that have effect on the total memory consumption by Suricata with mind of the set up referred to here  (afpacket with 16 threads) we have (in bytes or mb/gb depending how you have your yaml memcap settings):

<number_of_total_detection_threads>*<((1728)+(default_packet_size))>*<max-pending-packets>
+
<defrag.memcap>
+
< host.memcap>
+
< ippair.memcap>
+
 < flow.memcap>
+
 <number_of_threads>*<216>* <prealloc-sessions>
+
 [per af-packet interface enabled]<af-packet_number_of_threads> * <ringsize> * <((1728)+(default_packet_size))>
+
<stream.memcap>+<stream.reassembly.memcap>
+
 <app-layer.protocols.dns.global-memcap>
+
<app-layer.protocols.http.memcap>
=
Total memory that is configured and should be available to be used by Suricata


Thank you




by Peter Manev (noreply@blogger.com) at April 19, 2016 12:41 AM

April 04, 2016

suricata-ids.org

Suricata 3.0.1 released!

suri-400x400

We are pleased to announce Suricata 3.0.1. This release fixes many important issues in 3.0 and upgrading is highly recommended.

Highlights

– fixes for multiple stability issues
– many memory leak fixes
Hyperscan MPM support (experimental)

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-3.0.1.tar.gz

All Changes

For a complete list of closed tickets, please see:

3.0.1RC1 tickets
3.0.1 tickets

When upgrading from 3.0, please see these notes.

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

FireEye, Intel, ANSSI, Emerging Threats / Proofpoint, Stamus Networks,
NorCert, Ntop, Lastline, AFL project, CoverityScan

Justin Viiret, Tom Decanio, Mats Klepsland, Alexander Gozman,
Aleksey Katargin Maurizio Abba, Alessandro Guido, David Diallo,
Giuseppe Longo, Chris Wakelin, Jon Zeolla, Andreas Moe,
Nicolas Thill, Travis Green, bladeswords Alfredo Cardigliano,
Rob Mosher, Andrew Brown, Andre ten Bohmer

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

November 9-11 we’ll be in Washington, DC, for our 2nd Suricata User Conference: http://suricon.net

If you need help installing, updating, validating and tuning Suricata we have a training program. Please see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

by inliniac at April 04, 2016 01:11 PM

March 23, 2016

suricata-ids.org

Please help us test Suricata 3.0.1RC1

suri-400x400

We’re hoping for your feedback on a new release: Suricata 3.0.1RC1. We’ve fixed many issues in 3.0, including important stability issues and memory leaks. A final is expected within a week or so.

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-3.0.1RC1.tar.gz

New Features

– Feature #1535: Expose the certificate itself in TLS-lua
– Feature #1696: improve logged flow_id
– Feature #1700: enable “relro” and “now” in compile options for 3.0
– Feature #1734: gre: support transparent ethernet bridge decoding
– Feature #1740: Create counters for decode-events errors
– updated bundled libhtp to 0.5.19

Fixes

Many issues were fixed, including stability issues and many (potential) memory leaks.
Full list: https://redmine.openinfosecfoundation.org/versions/81

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:
FireEye, ANSSI, Emerging Threats / Proofpoint, Stamus Networks,
NorCert, Ntop, Lastline, AFL project, CoverityScan

Tom Decanio, Mats Klepsland, Alexander Gozman, Aleksey Katargin
Maurizio Abba, Alessandro Guido, David Diallo, Giuseppe Longo
Jon Zeolla, Andreas Moe, Nicolas Thill, Travis Green, bladeswords
Alfredo Cardigliano, Rob Mosher, Andre ten Bohmer

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

November 9-11 we’ll be in Washington, DC, for our 2nd Suricata User Conference: http://suricon.net

If you need help installing, updating, validating and tuning Suricata we have a training program. Please see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

by inliniac at March 23, 2016 12:51 PM

February 15, 2016

Victor Julien

Fuzzing Suricata with pcaps

Yesterday I wrote about fuzzing Suricata with AFL. Today I’m going to show another way. Since early in the project, we’ve shipped a perl based fuzzer called ‘wirefuzz’. The tool is very simple. It takes a list of pcaps, changes random bits in them using Wiresharks editcap and runs them through Suricata. Early in the project Will Metcalf, who wrote the tool, found a lot of issues with it.

Since it’s random based fuzzing, the fuzzing is quite shallow. It is still a great way of stressing the decoder layers of Suricata though, as we need to be able to process all junk input correctly.

Lately we had an issue that I thought should have been found using fuzzing: #1653, and indeed, when I started fuzzing the code I found the issue within an hour. Pretty embarrassing.

Another reason to revisit is Address Sanitizer. It’s great because it’s so unforgiving. If it finds something it blows up. This is great for fuzzing. It’s recommended to use AFL with Asan as well. Wirefuzz does support a valgrind mode, but that is very slow. With Asan things are quite fast again, while doing much more thorough checking.

So I decided to spend some time on improving this tool so that I can add it to my CI set up.

Here is how to use it.

git clone https://github.com/inliniac/suricata -b dev-fuzz-v3.1
cd suricata
git clone https://github.com/OISF/libhtp -b 0.5.x
bash autogen.sh
export CFLAGS="-fsanitize=address"
./configure --disable-shared --sysconfdir=/etc
make
mkdir fuzzer
# finally run the fuzzer
qa/wirefuzz.pl -r=/home/victor/pcaps/*/* -c=suricata.yaml -e=0.02 \
    -p=src/suricata -l=fuzzer/ -S=rules/http-events.rules -N=1

What this command does is:

  • run from the source dir, output into fuzzer/
  • modify 2% of each pcap randomly while making sure the pcap itself stays valid (-e=0.02)
  • use the rules file rules/http-events.rules exclusively (-S)
  • use all the pcaps from /home/victor/pcaps/*/*
  • return success if a single pass over the pcaps was done (-N=1)

One thing to keep in mind is that the script creates a copy of the pcap when randomizing it. This means that very large files may cause problems depending on your disk space.

I would encourage everyone to fuzz Suricata using your private pcap collections. Then report issues to me… pretty please?:)

*UPDATE 2/15*: the updated wirefuzz.pl is now part of the master branch.


by inliniac at February 15, 2016 03:42 PM

February 10, 2016

Security Onion

Suricata 3.0 STABLE now available for Security Onion 14.04!

Suricata 3.0 STABLE was recently released:
http://suricata-ids.org/2016/01/27/suricata-3-0-available/

I've packaged Suricata 3.0 STABLE for Security Onion 14.04 and the new package is as follows:
securityonion-suricata - 3.0stable-1ubuntu1securityonion1

This resolves the following issue:

Issue 847: Suricata 3.0
https://github.com/Security-Onion-Solutions/security-onion/issues/847

Wes Lambert tested this package.  Thanks, Wes!

Updating
These new packages are now available in our stable repo.  Please see the following page for full update instructions:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak and migrate your HOME_NET and EXTERNAL_NET variables.  You'll then need to do the following:

  • re-apply any other local customizations to your suricata.yaml file(s)
  • update ruleset and restart Suricata as follows:
    sudo rule-update

Want to show your support for Security Onion?
Several folks have asked about Security Onion t-shirts and they are available in our CafePress store!
http://www.cafepress.com/securityonion/11820053

Training
The next round of online training sessions is coming up soon:
http://blog.securityonion.net/2016/02/next-round-of-security-onion-online.html

Commercial Support
Need commercial support?  Please see:
http://securityonionsolutions.com

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists

Thanks!

by Doug Burks (noreply@blogger.com) at February 10, 2016 08:28 AM

February 09, 2016

Victor Julien

Fuzzing Suricata with AFL

AFL is a very powerful fuzzer, that tries to be smarter than random input generating fuzzers. It’s cool, but needs a bit more baby sitting. I’ve added some support to Suricata to assist AFL.

Here’s how to get started on fuzzing pcaps.

mkdir ~/tmp/fuzz
git clone https://github.com/inliniac/suricata -b dev-afl-v5
cd suricata
git clone https://github.com/OISF/libhtp -b 0.5.x
bash autogen.sh
export CFLAGS="-fsanitize=address"
export AFLDIR=/opt/afl-1.96b/bin/
export CC="${AFLDIR}/afl-gcc"
export CXX="${AFLDIR}/afl-g++"
./configure --disable-shared --sysconfdir=/etc --enable-afl

The configure output should show:
Compiler: /opt/afl-1.96b/bin//afl-gcc (exec name) / gcc (real)

make

# create tmp output dir for suricata
mkdir tmp/

# test the command to be fuzzed
src/suricata --runmode=single -k none -c suricata.yaml -l tmp/ \
    -S /dev/null \
    -r /opt/afl-1.96b/share/afl/testcases/others/pcap/small_capture.pcap

# start the fuzzer
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i /opt/afl-1.96b/share/afl/testcases/others/pcap/ -o aflout -- \
    src/suricata --runmode=single -k none -c suricata.yaml -l tmp/ \
    -S /dev/null -r @@

AFL should start running:

afl

Couple of things to keep in mind:

  • the above list assumes you have a /etc/suricata/ set up already, including a reference.config and classification.config
  • don’t skip the test step or you risk that AFL will just fuzz some basic error reporting by Suricata
  • the used ‘dev-afl-v5’ branch makes fuzzing faster and more reliable by disabling random, threading and a few other things
  • src/suricata –build-info should show the compiler is afl
  • keep your test cases small, even then runtime is going to be very long. AFL takes the input and modifies it to find as many unique code paths as possible

 

Fuzzing rules and YAMLs

For fuzzing rules and YAMLs the compilation steps are the same.

To fuzz rules, create a directory & test input:

mkdir testrules
echo 'alert http any any -> any any (content:"abc"; sid:1; rev:1;)' \
    > testrules/rules.txt

# test command
src/suricata -c suricata.yaml -l tmp/ --afl-parse-rules -T \
    -S testrules/rules.txt

# run AFL
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i testrules/ -o aflout -- \
    src/suricata -c suricata.yaml -l tmp/ --afl-parse-rules \
    -T -S @@

Finally, YAMLs:

mkdir testyamls/
cp suricata.yaml testyamls/

# test command
src/suricata -l tmp/ --afl-parse-rules -T -S testrules/rules.txt \
    -c testyamls/suricata.yaml

# run AFL
export AFL_SKIP_CPUFREQ=1
/opt/afl-1.96b/bin/afl-fuzz -t 100000 -m none \
    -i testyamls/ -o aflout -- \
    src/suricata -l tmp/ --afl-parse-rules \
    -T -S testrules/rules.txt -c @@

Note that the default YAML is HUGE for this purpose. It may be more efficient to use a sub set of it.

I plan to create some wrapper scripts to make things easier in the near future. Meanwhile, if you have crashes to report, please send them my way!


by inliniac at February 09, 2016 04:18 PM

January 27, 2016

Victor Julien

Suricata 3.0 is out!

suri-400x400Today, almost 2 years after the release of Suricata 2.0, we released 3.0! This new version of Suricata improves performance, scalability, accuracy and general robustness. Next to this, it brings a lot of new features.

New features are too numerous to mention here, but I’d like to highlight a few:

  • netmap support: finally a high speed capture method for our FreeBSD friends, IDS and IPS
  • multi-tenancy: single instance, multiple detection configs
  • JSON stats: making it much easier to graph the stats in ELK, etc
  • Much improved Lua support: many more fields/protocols available, output scripts

Check the full list here in the announcement: http://suricata-ids.org/2016/01/27/suricata-3-0-available/

New release model

As explained here, this is the first release of the new release model where we’ll be trying for 3 ‘major’ releases a year. We originally hoped for a month of release candidate cycles, but due to some issues found and the holidays + travel on my end it turned into 2 months.

My goal is to optimize our testing and planning to reduce this further, as this release cycle process is effectively an implicit ‘freeze’. Take a look at the number of open pull requests to see what I mean. For the next cycle I’ll also make the freeze explicit, and announce it.

Looking forward

While doing a release is great, my mind is already busy with the next steps. We have a bunch of things coming that are exciting to me.

Performance: my detection engine rewrite work has been tested by many already, and reports are quite positive. I’ve heard reports up to 25% increase, which is a great bonus considering the work was started to clean up this messy code.

ICS/SCADA: Jason Ish is finalizing a DNP3 parser that is very full featured, with detection, logging and lua support. Other protocols are also being developed.

Documentation: we’re in the process of moving our user docs from the wiki to sphinx. This means we’ll have versioned docs, nice pdf exports, etc. It’s already 180 pages!

Plus lots of other things. Keep an eye out on our mailing lists, bug tracker or IRC channel.


by inliniac at January 27, 2016 03:33 PM

suricata-ids.org

Suricata 3.0 Available!

suri-400x400We’re proud to announce Suricata 3.0. This is a major new release improving Suricata on many fronts.

Download

http://www.openinfosecfoundation.org/download/suricata-3.0.tar.gz

Features and Improvements

  • improved detection options, including multi-tenancy and xbits
  • performance and scalability much improved
  • much improved accuracy and robustness
  • Lua scripting capabilities expanded significantly
  • many output improvements, including much more JSON
  • NETMAP capture method support, especially interesting to FreeBSD users
  • SMTP inspection and file extraction

For a full list of features added, please see:
https://redmine.openinfosecfoundation.org/versions/80

Upgrading

Upgrades from 2.0 to 3.0 should be mostly seamless. Here are some notes:

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Upgrading_Suricata_20_to_Suricata_30

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

FireEye, ProtectWise, ANSSI, Emerging Threats /
Proofpoint, Stamus Networks, Ntop, AFL project, CoverityScan

Aaron Campbell, Aleksey Katargin, Alessandro Guido,
Alexander Gozman, Alexandre Macabies, Alfredo Cardigliano,
Andreas Moe, Anoop Saldanha, Antti Tönkyrä, Bill Meeks,
Darien Huss, David Abarbanel, David Cannings, David Diallo,
David Maciejak, Duarte Silva, Eduardo Arada, Giuseppe Longo,
Greg Siemon, Hayder Sinan, Helmut Schaa, Jason Ish,
Jeff Barber, Ken Steele, lessyv, Mark Webb-Johnson,
Mats Klepsland, Matt Carothers, Michael Rash, Nick Jones,
Pierre Chifflier, Ray Ruvinskiy, Samiux A, Schnaffon,
Stephen Donnelly, sxhlinux, Tom DeCanio, Torgeir Natvig,
Travis Green, Zachary Rasmor

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

November 9-11 we’ll be in Washington, DC, for our 2nd Suricata User Conference: http://oisfevents.net

If you need help installing, updating, validating and tuning Suricata we have a training program. Please see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

by inliniac at January 27, 2016 02:11 PM

December 21, 2015

suricata-ids.org

Suricata 3.0RC3 Available!

Photo by Eric Leblond

We’re happy to announce Suricata 3.0RC3. RC3 fixes a few issues in RC2 that require some more testing. The plan is to release the stable quickly after the holidays, so please help us test this release!

Fixes:

  • Bug #1632: Fail to download large file with browser
  • Bug #1634: Fix non thread safeness of Prelude analyzer
  • Bug #1640: drop log crashes
  • Bug #1645: Race condition in unix manager
  • Bug #1647: FlowGetKey flow-hash.c:240 segmentation fault (master)
  • Bug #1650: DER parsing issue (master)

Get the release here:

http://www.openinfosecfoundation.org/download/suricata-3.0RC3.tar.gz

Known issues & missing features

In a development release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at December 21, 2015 02:10 PM

Suricata 2.0.11 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.11. This release fixes a number of important issues in the 2.0 series.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.11.tar.gz

Changes

  • Bug #1572: 2.0.8 FlowGetKey flow-hash.c:240 segmentation fault (icmp destination unreachable)
  • Bug #1637: drop log crashes
  • Bug #1639: 2.0.x: Fix non thread safeness of Prelude analyzer
  • Bug #1649: DER parsing issue
  • Bug #1651: HTTP body tracking using excessive memory
  • Bug #1652: SMTP parsing issue (2.0.x)
  • Bug #1653: DNS over TCP parsing issue (2.0.x)
  • Bug #1654: TCP reassembly bug (2.0.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Mark Webb-Johnson
  • Nick Jones
  • Hayder Sinan
  • Samiux A

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at December 21, 2015 01:57 PM

December 08, 2015

suricata-ids.org

Suricata 3.0RC2 Available!

Photo by Eric Leblond

We’re happy to announce Suricata 3.0RC2. RC2 fixes a few issues in RC1 that require some more testing. The plan still is to release the stable within a few weeks, so please help us test this release!

Fixes:

  • Bug #1551: –enable-profiling-locks broken
  • Bug #1602: eve-log prefix field feature broken
  • Bug #1614: app_proto key missing from EVE file events
  • Bug #1615: disable modbus by default
  • Bug #1616: TCP reassembly bug
  • Bug #1617: DNS over TCP parsing issue
  • Bug #1618: SMTP parsing issue
  • Feature #1635: unified2 output: disable by default

Get the release here:

http://www.openinfosecfoundation.org/download/suricata-3.0RC2.tar.gz

Known issues & missing features

In a development release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at December 08, 2015 01:53 PM

November 25, 2015

suricata-ids.org

Suricata 3.0RC1 Available!

Photo by Eric Leblond

We’re happy to announce Suricata 3.0RC1. This release replaces 2.1beta4 as the new development release. The plan is to release the stable within a few weeks, so please help us test this release!

Lots of improvements:

  • Multi-tenancy for detection
  • Big email logging update by Eric Leblond
  • Work on Lua and JSON output for various protocols by Mats Klepsland
  • Redis output support by Eric Leblond
  • JSON output for stats, rules profiling
  • Colorized output on the commandline
  • Support for the base64_decode and base64_data keywords by Jason Ish
  • TLS and DNS lua support
  • file_data support for SMTP by Giuseppe Longo
  • Support wild cards in rule loading by Alexander Gozman

Packet capture got a lot of love:

  • PF_RING optimizations by Alfredo Cardigliano
  • Netmap updates by Aleksey Katargin
  • AF_PACKET updated by Eric Leblond
  • DAG fixes by Stephen Donnelly

Other than that, lots of cleanups and optimizations:

  • stateful detection overhaul
  • stream engine updates

Get the release here:

http://www.openinfosecfoundation.org/download/suricata-3.0RC1.tar.gz

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Alexander Gozman
  • Mats Klepsland
  • Giuseppe Longo
  • Alfredo Cardigliano
  • Aleksey Katargin
  • Alessandro Guido
  • Antti Tönkyrä
  • Tom DeCanio
  • Aaron Campbell
  • DIALLO David
  • David Cannings
  • Helmut Schaa
  • Jeff Barber
  • Schnaffon
  • Torgeir Natvig
  • Zachary Rasmor
  • Alexandre Macabies
  • Stephen Donnelly

Known issues & missing features

In a development release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at November 25, 2015 02:09 PM

Suricata 2.0.10 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.10. This release fixes a number of important issues in the 2.0 series.

A number of other issues were fixed. Upgrading is highly recommended.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.10.tar.gz

Changes

  • Bug #1596: dns parser issue reported & fixed by Aaron Campbell
  • Bug #1554: stored: false in files log when files were actually stored
  • Feature #1581: support LINKTYPE_NULL

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Aaron Campbell
  • Giuseppe Longo
  • Greg Siemon

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at November 25, 2015 09:44 AM

November 24, 2015

Victor Julien

New Suricata release model

suri-400x400As the team is back from a very successful week in Barcelona, I’d like to take a moment on what we discussed and decided on with regards to development.

One thing no one was happy with is how the release schedules are working. Releases were meant to reasonably frequent, but the time between major releases was growing longer and longer. The 2.0 branch for example, is closing in on 2 years as the stable branch. The result is that many people are missing out on many of the improvements we’ve been doing. Currently many people using Suricata actually use a beta version, of even our git master, in production!

What we’re going to try is time based releases. Pretty much releases will be more like snapshots of the development branch. We think this can work as our dev branch is more and more stable due to our extensive QA setup.

Of course, we’ll have to make sure we’re not going to merge super intrusive changes just before a release. We’ll likely get into some pattern of merge windows and (feature) freezes, but how this will exactly play out is something we’ll figure out as we go.

We’re going to try to shoot for 3 of such releases per year.

In our redmine ticket tracker, I’ve also created a new pseudo-version ‘Soon’. Things we think should be addressed for the next release, will be added there. But we’ll retarget the tickets when they are actually implemented.

Since it’s already almost 2 years since we’ve done 2.0, we think the next release warrants a larger jump in the versioning. So we’re going to call it 3.0. The first release candidate will likely be released this week hopefully followed by a stable in December.


by inliniac at November 24, 2015 03:54 PM

October 09, 2015

Victor Julien

Get paid to work on Suricata?

If you like fiddling with Suricata development, maybe you can get paid to do it.

Companies ask me regularly if I can recommend Suricata developers. I’m going to assemble a list of people who are interested in such work. If you like me to consider you in such cases, drop me an email.

If you really want me to *recommend* you, it’s important that I actually know you somewhat. So becoming a (volunteer) contributor will help a lot.

Things to mention in your email:
– interests
– github profile
– open source contributions
– social media, blog links
– availability, whether you’re a contractor or looking for a real J-O-B

Who knows, maybe something good will come from it!

Btw, if you’re not a dev but great at research, or deployments and tuning, I know some ppl that are always looking for such new hires as well!


by inliniac at October 09, 2015 01:59 PM

September 25, 2015

suricata-ids.org

Suricata 2.0.9 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.9. This release fixes a number of issues in the 2.0 series.

Couple of important fixes: defrag evasion, a crash when using certain rules (mixing regular content and relative bytejumps with dce option) and better detection of TCP retransmissions with different data.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.9.tar.gz

Changes

  • Bug#1558: stream: retransmission not detected (2.0.x)
  • Bug #1550: Segmentation Fault at detect-engine-content-inspection.c:438
  • Bug #1564: defrag: evasion issue
  • Bug #1431: stream: last_ack update issue leading to stream gaps (2.0.x)
  • Bug #1483: 2.0.x backport: Leading whitespace in flowbits variable names
  • Bug #1490: http_host payload validation erroring on uppercase PCRE metacharacters
  • Bug #1501: 2.0.x backport: Add HUP coverage to output json-log
  • Bug #1510: 2.0.x: address var parsing issue
  • Bug #1513: stream_size <= and >= modifiers function as < and > (equality is not functional) (2.0.x)
  • Update bundled libhtp to 0.5.18

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Jérémy Beaume
  • Erik Hjelmvik
  • Alessandro Guido
  • Alexandre Macabies
  • Darren Spruell
  • Jay MJ
  • Charles Smutz

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have a training coming up in Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at September 25, 2015 03:25 PM

August 30, 2015

Peter Manev

Suricata IDS/IPS - TCP segment pool size preallocation


In the default suricata.yaml stream section we have:
stream:
  memcap: 32mb
  checksum-validation: no      # reject wrong csums
  async-oneside: true
  midstream: true
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    #raw: yes
    #chunk-prealloc: 250
    #segments:
    #  - size: 4
    #    prealloc: 256
    #  - size: 16
    #    prealloc: 512
    #  - size: 112
    #    prealloc: 512
    #  - size: 248
    #    prealloc: 512
    #  - size: 512
    #    prealloc: 512
    #  - size: 768
    #    prealloc: 1024
    #  - size: 1448
    #    prealloc: 1024
    #  - size: 65535
    #    prealloc: 128


So what are these segment preallocations for?
Let's have a look. When Suricata exits (or kill -15 PidOfSuricata) it produces a lot of useful statistics in the suricata.log file (you can enable that from the suricata.yaml and use the "-v" switch (verbose) when starting Suricata):
The example below is for exit stats.
   
tail -20 StatsByDate/suricata-2014-06-01.log
[24344] 1/6/2014 -- 01:45:52 - (source-af-packet.c:1810) <Info> (ReceiveAFPThreadExitStats) -- (AFPacketeth314) Packets 7317661624, bytes 6132661347126
[24344] 1/6/2014 -- 01:45:52 - (stream-tcp.c:4643) <Info> (StreamTcpExitPrintStats) -- Stream TCP processed 3382528539 TCP packets
[24345] 1/6/2014 -- 01:45:52 - (source-af-packet.c:1807) <Info> (ReceiveAFPThreadExitStats) -- (AFPacketeth315) Kernel: Packets 8049357450, dropped 352658715
[24345] 1/6/2014 -- 01:45:52 - (source-af-packet.c:1810) <Info> (ReceiveAFPThreadExitStats) -- (AFPacketeth315) Packets 7696486934, bytes 6666577738944
[24345] 1/6/2014 -- 01:45:52 - (stream-tcp.c:4643) <Info> (StreamTcpExitPrintStats) -- Stream TCP processed 3357321803 TCP packets
[24346] 1/6/2014 -- 01:45:52 - (source-af-packet.c:1807) <Info> (ReceiveAFPThreadExitStats) -- (AFPacketeth316) Kernel: Packets 7573051188, dropped 292897219
[24346] 1/6/2014 -- 01:45:52 - (source-af-packet.c:1810) <Info> (ReceiveAFPThreadExitStats) -- (AFPacketeth316) Packets 7279948375, bytes 6046562324948
[24346] 1/6/2014 -- 01:45:52 - (stream-tcp.c:4643) <Info> (StreamTcpExitPrintStats) -- Stream TCP processed 3454330660 TCP packets
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 4 had a peak use of 60778 segments, more than the prealloc setting of 256
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 16 had a peak use of 314953 segments, more than the prealloc setting of 512
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 112 had a peak use of 113739 segments, more than the prealloc setting of 512
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 248 had a peak use of 17893 segments, more than the prealloc setting of 512
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 512 had a peak use of 31787 segments, more than the prealloc setting of 512
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 768 had a peak use of 30769 segments, more than the prealloc setting of 1024
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 1448 had a peak use of 89446 segments, more than the prealloc setting of 1024
[24329] 1/6/2014 -- 01:45:53 - (stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 65535 had a peak use of 81214 segments, more than the prealloc setting of 128
[24329] 1/6/2014 -- 01:45:53 - (stream.c:182) <Info> (StreamMsgQueuesDeinit) -- TCP segment chunk pool had a peak use of 20306 chunks, more than the prealloc setting of 250
[24329] 1/6/2014 -- 01:45:53 - (host.c:245) <Info> (HostPrintStats) -- host memory usage: 390144 bytes, maximum: 16777216
[24329] 1/6/2014 -- 01:45:55 - (detect.c:3890) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[24329] 1/6/2014 -- 01:45:55 - (util-device.c:185) <Notice> (LiveDeviceListClean) -- Stats for 'eth3':  pkts: 124068935209, drop: 5245430626 (4.23%), invalid chksum: 0


Notice all the "TCP segment pool" messages. This is the actual tcp segment pool reassembly stats for that period of time that Suricata was running. We could adjust accordingly in the suricata.yaml (as compared to the default settings above)
   
stream:
  memcap: 14gb
  checksum-validation: no      # reject wrong csums
  midstream: false
  prealloc-sessions: 375000
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 20gb
    depth: 12mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    raw: yes
    chunk-prealloc: 20556
    segments:
      - size: 4
        prealloc: 61034
      - size: 16
        prealloc: 315465
      - size: 112
        prealloc: 114251
      - size: 248
        prealloc: 18405
      - size: 512
        prealloc: 30769
      - size: 768
        prealloc: 31793
      - size: 1448
        prealloc: 90470
      - size: 65535
        prealloc: 81342
   


   
The total RAM (reserved) consumption for these preallocations (from the stream.reassembly.memcap value ) would be:

4*61034 + 16*315465 + 112*114251 + 248*18405 + 512*30769 + 768*31793 + 1448*90470 + 65535*81342 
= 5524571410 bytes
= 5.14 GB of RAM

So we could preallocate the tcp segments and take the Suricata tuning even a step further and improve performance as well.

So now when you start Suricata with the "-v" switch in your suricata.log with this specific set up described above you should see something like:
...
...
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 4, prealloc 61034
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 16, prealloc 315465
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 112, prealloc 114251
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 248, prealloc 18405
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 512, prealloc 30769
[30709] 1/6/2014 -- 12:17:34 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 768, prealloc 31793
[30709] 1/6/2014 -- 12:17:35 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 1448, prealloc 90470
[30709] 1/6/2014 -- 12:17:35 - (stream-tcp-reassemble.c:425) <Info> (StreamTcpReassemblyConfig) -- segment pool: pktsize 65535, prealloc 81342
[30709] 1/6/2014 -- 12:17:35 - (stream-tcp-reassemble.c:461) <Info> (StreamTcpReassemblyConfig) -- stream.reassembly "chunk-prealloc": 20556
...
...

NOTE:
Those 5.14 GB RAM in the example here will be preallocated (taken) from the stream.reassembly.memcap value. In other words it will not consume 5.14 GB of RAM more.

So be careful when setting up preallocation in order not to preallocate more of what you have.
In my case of 10Gbps suricata.yaml config I had:

stream:
  memcap: 14gb
  checksum-validation: no      # reject wrong csums
  midstream: false
  prealloc-sessions: 375000
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 20gb
    depth: 12mb                  # reassemble 1mb into a stream


What this helps with is that it lowers CPU usage/contention for TCP segment allocation during reassembly - it is already preallocated and Suricata just uses it instead of creating it everytime it needs it. It also helps minimize the initial drops during startup.

Highly adaptable and  flexible.








by Peter Manev (noreply@blogger.com) at August 30, 2015 09:41 AM

Suricata - preparing 10Gbps network cards for IDPS and file extraction


OS used/tested for this tutorial - Debian Wheezy and/or Ubuntu LTS 12.0.4
With 3.2.0 and 3.5.0 kernel level respectively with Suricata 2.0dev at the moment of this writing.



This article consists of the following major 3 sections:
  • Network card drivers and tuning
  • Kernel specific tunning
  • Suricata.yaml configuration  (file extraction specific)

Network and system  tools:
apt-get install ethtool bwm-ng iptraf htop

Network card drivers and tuning

Our card is Intel 82599EB 10-Gigabit SFI/SFP+


rmmod ixgbe
sudo modprobe ixgbe FdirPballoc=3
ifconfig eth3 up
then (we disable irqbalance and make sure it does not enable itself during reboot)
 killall irqbalance
 service irqbalance stop

 apt-get install chkconfig
 chkconfig irqbalance off
Get the Intel network driver form here (we will use them in a second) - https://downloadcenter.intel.com/default.aspx

Download to your directory of choice then unzip,compile and install:

wget http://sourceforge.net/projects/e1000/files/ixgbe%20stable/3.18.7/ixgbe-3.18.7.tar.gz
tar -zxf ixgbe-3.18.7.tar.gz      
cd /home/pevman/ixgbe-3.18.7/src      
make clean && make && make install

Set irq affinity - do not forget to change eth3  below with the name of the network interface you are using:
 cd ../scripts/
 ./set_irq_affinity  eth3


 You should see something like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ./set_irq_affinity  eth3
no rx vectors found on eth3
no tx vectors found on eth3
eth3 mask=1 for /proc/irq/101/smp_affinity
eth3 mask=2 for /proc/irq/102/smp_affinity
eth3 mask=4 for /proc/irq/103/smp_affinity
eth3 mask=8 for /proc/irq/104/smp_affinity
eth3 mask=10 for /proc/irq/105/smp_affinity
eth3 mask=20 for /proc/irq/106/smp_affinity
eth3 mask=40 for /proc/irq/107/smp_affinity
eth3 mask=80 for /proc/irq/108/smp_affinity
eth3 mask=100 for /proc/irq/109/smp_affinity
eth3 mask=200 for /proc/irq/110/smp_affinity
eth3 mask=400 for /proc/irq/111/smp_affinity
eth3 mask=800 for /proc/irq/112/smp_affinity
eth3 mask=1000 for /proc/irq/113/smp_affinity
eth3 mask=2000 for /proc/irq/114/smp_affinity
eth3 mask=4000 for /proc/irq/115/smp_affinity
eth3 mask=8000 for /proc/irq/116/smp_affinity
root@suricata:/home/pevman/ixgbe-3.18.7/scripts#
Now we have the latest drivers installed (at the time of this writing) and we have run the affinity script:
   *-network:1
       description: Ethernet interface
       product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:04:00.1
       logical name: eth3
       version: 01
       serial: 00:e0:ed:19:e3:e1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list ethernet physical fibre
       configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=3.18.7 duplex=full firmware=0x800000cb latency=0 link=yes multicast=yes port=fibre promiscuous=yes
       resources: irq:37 memory:fbc00000-fbc1ffff ioport:e000(size=32) memory:fbc40000-fbc43fff memory:fa700000-fa7fffff memory:fa600000-fa6fffff



We need to disable all offloading on the network card in order for the IDS to be able to see the traffic as it is supposed to be (without checksums,tcp-segmentation-offloading and such..) Otherwise your IDPS would not be able to see all "natural" network traffic the way it is supposed to and will not inspect it properly.

This would influence the correctness of ALL outputs including file extraction. So make sure all offloading features are OFF !!!

When you first install the drivers and card your offloading settings might look like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
root@suricata:/home/pevman/ixgbe-3.18.7/scripts#

So we disable all of them, like so (and we load balance the UDP flows for that particular network card):

ethtool -K eth3 tso off
ethtool -K eth3 gro off
ethtool -K eth3 ufo off
ethtool -K eth3 lro off
ethtool -K eth3 gso off
ethtool -K eth3 rx off
ethtool -K eth3 tx off
ethtool -K eth3 sg off
ethtool -K eth3 rxvlan off
ethtool -K eth3 txvlan off
ethtool -N eth3 rx-flow-hash udp4 sdfn
ethtool -N eth3 rx-flow-hash udp6 sdfn
ethtool -C eth3 rx-usecs 1 rx-frames 0
ethtool -C eth3 adaptive-rx off

Your output should look something like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 tso off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 gro off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 lro off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 gso off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 rx off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 tx off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 sg off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 rxvlan off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 txvlan off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -N eth3 rx-flow-hash udp4 sdfn
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -N eth3 rx-flow-hash udp6 sdfn
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -n eth3 rx-flow-hash udp6
UDP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -n eth3 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -C eth3 rx-usecs 0 rx-frames 0
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -C eth3 adaptive-rx off

Now we doublecheck and run ethtool again to verify that the offloading is OFF:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off

Ring parameters on the network card:

root@suricata:~# ethtool -g eth3
Ring parameters for eth3:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:            4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512


We can increase that to the max Pre-set RX:

root@suricata:~# ethtool -G eth3 rx 4096

Then we  have a look again:

root@suricata:~# ethtool -g eth3
Ring parameters for eth3:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             512

Making network changes permanent across reboots


On Ubuntu for example you can do:
root@suricata:~# crontab -e

Add the following:
# add cronjob at reboot - disbale network offload
@reboot /opt/tmp/disable-network-offload.sh

and your disable-network-offload.sh script (in this case under /opt/tmp/ ) will contain the following:



#!/bin/bash
ethtool -K eth3 tso off
ethtool -K eth3 gro off
ethtool -K eth3 ufo off
ethtool -K eth3 lro off
ethtool -K eth3 gso off
ethtool -K eth3 rx off
ethtool -K eth3 tx off
ethtool -K eth3 sg off
ethtool -K eth3 rxvlan off
ethtool -K eth3 txvlan off
ethtool -N eth3 rx-flow-hash udp4 sdfn
ethtool -N eth3 rx-flow-hash udp6 sdfn
ethtool -C eth3 rx-usecs 1 rx-frames 0
ethtool -C eth3 adaptive-rx off


with:
chmod 755 disable-network-offload.sh
To make sure you have the ixgbe module always loaded at boot time you can add "ixgbe" to the  /etc/modules file.

Kernel specific tunning


Certain adjustments in parameters in the kernel can help as well :

sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.optmem_max=16777216


Making kernel changes permanent across reboots


example:
echo 'net.core.netdev_max_backlog=250000' >> /etc/sysctl.conf

reload the changes:
sysctl -p

OR for all the above adjustments:

echo 'net.core.netdev_max_backlog=250000' >> /etc/sysctl.conf
echo 'net.core.rmem_max=16777216' >> /etc/sysctl.conf
echo 'net.core.rmem_default=16777216' >> /etc/sysctl.conf
echo 'net.core.optmem_max=16777216' >> /etc/sysctl.conf
sysctl -p


Suricata.yaml configuration  (file extraction specific)

As of Suricata 1.2  - it is possible to detect and extract/store over 5000 types of files from HTTP sessions.

Specific file extraction instructions can also be found in the official page documentation.

The following libraries are needed on the system running Suricata :
apt-get install libnss3-dev libnspr4-dev

Suricata also needs to be compiled with file extraction enabled (not covered here).

In short in the suriacta.yaml, those are the sections below that can be tuned/configured and affect the file extraction and logging:
(the bigger the mem values the better on a busy link )


  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh


For file store to disk/extraction:
   - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      force-md5: yes     # force logging of md5 checksums
      #waldo: file.waldo # waldo file to store the file_id across runs


 stream:
  memcap: 32mb
  checksum-validation: no      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 128mb
    depth: 1mb                  # reassemble 1mb into a stream
  
depth: 1mb , would mean that in one tcp reassembled flow, the max size of the file that can be extracted is just about 1mb.

Both stream.memcap and reassembly.memcap (if reassembly is needed) must be big enough to accommodate the whole file on the fly in traffic that needs to be extracted PLUS any other stream and reassembly tasks that the engine needs to do while inspecting the traffic on a particular link.

 app-layer:
  protocols:
....
....
     http:
      enabled: yes
      # memcap: 64mb

The default limit for mem usage for http is 64mb   , that could be increased , ex - memcap: 4GB -  since HTTP is present everywhere and a low memcap on a busy HTTP link would limit the inspection and extraction size ability.

       libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 3072
           response-body-limit: 3072

The default values above control how far the HTTP request and response body is tracked and also limit file inspection. This should be set to a much higher value:

        libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 1gb
           response-body-limit: 1gb

 or 0 (which would mean unlimited):

       libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 0
           response-body-limit: 0

and then of course you would need a rule loaded(example):
alert http any any -> any any (msg:"PDF file Extracted"; filemagic:"PDF document"; filestore; sid:11; rev:11;)



That's it.
























by Peter Manev (noreply@blogger.com) at August 30, 2015 06:23 AM

August 29, 2015

Peter Manev

Failed to open ethX: pfring_open error


This is a blogpost about getting around the following error when using Suricata with pfring:

(source-pfring.c:444) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open eth2: pfring_open error. Check if eth2 exists and pf_ring module is loaded.
(tmqh-packetpool.c:394) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 230679680
pfring_set_channel_id() failed: -1

However in my case eth2 existed, was up and running and the pfring module was loaded. So what happened in a bit more detail below :

I experienced this after a git pull update/upgrade of Suricata (latest at the moment of this writing) and after I re compiled pfring (using the latest pfring from git (https://github.com/ntop/PF_RING.git).

My set up (linux Debian/Ubuntu like systems):

root@suricata:/var/data/log/suricata# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 00:e0:ed:19:e3:e0
          inet6 addr: fe80::2e0:edff:fe19:e3e0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2962266192 errors:0 dropped:5527381 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2867936692537 (2.8 TB)  TX bytes:3345 (3.3 KB)
The pfring set up I had was configured like this below:
root@suricata:/var/data/log/suricata# modprobe pf_ring transparent_mode=0 min_num_slots=65534

A regular check reveals nothing abnormal:
root@suricata:/var/data/log/suricata# modinfo pf_ring && cat /proc/net/pf_ring/info
filename:       /lib/modules/3.14.0-031400-generic/kernel/net/pf_ring/pf_ring.ko
alias:          net-pf-27
description:    Packet capture acceleration and analysis
author:         ntop.org
license:        GPL
srcversion:     E344EB01757B55E97A93D0C
depends:     
vermagic:       3.14.0-031400-generic SMP mod_unload modversions
parm:           min_num_slots:Min number of ring slots (uint)
parm:           perfect_rules_hash_size:Perfect rules hash size (uint)
parm:           transparent_mode:(deprecated) (uint)
parm:           enable_debug:Set to 1 to enable PF_RING debug tracing into the syslog (uint)
parm:           enable_tx_capture:Set to 1 to capture outgoing packets (uint)
parm:           enable_frag_coherence:Set to 1 to handle fragments (flow coherence) in clusters (uint)
parm:           enable_ip_defrag:Set to 1 to enable IP defragmentation(only rx traffic is defragmentead) (uint)
parm:           quick_mode:Set to 1 to run at full speed but with upto one socket per interface (uint)
PF_RING Version          : 6.1.1 (dev:250a67fe1082121ac511a19ebc3fe1fc5f494bfe)
Total rings              : 0

Standard (non DNA/ZC) Options
Ring slots               : 65534
Slot version             : 16
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0
Suricata and pfring have been installed as explained here - on the Suricata redmine wiki.
root@suricata:~# ldd /usr/local/bin/suricata
    linux-vdso.so.1 =>  (0x00007fff419fe000)
    libhtp-0.5.17.so.1 => /usr/local/lib/libhtp-0.5.17.so.1 (0x00007f32af5a1000)
    libGeoIP.so.1 => /usr/lib/x86_64-linux-gnu/libGeoIP.so.1 (0x00007f32af372000)
    libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x00007f32af103000)
    libmagic.so.1 => /usr/lib/x86_64-linux-gnu/libmagic.so.1 (0x00007f32aeee7000)
    libcap-ng.so.0 => /usr/local/lib/libcap-ng.so.0 (0x00007f32aece2000)
    libpfring.so => /usr/local/lib/libpfring.so (0x00007f32aeaa3000)
    libpcap.so.1 => /usr/local/pfring/lib/libpcap.so.1 (0x00007f32ae80e000)
    libnet.so.1 => /usr/lib/x86_64-linux-gnu/libnet.so.1 (0x00007f32ae5f5000)
    libjansson.so.4 => /usr/lib/x86_64-linux-gnu/libjansson.so.4 (0x00007f32ae3e8000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f32ae1ca000)
    libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f32adfaa000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f32add6b000)
    libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007f32ada31000)
    libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so (0x00007f32ad7f4000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f32ad42e000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f32ad215000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f32acf0f000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f32acd0a000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f32acaf4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f32af7d4000)
    libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f32ac8e9000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f32ac6e0000)
    libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so (0x00007f32ac4b5000)
    libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so (0x00007f32ac2b0000)
    libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so (0x00007f32ac0ab000)


Further more my Suricata start line was like this:

suricata --pfring-int=eth2 --pfring-cluster-id=99 --pfring-cluster-type=cluster_flow -c /etc/suricata/peter-yaml/suricata-pfring.yaml --pidfile /var/run/suricata.pid -v

Even though everything seems fine - I could  not start Suricata with pfring:

[31591] 5/8/2015 -- 17:10:31 - (tmqh-packetpool.c:394) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 230679680
pfring_set_channel_id() failed: -1
[31591] 5/8/2015 -- 17:10:31 - (source-pfring.c:444) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open eth2: pfring_open error. Check if eth2 exists and pf_ring module is loaded.
[31592] 5/8/2015 -- 17:10:31 - (tmqh-packetpool.c:394) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 230679680
pfring_set_channel_id() failed: -1
[31592] 5/8/2015 -- 17:10:31 - (source-pfring.c:444) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open eth2: pfring_open error. Check if eth2 exists and pf_ring module is loaded.
[31593] 5/8/2015 -- 17:10:32 - (tmqh-packetpool.c:394) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 230679680
pfring_set_channel_id() failed: -1
[31593] 5/8/2015 -- 17:10:32 - (source-pfring.c:444) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open eth2: pfring_open error. Check if eth2 exists and pf_ring module is loaded.
[31594] 5/8/2015 -- 17:10:32 - (tmqh-packetpool.c:394) <Info> (PacketPoolInit) -- preallocated 65534 packets. Total memory 230679680
pfring_set_channel_id() failed: -1
[31594] 5/8/2015 -- 17:10:32 - (source-pfring.c:444) <Error> (ReceivePfringThreadInit) -- [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - Failed to open eth2: pfring_open error. Check if eth2 exists and pf_ring module is loaded.
....

I was getting that error even though I reloaded the pfring module:
rmmod pr_ring
modprobe pf_ring transparent_mode=0 min_num_slots=65534
the way I usually do...

In short - this is the fix:

LD_LIBRARY_PATH=/usr/local/pfring/lib suricata --pfring-int=eth2  --pfring-cluster-id=99 --pfring-cluster-type=cluster_flow  -c /etc/suricata/peter-yaml/suricata-pfring.yaml --pidfile /var/run/suricata.pid -v

Notice the use of:
LD_LIBRARY_PATH=/usr/local/pfring/lib suricata 

More information about what is LD_LIBRARY_PATH     

To get rid of LD_LIBRARY_PATH you can create a pfring.conf file in /etc/ld.so.conf.d/ containing:
/usr/local/pfring/lib
and run
sudo ldconfig




by Peter Manev (noreply@blogger.com) at August 29, 2015 02:42 AM

June 08, 2015

Security Onion

New securityonion-suricata package resolves an issue

We recently released a securityonion-suricata package for Suricata 2.0.8:
http://blog.securityonion.net/2015/05/suricata-208.html

An issue was found in the packaging:
https://groups.google.com/d/topic/security-onion/1MmmmO2XOyc/discussion

I've updated the securityonion-suricata package to resolve this issue.

The new package version is:
securityonion-suricata - 2.0.8-0ubuntu0securityonion2

Issues Resolved

Issue 742: securityonion-suricata package missing debian/install
https://github.com/Security-Onion-Solutions/security-onion/issues/742

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists

Commercial Support
Need commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://github.com/Security-Onion-Solutions/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at June 08, 2015 10:28 AM

May 28, 2015

suricata-ids.org

Suricata Ubuntu PPA updated to 2.1beta4

We have updated the official Ubuntu PPA to Suricata 2.1beta4. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at May 28, 2015 01:02 PM

Suricata 2.1beta4 Windows Installer Available

The Windows MSI installer of the Suricata 2.1beta4 release is now available.

Download it here: suricata-2.1beta4-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at May 28, 2015 12:57 PM

Suricata Ubuntu PPA updated to 2.0.8

We have updated the official Ubuntu PPA to Suricata 2.0.8. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at May 28, 2015 12:23 PM

Suricata 2.0.8 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.8 release is now available.

Download it here: Suricata-2.0.8-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at May 28, 2015 12:19 PM

May 22, 2015

Peter Manev

Suricata - wildcard rule loading


Recently (few hrs ago as of writing this blog) there was  a new feature (thanks to gozzy) introduced in Suricata IDS/IPS/NSM  - wildcard rule loading capability.

As of the moment the feature is available in our git master. If you are wondering how to get that up and running or do not have the latest Suricata from git master - here is a quick tutorial (Debian/Ubuntu):

1)
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev build-essential \
autoconf automake libtool libpcap-dev libnet1-dev libyaml-0-2 \
libyaml-dev zlib1g zlib1g-dev libmagic-dev libcap-ng-dev \
libjansson-dev pkg-config libnss3-dev libnspr4-dev git-core

2)
git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

3)
 ./autogen.sh && \
 ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ \
 --enable-geoip --enable-unix-socket \
 --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ \
 --with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr \
 && make clean && make && make install-full && ldconfig

To confirm -
suricata --build-info

Now that you have latest Suricta up and running - here it is what this blog post is all about  - wildcard rule loading for Suricata IDPS. Some possible scenarios of use are loading wildcarded rules form the :

Command line


Please note the "quotes" !
suricata -c /etc/suricata/suricata.yaml  -v -i eth0 -S "/etc/suricata/rules/*.rules"

Pretty self explanatory. The command above will load all .rules files from /etc/suricata/rules/
suricata -c /etc/suricata/suricata.yaml  -v -i eth0 -S "/etc/suricata/rules/emerging*"
The command above will load all emerging* rules files from /etc/suricata/rules/

Config file


You can also set that up in the suricata.yaml config file. Here is how (please note the "quotes").

In your rules section in the suricata.yaml:

# Set the default rule path here to search for the files.
# if not set, it will look at the current working dir
default-rule-path: /etc/suricata/rules
rule-files:
 #- "*.rules"
 - "emerging*"
 #- botcc.rules
 #- ciarmy.rules
 #- compromised.rules
 #- drop.rules
 #- dshield.rules
 #- emerging-activex.rules
 #- emerging-attack_response.rules
The set up above will load all emerging* files and the rules residing in those. Then you can start Suricata anyway you would like, examples:

suricata -c /etc/suricata/suricata.yaml  -v -i eth0
suricata -c /etc/suricata/suricata.yaml  -v --af-packet=eth0

 and in suricata.log you should see all emerging* rule files being loaded:

......
[13558] 22/5/2015 -- 17:19:39 - (reputation.c:620) <Info> (SRepInit) -- IP reputation disabled
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-activex.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-attack_response.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-chat.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-current_events.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-deleted.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:420) <Warning> (ProcessSigFiles) -- [ERRCODE: SC_ERR_NO_RULES(42)] - No rules loaded from /etc/suricata/rules/emerging-deleted.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-dns.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-dos.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-exploit.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-ftp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-games.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-icmp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:420) <Warning> (ProcessSigFiles) -- [ERRCODE: SC_ERR_NO_RULES(42)] - No rules loaded from /etc/suricata/rules/emerging-icmp.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-icmp_info.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-imap.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-inappropriate.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-info.rules
[13558] 22/5/2015 -- 17:19:39 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-malware.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-misc.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-mobile_malware.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-netbios.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-p2p.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-policy.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-pop3.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-rpc.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-scada.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-scan.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-shellcode.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-smtp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-snmp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-sql.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-telnet.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-tftp.rules
[13558] 22/5/2015 -- 17:19:40 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-trojan.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-user_agents.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-voip.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_client.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_server.rules
[13558] 22/5/2015 -- 17:19:41 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-web_specific_apps.rules
[13558] 22/5/2015 -- 17:19:43 - (detect.c:410) <Info> (ProcessSigFiles) -- Loading rule file: /etc/suricata/rules/emerging-worm.rules
.......

You can also use it like that :

# Set the default rule path here to search for the files.
# if not set, it will look at the current working dir
default-rule-path: /etc/suricata/rules
rule-files:
 #- "*.rules"
 - "*web*"
 #- "emerging*"
 #- botcc.rules
 #- ciarmy.rules
 #- compromised.rules
 #- drop.rules


That's it.


by Peter Manev (noreply@blogger.com) at May 22, 2015 09:38 AM

May 21, 2015

Peter Manev

Suricata - multiple interface configuration with af-packet



Suricata is a very flexible and powerful multithreading  IDS/IPS/NSM.

Here is a simple tutorial (tested on Debian/Ubuntu) of how to configure multiple interfaces for af-packet mode with Suricata (af-packet mode works by default/out of the box on kernels 3.2 and above). Lets say you would like to start simple IDSing with Suricata on eth1, eth2 and eth3 on a particular machine/server.


In your suricata.yaml config (usually located in /etc/suricata/) find the af-packet section and do the following:


af-packet:
  - interface: eth2
    threads: 16
    cluster-id: 98
    cluster-type: cluster_cpu
    defrag: no
    use-mmap: yes
    ring-size: 200000
    checksum-checks: kernel
  - interface: eth1
    threads: 2
    cluster-id: 97
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    ring-size: 30000
  - interface: eth3
    threads: 2
    cluster-id: 96
    cluster-type: cluster_flow
    defrag: no
    use-mmap: yes
    ring-size: 20000
Of course feel free to adjust the ring-sizes (packet buffers) as you see fit for your particular set up.
NOTE:  do not forget to use a different cluster-id

so now you can start suricata like so:

suricata -c /etc/suricata/suricata.yaml -v --af-packet 

That above will start Suricata which will listen on eth2 with 16 threads with cluster_type: cluster_cpu and on eth1,eth3 with 2 threads each with cluster_type: cluster_flow. Have a look in your suricata.log file for more info.

If you would like to just test and see how it goes for eth2 only:
suricata -c /etc/suricata/suricata.yaml -v --af-packet=eth2

...easy and flexible.







by Peter Manev (noreply@blogger.com) at May 21, 2015 11:51 AM

May 18, 2015

Security Onion

Suricata 2.0.8

Suricata 2.0.8 was recently released:
http://suricata-ids.org/2015/05/06/suricata-2-0-8-available/

I've packaged Suricata 2.0.8 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.8-0ubuntu0securityonion1

Issues Resolved

Issue 725: Suricata 2.0.8
https://github.com/Security-Onion-Solutions/security-onion/issues/725

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak and migrate the HOME_NET and EXTERNAL_NET variables.  You'll then need to do the following:


  • re-apply any other local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:
    sudo rule-update


Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists

Training
Need training?  We have 3-hour online classes this week:
http://securityonionsolutions.com

Commercial Support
Need commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://github.com/Security-Onion-Solutions/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at May 18, 2015 12:20 PM

May 09, 2015

Security Onion

Testers Needed!

We have lots of new packages queued up for release, but we need your help testing them!

If you're not already a member of our security-onion-testing group, please join and then click the following links for testing guidelines.

ELSA rev1205
https://groups.google.com/d/topic/security-onion-testing/OHhNEapIUgE/discussion

Suricata 2.0.8
https://groups.google.com/d/topic/security-onion-testing/WKeR1RViDlc/discussion

ossec_agent
https://groups.google.com/d/topic/security-onion-testing/N5gpeSHmIlk/discussion

Sphinxsearch 2.1.9
https://groups.google.com/d/topic/security-onion-testing/VWjichsRqPw/discussion

NSM
https://groups.google.com/d/topic/security-onion-testing/-cbA8FgH7lg/discussion

Setup
https://groups.google.com/d/topic/security-onion-testing/PBY2wJH9ruo/discussion

As you test each package, please add your test results to the thread.

Thanks in advance for your time and effort!

by Doug Burks (noreply@blogger.com) at May 09, 2015 10:04 AM

May 08, 2015

Open Information Security Foundation

Suricata 2.1beta4 Available!

The OISF development team is proud to announce Suricata 2.1beta4. This is the fourth beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta4.tar.gz

New Features

  • Feature #1448: xbits support
  • Feature #336: Add support for NETMAP to Suricata
  • Feature #885: smtp file_data support
  • Feature #1394: Improve TCP reuse support
  • Feature #1445: Suricata does not work on pfSense/FreeBSD interfaces using PPPoE
  • Feature #1447: Ability to reject ICMP traffic
  • Feature #1410: add alerts to EVE’s drop logs

Improvements

  • Optimization #1014: app layer reassembly fast-path
  • Optimization #1377: flow manager: reduce (try)locking
  • Optimization #1403: autofp packet pool performance problems
  • Optimization #1409: http pipeline support for stateful detection
  • Bug #1314: http-events performance issues

Bugs

  • Bug #1340: null ptr dereference in Suricata v2.1beta2
  • Bug #1352: file list is not cleaned up
  • Bug #1358: Gradual memory leak using reload (kill -USR2 $pid)
  • Bug #1366: Crash if default_packet_size is below 32 bytes
  • Bug #1378: stats api doesn’t call thread deinit funcs
  • Bug #1384: tcp midstream window issue (master)
  • Bug #1388: pcap-file hangs on systems w/o atomics support (master)
  • Bug #1392: http uri parsing issue (master)
  • Bug #1393: CentOS 5.11 build failures
  • Bug #1398: DCERPC traffic parsing issue (master)
  • Bug #1401: inverted matching on incomplete session
  • Bug #1402: When re-opening files on HUP (rotation) always use the append flag.
  • Bug #1417: no rules loaded – latest git – rev e250040
  • Bug #1425: dead lock in de_state vs flowints/flowvars
  • Bug #1426: Files prematurely truncated by detection engine even though force-md5 is enabled
  • Bug #1429: stream: last_ack update issue leading to stream gaps
  • Bug #1435: EVE-Log alert payload option loses data
  • Bug #1441: Local timestamps in json events
  • Bug #1446: Unit ID check in Modbus packet error
  • Bug #1449: smtp parsing issue
  • Bug #1451: Fix list-keywords regressions
  • Bug #1463: modbus parsing issue

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Kostya Kortchinsky of the Google Security Team
  • the Yahoo Pentest Team
  • Giuseppe Longo
  • Alexander Gozman
  • Ken Steele
  • Andreas Moe
  • David Diallo
  • David Cannings
  • David Maciejak
  • Pierre Chifflier
  • Tom DeCanio
  • Zachary Rasmor
  • Aleksey Katargin
  • FireEye
  • ANSSI
  • Emerging Threats
  • AFL project
  • Coverity Scan
  • Travis Green
  • Darien Huss
  • Greg Siemon
  • Alessandro Guido
  • Antti Tönkyrä
  • Ray Ruvinskiy
  • Eduardo Arada
  • Michael Rash

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at May 08, 2015 03:25 PM

suricata-ids.org

Suricata 2.1beta4 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.1beta4. This is the fourth beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta4.tar.gz

New Features

  • Feature #1448: xbits support
  • Feature #336: Add support for NETMAP to Suricata
  • Feature #885: smtp file_data support
  • Feature #1394: Improve TCP reuse support
  • Feature #1445: Suricata does not work on pfSense/FreeBSD interfaces using PPPoE
  • Feature #1447: Ability to reject ICMP traffic
  • Feature #1410: add alerts to EVE’s drop logs

Improvements

  • Optimization #1014: app layer reassembly fast-path
  • Optimization #1377: flow manager: reduce (try)locking
  • Optimization #1403: autofp packet pool performance problems
  • Optimization #1409: http pipeline support for stateful detection
  • Bug #1314: http-events performance issues

Bugs

  • Bug #1340: null ptr dereference in Suricata v2.1beta2
  • Bug #1352: file list is not cleaned up
  • Bug #1358: Gradual memory leak using reload (kill -USR2 $pid)
  • Bug #1366: Crash if default_packet_size is below 32 bytes
  • Bug #1378: stats api doesn’t call thread deinit funcs
  • Bug #1384: tcp midstream window issue (master)
  • Bug #1388: pcap-file hangs on systems w/o atomics support (master)
  • Bug #1392: http uri parsing issue (master)
  • Bug #1393: CentOS 5.11 build failures
  • Bug #1398: DCERPC traffic parsing issue (master)
  • Bug #1401: inverted matching on incomplete session
  • Bug #1402: When re-opening files on HUP (rotation) always use the append flag.
  • Bug #1417: no rules loaded – latest git – rev e250040
  • Bug #1425: dead lock in de_state vs flowints/flowvars
  • Bug #1426: Files prematurely truncated by detection engine even though force-md5 is enabled
  • Bug #1429: stream: last_ack update issue leading to stream gaps
  • Bug #1435: EVE-Log alert payload option loses data
  • Bug #1441: Local timestamps in json events
  • Bug #1446: Unit ID check in Modbus packet error
  • Bug #1449: smtp parsing issue
  • Bug #1451: Fix list-keywords regressions
  • Bug #1463: modbus parsing issue

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Kostya Kortchinsky of the Google Security Team
  • the Yahoo Pentest Team
  • Giuseppe Longo
  • Alexander Gozman
  • Ken Steele
  • Andreas Moe
  • David Diallo
  • David Cannings
  • David Maciejak
  • Pierre Chifflier
  • Tom DeCanio
  • Zachary Rasmor
  • Aleksey Katargin
  • FireEye
  • ANSSI
  • Emerging Threats
  • AFL project
  • Coverity Scan
  • Travis Green
  • Darien Huss
  • Greg Siemon
  • Alessandro Guido
  • Antti Tönkyrä
  • Ray Ruvinskiy
  • Eduardo Arada
  • Michael Rash

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at May 08, 2015 02:52 PM

May 06, 2015

Open Information Security Foundation

Suricata 2.0.8 Available!

The OISF development team is pleased to announce Suricata 2.0.8. This release fixes a number of important issues in the 2.0 series.

The most important issue is a bug in the DER parser which is used to decode SSL/TLS certificates could crash Suricata. This issue was reported by Kostya Kortchinsky of the Google Security Team and was fixed by Pierre Chifflier of ANSSI.

Those processing large numbers of (untrusted) pcap files need to update as a malformed pcap could crash Suricata. Again, credits go to Kostya Kortchinsky.

A number of other issues were fixed. Upgrading is highly recommended.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.8.tar.gz

We have a new release key (the previous expired): http://www.openinfosecfoundation.org/download/OISF.pub (00C1B70D)

Changes

  • Bug #1450: tls parsing issue
  • Bug #1460: pcap parsing issue
  • Bug #1461: potential deadlock
  • Bug #1404: Alert-Debuglog not being rotated on SIGHUP
  • Bug #1420: inverted matching on incomplete session
  • Bug #1462: various issues in rule and yaml parsing

Security

The TLS/DER parsing issue has CVE-2015-0971 assigned to it.

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Kostya Kortchinsky of the Google Security Team
  • Pierre Chifflier of ANSSI
  • Sundar Jeyaraman of FireEye
  • Darien Huss — Emerging Threats
  • Alexander Gozman
  • AFL project
  • Coverity Scan

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at May 06, 2015 05:00 PM

suricata-ids.org

Suricata 2.0.8 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.8. This release fixes a number of important issues in the 2.0 series.

The most important issue is a bug in the DER parser which is used to decode SSL/TLS certificates could crash Suricata. This issue was reported by Kostya Kortchinsky of the Google Security Team and was fixed by Pierre Chifflier of ANSSI.

Those processing large numbers of (untrusted) pcap files need to update as a malformed pcap could crash Suricata. Again, credits go to Kostya Kortchinsky.

A number of other issues were fixed. Upgrading is highly recommended.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.8.tar.gz

We have a new release key (the previous expired): http://www.openinfosecfoundation.org/download/OISF.pub (00C1B70D)

Changes

  • Bug #1450: tls parsing issue
  • Bug #1460: pcap parsing issue
  • Bug #1461: potential deadlock
  • Bug #1404: Alert-Debuglog not being rotated on SIGHUP
  • Bug #1420: inverted matching on incomplete session
  • Bug #1462: various issues in rule and yaml parsing

Security

The TLS/DER parsing issue has CVE-2015-0971 assigned to it.

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Kostya Kortchinsky of the Google Security Team
  • Pierre Chifflier of ANSSI
  • Sundar Jeyaraman of FireEye
  • Darien Huss — Emerging Threats
  • Alexander Gozman
  • AFL project
  • Coverity Scan

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

Training & Support

Need help installing, updating, validating and tuning Suricata? We have trainings coming up. Paris in July, Barcelona in November: see http://suricata-ids.org/training/

For support options also see http://suricata-ids.org/support/

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at May 06, 2015 04:56 PM

April 25, 2015

Peter Manev

Suricata - check loaded yaml config settings with --dump-config



There is a very useful command available to Suricata IDS/IPS/NSM :
suricata --dump-config

The command above will dump all the config parameters and their respective values that are loaded by Suricata from the config file. You can run the command in any case - it does not matter if Suricata is running or not.

There is a peculiarity however. Sometimes people would think that the command(above) would dump the currently loaded config values by Suricata.... in some case it will and in some cases it will not.

So what does it depend on?.... simple:
suricata --dump-config

will dump the config settings that are loaded (or will be loaded) by Suricata by default from
/etc/suricata/suricata.yaml

So if you are running Suricata with a config file called suricata-test.yaml (or suricata.yaml located in a different directory) - you will not see those settings...unless you specify that config file in particular:
suricata --dump-config -c /etc/suricata/suricata-test.yaml
Here is a real case example.
I run Suricata for a specific test where I had specified the defrag memcap to be 512mb :
defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535 # number of defragmented flows to follow
  max-frags: 65535 # number of fragments to keep (higher than trackers)
  prealloc: yes
  timeout: 60

Suricata up and running:
root@LTS-64-1:~/Work # ps aux |grep suricata
root      8109  2.3  7.6 878444 308372 pts/6   Sl+  12:45   1:02 suricata -c /etc/suricata/suricata-test.yaml --af-packet=eth0 -v
root@LTS-64-1:~/Work #

And the peculiarity that this blogpost is trying to emphasize on about :
root@LTS-64-1:~/Work # suricata --dump-config  |grep defrag.memcap
defrag.memcap = 32mb
root@LTS-64-1:~/Work # suricata --dump-config -c /etc/suricata/suricata-test.yaml |grep defrag.memcap
defrag.memcap = 512mb
root@LTS-64-1:~/Work #



suricata --dump-config dumps the settings loaded(or to be loaded) from the default location /etc/suricata/suricata.yaml if you are running suricata with a yaml config with a different name than the default or with a different location that the default - in order to get those settings - you need to specify that particular yaml location, like so:

suricata --dump-config -c /etc/local/some_test_dir/suricata/suricata-test.yaml


Thanks

related article:
http://pevma.blogspot.se/2014/02/suricata-override-config-parameters-on.html


by Peter Manev (noreply@blogger.com) at April 25, 2015 01:46 AM

April 21, 2015

suricata-ids.org

New OISF Board Announced!

The Open Information Security Foundation is thrilled to announce the new OISF Board of Directors:

Welcoming back returning board members: Randy Caldejon, Alexandre Dulaunoy, and Ken Steele.

And join us in welcoming new board members: Leonard Jacobs, Christian Kreibich, Eric Meyer, and Amar Rathore.

This is an important time for OISF, Suricata, and open source technologies — we are excited to be working with such an esteemed board over the next 2 years.  Thanks goes out to the OISF community for casting their votes!

by kmisata at April 21, 2015 02:18 PM

April 20, 2015

suricata-ids.org

Announcing the first Suricata User Conference in Barcelona

oisf-barcelonaWe are excited to announce our first annual OISF / Suricata User Conference happening this November in Barcelona, Spain!

Join us for an exciting two days of Suricata and IDS/IPS development talks, brainstorming sessions, and amazing speakers. You can also attend a 2-day Suricata training event prior to the conference to make it a full-week of learning!

The conference is FREE and open to the public – however, we ask that people register via our events website so we can ensure to accommodate everyone in regards to space and lunches. Visit: http://www.oisfevents.net.

NOTE: the 2-day Suricata training during this week is a paid event and space will fill up – so register at https://suricata-2day-barcelona.eventbrite.com

We look forward to seeing you there! As always if you have questions, contact us at info@oisf.net.

The OISF Team

by inliniac at April 20, 2015 08:36 AM

April 06, 2015

Peter Manev

Suricata IDPS - Application layer anomalies protocol detection




Suricata IDS/IPS/NSM also allows you to do application layer anomaly  detection.
I started talking to inliniac about protocol anomaly detection rules one day on the Suricata IRC chat room...which evolved more into a discussion resulting in us updating the rule sets with some examples of how to do that.

Below are a few examples for rules usage:

HTTP

alert tcp any any -> any ![80,8080] (msg:"SURICATA HTTP not tcp port 80, 8080"; flow:to_server; app-layer-protocol:http; sid:2271001; rev:1;)
The above rule finds http traffic that is not using dest port 80 or 8080.


alert tcp any any -> any 80 (msg:"SURICATA Port 80 but not HTTP"; flow:to_server; app-layer-protocol:!http; sid:2271002; rev:1;)
The above rule is kind of the reverse of the previous one - it will alert if the tcp traffic with destination port 80 is not http.

Here is another example

TLS

alert tcp any any -> any 443 (msg:"SURICATA Port 443 but not TLS"; flow:to_server; app-layer-protocol:!tls; sid:2271003; rev:1;)

HTTPS

Detecting HTTP traffic over HTTPS port -

alert http any any -> any 443 (msg:"SURICATA HTTP clear text on port 443"; flow:to_server; app-layer-protocol:http; sid:2271019; rev:1;)

You can find the full ruleset (open source and free to use) with examples for HTTP, HTTPS, TLS, FTP, SMTP, SSH, IMAP, SMB, DCERPC, DNS, MODBUS application layer anomaly  detection  here:

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Protocol_Anomalies_Detection







by Peter Manev (noreply@blogger.com) at April 06, 2015 05:01 AM

April 02, 2015

suricata-ids.org

2015 OISF Board Elections have started

OISF_tmThe Open Information Security Foundation (OISF) is conducting its annual online elections to fill 7 positions on the OISF board of directors.  Board members serve a two year term, therefore, current board members along with new nominees are included on this year’s ballot.

The upcoming OISF board will consist of 10 board members in total:  7 elected directors, President of OISF, Matt Jonkman, General Manager of OISF, Kelley Misata, and Suricata’s Founder and Lead Developer, Victor Julien.

Each nominee has provided a brief summary highlighting their industry experience and their passion for OISF; please take a minute to read about each of our distinguished nominees and to cast your votes NOW!

Simply follow this link:  https://www.surveymonkey.com/s/Z2L6GXZ

Polls will close Wednesday, April 15, 2015 with the new OISF Board announced on Thursday, April 16, 2015.

Best of luck and thanks goes out to all of our nominees! Questions regarding elections can be sent to info@oisf.net.

Thank you,

The OISF Team

by inliniac at April 02, 2015 10:29 AM

Open Information Security Foundation

2015 OISF Board Elections have started

The Open Information Security Foundation (OISF) is conducting its annual online elections to fill 7 positions on the OISF board of directors.  Board members serve a two year term, therefore, current board members along with new nominees are included on this year's ballot.

The upcoming OISF board will consist of 10 board members in total:  7 elected directors, President of OISF, Matt Jonkman, General Manager of OISF, Kelley Misata, and Suricata's Founder and Lead Developer, Victor Julien.

Each nominee has provided a brief summary highlighting their industry experience and their passion for OISF; please take a minute to read about each of our distinguished nominees and to cast your votes NOW!

Simply follow this link:  https://www.surveymonkey.com/s/Z2L6GXZ

Polls will close Wednesday, April 15, 2015 with the new OISF Board announced on Thursday, April 16, 2015.

Best of luck and thanks goes out to all of our nominees! Questions regarding elections can be sent to info@oisf.net.

Thank you,

The OISF Team

by Victor Julien (postmaster@inliniac.net) at April 02, 2015 10:18 AM

March 26, 2015

suricata-ids.org

5-Day Suricata Developer Training

A short time ago we announced the first edition of the Suricata Developer training. I thought it be a good time to explain what we have in mind.

First, we’re planning to make this an annual event. I’m very excited about this. It should be great fun to have a week of development related discussions. I’m sure we can all learn a great deal, and share lots of ideas. Of course the social part is a nice addition.

Also, this is a great “stick behind the door” (as we’d say in Dutch) to finally get some much needed dev docs done, including architecture overview diagrams, etc.
What will we be teaching:

  1. General Suricata development basics: everything from git, how to QA, unittests, debugging, etc.
  2. Architecture overview
    1. API’s
    2. threading
    3. packets, flows, detection and output
  3. Extending Suricata — the beef of the training:
    1. packet decoders and detection plugins
    2. app layer protocol detection, parsing, state keeping
    3. app layer detection engine integration
    4. adding logging modules

Structure

Each day will start with lectures on each of the topics. You will get an overview of the API, learn about performance aspects, how threading comes into play, etc.

After this there will be assignments/challenges to apply the newly learned skills. This should be very interactive with lots of room for questions and discussions. We’ll be providing various assignments for multiple skill levels.

Teachers

From OISF we will have lead developer Victor Julien, core developers Eric Leblond and Jason Ish in the room to give the lectures and help answer questions. Also present to assist in general Suricata related questions: Matt Jonkman and Peter Manev.

Participants

So who is this for? We target people who what to learn how to extend Suricata. If you want to add protocols, detection options or maybe new output methods, then this is for you. Or maybe you want to be a ‘core’ developer on Suricata. Then this would be an excellent start of getting into it.

Skill-set:
– advanced C experience
– Linux Experience
– Network / Security Experience
– Basic Suricata End User Experience
=> for testing your code

If you’re not yet experienced with running Suricata, we would like to suggest attending a user training first. If you book your dev seat first hit us up for discount code on one of the training sessions. We’re planning one in Europe before summer, exact date and location is to be announced.

Money

We’ll be charging for the trainings. The revenue is used to cover the cost of the event itself (travel, hotels, etc). Whats left goes into the foundations general development budget. So by attending the training you will support Suricata’s development.

Location

This first edition is generously hosted by Napatech in Copenhagen, Denmark.
If you want to book, please do so through this Eventbrite link:
https://www.eventbrite.com/e/5-day-suricata-developer-training-in-copenhagen-denmark-open-to-the-public-tickets-15667305332
Questions and general feedback and thoughts are welcome!

by inliniac at March 26, 2015 05:38 PM

March 25, 2015

suricata-ids.org

Call for Nominations – OISF Board of Directors

OISF_tmThe Open Information Security Foundation is preparing hold the biennial (every two years) Board of Director elections and are putting out a call for nominations. We are anticipating the next several years to be both exciting and critical for OISF and Suricata, therefore, we are looking for candidates passionate about security and open source communities willing to serve as advisors on our board of directors. Your voice has a direct impact on future of OISF and Suricata – join us! The call for nominations begins today until March 31, 2015.

Read more at the OISF website.

by inliniac at March 25, 2015 06:31 PM

March 16, 2015

Open Information Security Foundation

Call for Nominations - OISF Board of Directors

The Open Information Security Foundation is preparing hold the biennial (every two years) Board of Director elections and are putting out a call for nominations.  We are anticipating the next several years to be both exciting and critical for OISF and Suricata, therefore, we are looking for candidates passionate about security and open source communities willing to serve as advisors on our board of directors.  Your voice has a direct impact on future of OISF and Suricata - join us! The call for nominations begins today until March 31, 2015.

Online elections will begin April 1, 2015.

Please consider joining our Board of Directors or nominating someone else who would be a great asset.  To help you decided, below are some answer to some common questions:

1. As an OISF board member what will I be asked to do?
Meetings: The OISF Board of Directors meet quarterly to review foundation activities, upcoming events, financial status and strategic objectives.  Meetings are held via conference call and pre-scheduled to respect the busy schedules of our board members.  Additionally, we host annual OISF User Conferences in locations around the world with our objectives of building Suricata's development roadmap, showing appreciation for OISF's consortium members, and growing the community.  We would hope that board members make every effort to attend this important event.  Our 2015 OISF User Conference is currently being planned for early November in Barcelona, Spain.

Advocacy: Board members will be asked to actively promote OISF, Suricata, and our events throughout the year.  OISF and Suricata exist because of the commitment of our community and we look to our board members to actively help us grow our presence in the world.

Expert Advice: Board members are expected to actively provide expertise, advice and professional connections necessary to help OISF make great strides both technologically and growing the community.

2. How large is the OISF board?
The current OISF board currently consisted of 6 members from the community - led by Matt Jonkman, Kelley Misata, and Victor Julien of OISF.  We will be expanding our board to 7 members serving for a 2 year term.

3. What is in it for me if I become an OISF board member?
As a board member you will have the opportunity to steer an innovative and cutting edge open source technology, to be an integral part of the decision making process for OISF and have a beneficiary priority status in all OISF and Suricata related public or private events.  Board members will be publicly acknowledged in OISF or Suricata related events and added to the OISF website spotlighting their professional bios.  Depending on OISF's financial capacity we are hoping to offer board members partial travel reimbursement to attend the annual OISF User Conferences - this is not guaranteed, but something we are hoping to be able to offer our board members.

4. I'm interested in nominating myself or someone I know - how do I do it?
It's simple - submit your name, name of your employer and a brief statement outlining your experience and reasons for running to be on the OISF board to info@oisf.net by 5 pm EST Tuesday, March 31, 2015.  Please note, the information provided in the nomination will be included on the PUBLIC election ballots so please be brief.

Elections will begin Wednesday, April 1st and conclude on Wednesday, April 15th.  The OISF Board Members will then be announced on Thursday, April 16th.

If you have any questions please do not hesitate to reach out to us directly at info@oisf.net OR reply to list to start a conversation with the community about this process.

Thank you,
The OISF Team

by Victor Julien (postmaster@inliniac.net) at March 16, 2015 03:09 PM

March 02, 2015

Security Onion

Suricata 2.0.7

Suricata 2.0.7 was recently released:
http://www.openinfosecfoundation.org/index.php/component/content/article/1-latest-news/204-suricata-207-available

I've packaged Suricata 2.0.7 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.7-0ubuntu0securityonion1

Issues Resolved

Issue 695: Suricata 2.0.7
https://code.google.com/p/security-onion/issues/detail?id=695

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:
  • sudo rule-update

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Training
Need training?  We have 4-day classes coming up in Atlanta, Seattle, and Houston.  Please see:
http://securityonionsolutions.com

Commercial Support
Need commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at March 02, 2015 02:53 PM

February 25, 2015

Open Information Security Foundation

Suricata 2.0.7 Available!

The OISF development team is pleased to announce Suricata 2.0.7. This release fixes a number of important issues in the 2.0 series.

Two major issues. The first was brought to our attention by the Yahoo Pentest Team. It’s a parsing issue in the DCERPC parser that can happen when Suricata runs out of memory. The exact scope of the problem isn’t clear, but it could certainly lead to crashes. RCE might theoretically be possible but looks like it’s very hard.

The second issue was reported by Darien Huss of Emerging Threats. This is technically a libhtp issue, but it affects Suricata detection and logging. Certain characters in the URI could confuse the parsing of the HTTP request line, leading to possible detection bypass for ‘http_uri’ and to incomplete logging of the URI. Libhtp 0.5.17 has been released to address this and is bundled in 2.0.7.

Other than that a bunch of improvements and fixes. It should work again on CentOS 5. Midstream TCP was improved and some performance optimizations for HTTP proxy traffic were made.

Upgrading is highly recommended.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.7.tar.gz

Changes

  • Bug #1385: DCERPC traffic parsing issue
  • Bug #1391: http uri parsing issue
  • Bug #1383: tcp midstream window issue
  • Bug #1318: A thread-sync issue in streamTCP
  • Bug #1375: Regressions in list keywords option
  • Bug #1387: pcap-file hangs on systems w/o atomics support
  • Bug #1395: dump-counters unix socket command failure
  • Optimization #1376: file list is not cleaned up

Security

The DCERPC parsing issue has CVE-2015-0928 assigned to it.

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • The Yahoo Pentest Team
  • Darien Huss — Emerging Threats
  • FireEye
  • Dennis Lee

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at February 25, 2015 03:47 PM

suricata-ids.org

Suricata 2.0.7 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.7. This release fixes a number of important issues in the 2.0 series.

Two major issues. The first was brought to our attention by the Yahoo Pentest Team. It’s a parsing issue in the DCERPC parser that can happen when Suricata runs out of memory. The exact scope of the problem isn’t clear, but it could certainly lead to crashes. RCE might theoretically be possible but looks like it’s very hard.

The second issue was reported by Darien Huss of Emerging Threats. This is technically a libhtp issue, but it affects Suricata detection and logging. Certain characters in the URI could confuse the parsing of the HTTP request line, leading to possible detection bypass for ‘http_uri’ and to incomplete logging of the URI. Libhtp 0.5.17 has been released to address this and is bundled in 2.0.7.

Other than that a bunch of improvements and fixes. It should work again on CentOS 5. Midstream TCP was improved and some performance optimizations for HTTP proxy traffic were made.

Upgrading is highly recommended.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.7.tar.gz

Changes

  • Bug #1385: DCERPC traffic parsing issue
  • Bug #1391: http uri parsing issue
  • Bug #1383: tcp midstream window issue
  • Bug #1318: A thread-sync issue in streamTCP
  • Bug #1375: Regressions in list keywords option
  • Bug #1387: pcap-file hangs on systems w/o atomics support
  • Bug #1395: dump-counters unix socket command failure
  • Optimization #1376: file list is not cleaned up

Security

The DCERPC parsing issue has CVE-2015-0928 assigned to it.

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • The Yahoo Pentest Team
  • Darien Huss — Emerging Threats
  • FireEye
  • Dennis Lee

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at February 25, 2015 03:46 PM

February 21, 2015

Peter Manev

Chasing MTUs


Setting up (configuring) the right MTU (maximum transmission unit) size when running Suricata IDS/IPS.

Sometimes you can end up in a situation as follows :


capture.kernel_packets    | AFPacketeth12              | 1143428204
decoder.pkts                    | AFPacketeth12             | 1143428143
decoder.invalid                | AFPacketeth12              | 416889536

a whole lot of  decoder.invalid. Not good. What could be the reason for that? One thing you should check right away is the MTU of the traffic that is being mirrored.

What does it mean? Well there is the MTU that you set up on the server that you run Suricata on and there is the MTU that is present in the "mirrored" traffic.

What is the difference?Why should it matter?
It matters because if not set correct  it will result in a lot of decoder.invalids (dropped by Suricata) and you will be missing on a lot of traffic inspection.
Example: if  on the sniffing interface that you run Suricata on has a MTU set as 1500  and in the traffic that you mirror you have jumbo frames (MTU 9000) - most likely your decoder.invalids will show a whole lotta love in your stats.log.

How can you adjust the MTU on the interface (NIC) ? (example)
First a have  look what is the current value:
ifconfig eth0
then adjust it
ifconfig eth0 mtu 1514

By the way - what could be the max size of the MTU (and what sizes there are in general)  -
(short answer - 9216)


This is the easy part :). There are situations where you do not know what is the MTU of the "mirrored" traffic. There is a few ways to find this  - ask the network team/guy, make a phone call or two, start manually testing and setting it on the NIC to find a middle ground ....however you can also make use of the procedure shown below (in order to get the byte size of the MTU):


On your Server/Sensor
1)
Stop Suricata.

2)
Change the MTU to 9216
(the interface that Suri is sniffing on)

example - ifconfig eth0 mtu 9216
(non boot persistent)

3)
install tcpstat - if you do not have it
apt-get install tcpstat

5)
run the following (substitute the interface name with yours - that Suri is sniffing on)
tcpstat -i eth0 -l -o "Time:%S\tn=%n\tavg=%a\tstddev=%d\tbps=%b\tMaxPacketSize=%M\n"  5
6)
Give it a minute or two
If there are Jumbo frames you should see that in the output (something like) -
"MaxPacketSize=9000", if not you should see whatever the max size is.

7)
Adjust your interface MTU accordingly  - the one that Suri is sniffing
on. -> Start Suri

8)
Let it run for  a while - lets say 1 hr. Have a look at the decoder.invalid stats in stats.log

NOTE: Do NOT just set the MTU to 9216 directly ("just to be on the safe side"). Only set it that high if needed !!

NOTE: This example below is not using the "-l" option of tcpstat as denoted in point 5) above - look at man tcpstat for more info



(tested on Ubuntu/Debian)
That's all ....feedback welcome.









by Peter Manev (noreply@blogger.com) at February 21, 2015 06:34 AM

February 18, 2015

Eric Leblond

Slides of my talks at Lecce

I’ve been invited by SaLUG to Lecce to give some talks during their Geek Evening. I’ve done a talk on nftables and one of suricata.

Lecce by night

Lecce by night

The nftables talk was about the motivation behind the change from iptables. Here are the slides: Nftables

The talk on Suricata was explaining the different feature of Suricata and was showing how I’ve used it to make a study of SSH bruteforce. Here are the slides: Suricata, Netfilter and the PRC.

Thanks a lot to Giuseppe Longo, Luca Greco and all the SaLUG team, you have been wonderful hosts!

by Regit at February 18, 2015 10:44 AM

January 29, 2015

Open Information Security Foundation

Suricata 2.1beta3 Available!

The OISF development team is proud to announce Suricata 2.1beta3. This is the third beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta3.tar.gz

New Features

  • Feature #1309: Lua support for Stats output
  • Feature #1310: Modbus parsing and matching

Improvements

  • Optimization #1339: flow timeout optimization
  • Optimization #1371: mpm optimization
  • Feature #1317: Lua: Indicator for end of flow
  • Feature #1333: unix-socket: allow (easier) non-root usage
  • Feature #1261: Request for Additional Lua Capabilities

Bugs

  • Bug #977: WARNING on empty rules file is fatal (should not be)
  • Bug #1184: pfring: cppcheck warnings
  • Bug #1321: Flow memuse bookkeeping error
  • Bug #1327: pcre pkt/flowvar capture broken for non-relative matches (master)
  • Bug #1332: cppcheck: ioctl
  • Bug #1336: modbus: CID 1257762: Logically dead code (DEADCODE)
  • Bug #1351: output-json: duplicate logging (2.1.x)
  • Bug #1354: coredumps on quitting on OpenBSD
  • Bug #1355: Bus error when reading pcap-file on OpenBSD
  • Bug #1363: Suricata does not compile on OS X/Clang due to redefinition of string functions (2.1.x)
  • Bug #1365: evasion issues (2.1.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera/EZchip
  • David Diallo
  • Duarte Silva
  • Giuseppe Longo
  • Jason Ish
  • Travis Green — Emerging Threats

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at January 29, 2015 04:38 PM

suricata-ids.org

Suricata 2.1beta3 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.1beta3. This is the third beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta3.tar.gz

New Features

  • Feature #1309: Lua support for Stats output
  • Feature #1310: Modbus parsing and matching

Improvements

  • Optimization #1339: flow timeout optimization
  • Optimization #1371: mpm optimization
  • Feature #1317: Lua: Indicator for end of flow
  • Feature #1333: unix-socket: allow (easier) non-root usage
  • Feature #1261: Request for Additional Lua Capabilities

Bugs

  • Bug #977: WARNING on empty rules file is fatal (should not be)
  • Bug #1184: pfring: cppcheck warnings
  • Bug #1321: Flow memuse bookkeeping error
  • Bug #1327: pcre pkt/flowvar capture broken for non-relative matches (master)
  • Bug #1332: cppcheck: ioctl
  • Bug #1336: modbus: CID 1257762: Logically dead code (DEADCODE)
  • Bug #1351: output-json: duplicate logging (2.1.x)
  • Bug #1354: coredumps on quitting on OpenBSD
  • Bug #1355: Bus error when reading pcap-file on OpenBSD
  • Bug #1363: Suricata does not compile on OS X/Clang due to redefinition of string functions (2.1.x)
  • Bug #1365: evasion issues (2.1.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera/EZchip
  • David Diallo
  • Duarte Silva
  • Giuseppe Longo
  • Jason Ish
  • Travis Green — Emerging Threats

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at January 29, 2015 04:33 PM

January 21, 2015

suricata-ids.org

Suricata Ubuntu PPA updated to 2.0.6

We have updated the official Ubuntu PPA to Suricata 2.0.6. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at January 21, 2015 12:06 PM

Suricata 2.0.6 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.6 release is now available.

Download it here: Suricata-2.0.6-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at January 21, 2015 12:03 PM

January 20, 2015

Security Onion

Suricata 2.0.6

Suricata 2.0.6 was recently released:
http://suricata-ids.org/2015/01/15/suricata-2-0-6-available/

I've packaged Suricata 2.0.6 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.6-0ubuntu0securityonion1

Issues Resolved

Issue 673: Suricata 2.0.6
https://code.google.com/p/security-onion/issues/detail?id=673

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:
    sudo rule-update


Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Commercial Support
Need training and/or commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at January 20, 2015 08:55 AM

January 16, 2015

suricata-ids.org

Suricata 2.0.6 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.6. This release fixes a number of important issues in the 2.0 series. The most important part is the fixing of evasion issues, therefore upgrading is highly recommended!

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.6.tar.gz

Changes

  • Bug #1364: evasion issues
  • Bug #1337: output-json: duplicate logging
  • Bug #1325: tls detection leads to tcp stream reassembly sequence gaps (IPS)
  • Bug #1192: Suricata does not compile on OS X/Clang due to redefinition of string functions
  • Bug #1183: pcap: cppcheck warning

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Martin Küchler

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at January 16, 2015 03:55 PM

January 15, 2015

Open Information Security Foundation

Suricata 2.0.6 Available!

The OISF development team is pleased to announce Suricata 2.0.6. This release fixes a number of important issues in the 2.0 series. The most important part is the fixing of evasion issues, therefore upgrading is highly recommended!

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.6.tar.gz

Changes

  • Bug #1364: evasion issues
  • Bug #1337: output-json: duplicate logging
  • Bug #1325: tls detection leads to tcp stream reassembly sequence gaps (IPS)
  • Bug #1192: Suricata does not compile on OS X/Clang due to redefinition of string functions
  • Bug #1183: pcap: cppcheck warning

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Martin Küchler

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at January 15, 2015 09:44 AM

January 08, 2015

Victor Julien

Suricata has been added to Debian Backports

Thanks to the hard work of Arturo Borrero Gonzalez, Suricata has just been added to the openlogo-100Debian ‘backports’ repository. This allows users of Debian stable to run up to date versions of Suricata.

The ‘Backports’ repository makes the Suricata and libhtp packages from Debian Testing available to ‘stable’ users. As ‘testing’ is currently in a freeze, it may take a bit of time before 2.0.5 and libhtp 0.5.16 appear.

Anyway, here is how to use it.

Install

First add backports repo to your sources:

# echo "deb http://http.debian.net/debian wheezy-backports main" > /etc/apt/sources.list.d/backports.list
# apt-get update

As explained here http://backports.debian.org/Instructions/, this will not affect your normal packages.

To prove this, check:

# apt-get install suricata -s
Conf libhtp1 (0.2.6-2 Debian:7.7/stable [amd64])
Conf suricata (1.2.1-2 Debian:7.7/stable [amd64])

Not what we want, as that is still the old version.

To install Suricata from backports, we need to specify the repo:

# apt-get install -t wheezy-backports suricata -s
Conf libhtp1 (0.5.15-1~bpo70+1 Debian Backports:/wheezy-backports [amd64])
Conf suricata (2.0.4-1~bpo70+1 Debian Backports:/wheezy-backports [amd64])

Let’s do it!

# apt-get install -t wheezy-backports suricata
...
Setting up suricata (2.0.4-1~bpo70+1) ...
[FAIL] suricata disabled, please adjust the configuration to your needs ... failed!
[FAIL] and then set RUN to 'yes' in /etc/default/suricata to enable it. ... failed!

Suricata 2.0.4 is now installed, but it’s not yet running.
To see what features have been compiled in, run:

# suricata --build-info
This is Suricata version 2.0.4 RELEASE

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         yes
  NFLOG support:                           no
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  Prelude support:                         yes
  PCRE jit:                                yes
  LUA support:                             yes
  libluajit:                               yes
  libgeoip:                                no
  Non-bundled htp:                         yes
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

It has Luajit enabled, libjansson for the JSON output, NFQ and AF_PACKET IPS modes, NSS for MD5 checksums and unix sockets. Quite a good feature set.

Run

To get it running, we need a few more steps:

Edit /etc/default/suricata:

1. Change RUN=no to RUN=yes
2. Change LISTENMODE to “af-packet”:

Now lets start it.

# service suricata start
Starting suricata in IDS (af-packet) mode... done.

And confirm that it’s running.

# ps aux|grep suricata
root     20295  1.8  4.1 200212 42544 ?        Ssl  00:50   0:00 /usr/bin/suricata -c /etc/suricata/suricata-debian.yaml --pidfile /var/run/suricata.pid --af-packet -D

Check if we’re seeing traffic:

# tail /var/log/suricata/stats.log -f|grep capture
capture.kernel_packets    | RxAFPeth01                | 406
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 411
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 417
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 587
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 593
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0

Logging

As the init script starts Suricata in daemon mode, we need to enable logging to file:

Edit /etc/suricata/suricata-debian.yaml and go to the “logging:” section, there change the “file” portion to look like:

  - file:
      enabled: yes
      filename: /var/log/suricata/suricata.log

Note: in the YAML indentation matters, so make sure it’s exactly right.

Rules

Oinkmaster is automatically installed, so lets use that:

First create the rules directory:

mkdir /etc/suricata/rules/

Open /etc/oinkmaster.conf in your editor and add:

url = https://rules.emergingthreats.net/open/suricata-2.0/emerging.rules.tar.gz

Then run:

# oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules
Loading /etc/oinkmaster.conf
Downloading file from https://rules.emergingthreats.net/open/suricata-2.0/emerging.rules.tar.gz... done.
...

Edit /etc/suricata/suricata-debian.yaml and change “default-rule-path” to:

default-rule-path: /etc/suricata/rules

Finally, restart to load the new rules:

# service suricata restart

Validate

Now that Suricata is running with rules, lets see if it works:

# wget http://www.testmyids.com
--2015-01-08 01:21:30--  http://www.testmyids.com/
Resolving www.testmyids.com (www.testmyids.com)... 82.165.177.154

This should trigger a specific rule:

# tail /var/log/suricata/fast.log 
01/08/2015-01:21:30.870346  [**] [1:2100498:7] GPL ATTACK_RESPONSE id check returned root [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 82.165.177.154:80 -> 192.168.122.181:59190

Success!:)

Thanks

Thanks to Arturo Borrero Gonzalez for taking on this work for us. Also many thanks for Pierre Chifflier for maintaining the Suricata and libhtp packages in Debian.


by inliniac at January 08, 2015 12:34 AM

January 05, 2015

Security Onion

Suricata 2.0.5

Suricata 2.0.5 was recently released:
http://www.openinfosecfoundation.org/index.php/component/content/article/1-latest-news/201-suricata-205-available

I've packaged Suricata 2.0.5 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.5-0ubuntu0securityonion1

Issues Resolved

Issue 655: Suricata 2.0.5
https://code.google.com/p/security-onion/issues/detail?id=655

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:

sudo rule-update

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Commercial Support
Need training and/or commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at January 05, 2015 10:15 AM

December 24, 2014

suricata-ids.org

Suricata Ubuntu PPA updated to 2.0.5

We have updated the official Ubuntu PPA to Suricata 2.0.5. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at December 24, 2014 07:02 PM

December 23, 2014

Victor Julien

Profiling Suricata with JEMALLOC

JEMALLOC is a memory allocation library: http://www.canonware.com/jemalloc/

It offers many interesting things for a tool like Suricata. Ken Steele of EZchip (formerly Tilera) made me aware of it. In Ken’s testing it helps performance.

Install

wget http://www.canonware.com/download/jemalloc/jemalloc-3.6.0.tar.bz2
tar xvfj jemalloc-3.6.0.tar.bz2
cd jemalloc-3.6.0
./configure --prefix=/opt/jemalloc/
make
sudo make install

Then use it by preloading it:

LD_PRELOAD=/opt/jemalloc/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

I haven’t benchmarked this, but if you’re running a high performance setup it may certainly be worth a shot.

Profiling

The library comes with many interesting profiling and debugging features.

make clean
./configure --prefix=/opt/jemalloc-prof/ --enable-prof
make
sudo make install

Start Suricata like this:

LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

Now we don’t see any change as we need to tell jemalloc what we want.

Exit stats

Dumps a lot of stats to the screen.

MALLOC_CONF=stats_print:true LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

Memory leak checks

MALLOC_CONF=prof_leak:true,lg_prof_sample:0 LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v
[... suricata output ...]
<jemalloc>: Leak summary: 2011400 bytes, 4523 objects, 645 contexts
<jemalloc>: Run pprof on "jeprof.22760.0.f.heap" for leak detail

Then use the pprof tool that comes with jemalloc to inspect the dumped stats.

$ /opt/jemalloc-prof/bin/pprof --show_bytes ./src/suricata jeprof.22760.0.f.heap
Using local file ./src/suricata.
Using local file jeprof.22760.0.f.heap.
Welcome to pprof!  For help, type 'help'.
(pprof) top
Total: 2011400 B
1050112  52.2%  52.2%  1050112  52.2% PacketGetFromAlloc
600064  29.8%  82.0%   600064  29.8% SCProfilePacketStart
103936   5.2%  87.2%   103936   5.2% SCACCreateDeltaTable
65536   3.3%  90.5%    66192   3.3% pcap_fopen_offline
35520   1.8%  92.2%    35520   1.8% ConfNodeNew
26688   1.3%  93.6%    26688   1.3% __GI___strdup
20480   1.0%  94.6%    20480   1.0% MemBufferCreateNew
20480   1.0%  95.6%    20480   1.0% _TmSlotSetFuncAppend
14304   0.7%  96.3%    14304   0.7% pcre_compile2
14064   0.7%  97.0%    25736   1.3% SCPerfRegisterQualifiedCounter

So it seems we don’t properly clean up our packet pools yet.

Create a PDF of this info:

$ /opt/jemalloc-prof/bin/pprof --show_bytes --pdf ./src/suricata jeprof.22760.0.f.heap > jemalloc.pdf

Dumping stats during runtime

Dump stats after every 16MiB of allocations (lg_prof_interval:24, means every 2^24 bytes, so 16MiB)

I’ve done this in a separate directory since it dumps many files.

$ mkdir jemalloc-profile
$ cd jemalloc-profile/
$ MALLOC_CONF="prof:true,prof_prefix:victor.out,lg_prof_interval:24" LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ../src/suricata -c ../suricata.yaml -l ../tmp/ -r ~/sync/pcap/sandnet.pcap -S ../emerging-all.rules -v

Then you should see new *.heap files appear, many during startup. But after some time it should slow down.

You can then visualize the diff between two dumps:

$ /opt/jemalloc-prof/bin/pprof --show_bytes --pdf ../src/suricata --base victor.out.24159.150.i150.heap victor.out.24159.200.i200.heap > jemalloc.pdf

This creates a PDF of the 200th dump taking the 150th dump as a baseline. As we dump every ~16MiB, this covers about 50 * 16 = 800MiB worth of allocations.

Further reading

http://www.canonware.com/jemalloc/
https://github.com/jemalloc/jemalloc/wiki
https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

Many thanks to Ken Steele for pointing me to the lib and providing me with some good examples.


by inliniac at December 23, 2014 03:50 PM

December 20, 2014

Victor Julien

Crossing the Streams in Suricata

At it’s core, Suricata is a packet processor. It reads packets and pushes them through a configurable pipeline. The 2nd most important processing unit in Suricata is the flow. In Suricata we use the term flow for the bidirectional flows of packets with the same 5 tuple (proto, src ip, dst ip, sp, dp. Vlans can be added as well). In fact, much of Suricata’s threading effort revolves around the flow. In the 2 main runmodes, autofp and workers, flow based load balancing makes sure that a all packets of a single flow always go through the same threading pipeline. In workers this means one single thread, in autofp 2: the capture thread and a stream/detect/output thread.

Flows are the central unit for out ‘app layer’ parsing. Protocol parsers like HTTP don’t even have access to the original packet. It all runs on top of the stream engine, which tracks TCP flows in … our flow structure.

Another place where the flow is crucial is in many of the rules. Rules extensively use the concept of ‘flowbits’. This allows one rule to ‘flag’ a flow, and then another to check this flag. In Emerging Threats many hundreds of rules use this logic.

Ever since we started Suricata, we’ve been talking about what some called ‘global flowbits’. A bit of a strange and contradictory name, but pretty much rule writers wanted the logic of flowbits, but then applied to other units as well. So a few weeks ago I (finally) decided to check if I could quickly implement ‘hostbits’. As Suricata already has a scalable ‘host table’, it was easy add the storage of ‘bits’ there. In a few hours I had the basics working and made it public: see this pull request.

Although I got some nice feedback, I was mostly interested in what the ET folks would think, since they would be the main consumers. While presenting the work I also mentioned the xbits ideas by Michael Rash and the response was “wow, do we have ip_pair tracking now?”. Ehh, no, just ip/host based… “Ah well, I guess that is nice too”. Not exactly the response I hoped for:)

IP pair tracking is not something Suricata already did. But as the need was clear I decided to have a look at it. Turned out it was quite simple to do. The IPPair tracker is much like the Host tracking. It’s only done on demand, which sets it apart from the Flow tracking which is done unconditionally. In this case only the new keyword is making use of the IP Pair storage.

So, what I have implemented is pretty much ‘xbits’. It supports tracking by ‘ip_src’, ‘ip_dst’ and ‘ip_pair’. It uses the syntax as suggested by Michael Rash:

xbits:<set|unset|isset|isnotset|toggle>,<bitname>,\
      track <ip_src|ip_dst|ip_pair>,expire <seconds>

It’s only lightly tested, so I would appreciate testing feedback!

You’ll find the code here in PR 1275 at github. This should normally end up in Suricata 2.1, which will come out early next year.


by inliniac at December 20, 2014 11:37 PM