Planet Suricata

January 29, 2015

Open Information Security Foundation

Suricata 2.1beta3 Available!

The OISF development team is proud to announce Suricata 2.1beta3. This is the third beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta3.tar.gz

New Features

  • Feature #1309: Lua support for Stats output
  • Feature #1310: Modbus parsing and matching

Improvements

  • Optimization #1339: flow timeout optimization
  • Optimization #1371: mpm optimization
  • Feature #1317: Lua: Indicator for end of flow
  • Feature #1333: unix-socket: allow (easier) non-root usage
  • Feature #1261: Request for Additional Lua Capabilities

Bugs

  • Bug #977: WARNING on empty rules file is fatal (should not be)
  • Bug #1184: pfring: cppcheck warnings
  • Bug #1321: Flow memuse bookkeeping error
  • Bug #1327: pcre pkt/flowvar capture broken for non-relative matches (master)
  • Bug #1332: cppcheck: ioctl
  • Bug #1336: modbus: CID 1257762: Logically dead code (DEADCODE)
  • Bug #1351: output-json: duplicate logging (2.1.x)
  • Bug #1354: coredumps on quitting on OpenBSD
  • Bug #1355: Bus error when reading pcap-file on OpenBSD
  • Bug #1363: Suricata does not compile on OS X/Clang due to redefinition of string functions (2.1.x)
  • Bug #1365: evasion issues (2.1.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera/EZchip
  • David Diallo
  • Duarte Silva
  • Giuseppe Longo
  • Jason Ish
  • Travis Green — Emerging Threats

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at January 29, 2015 04:38 PM

suricata-ids.org

Suricata 2.1beta3 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.1beta3. This is the third beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta3.tar.gz

New Features

  • Feature #1309: Lua support for Stats output
  • Feature #1310: Modbus parsing and matching

Improvements

  • Optimization #1339: flow timeout optimization
  • Optimization #1371: mpm optimization
  • Feature #1317: Lua: Indicator for end of flow
  • Feature #1333: unix-socket: allow (easier) non-root usage
  • Feature #1261: Request for Additional Lua Capabilities

Bugs

  • Bug #977: WARNING on empty rules file is fatal (should not be)
  • Bug #1184: pfring: cppcheck warnings
  • Bug #1321: Flow memuse bookkeeping error
  • Bug #1327: pcre pkt/flowvar capture broken for non-relative matches (master)
  • Bug #1332: cppcheck: ioctl
  • Bug #1336: modbus: CID 1257762: Logically dead code (DEADCODE)
  • Bug #1351: output-json: duplicate logging (2.1.x)
  • Bug #1354: coredumps on quitting on OpenBSD
  • Bug #1355: Bus error when reading pcap-file on OpenBSD
  • Bug #1363: Suricata does not compile on OS X/Clang due to redefinition of string functions (2.1.x)
  • Bug #1365: evasion issues (2.1.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera/EZchip
  • David Diallo
  • Duarte Silva
  • Giuseppe Longo
  • Jason Ish
  • Travis Green — Emerging Threats

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at January 29, 2015 04:33 PM

January 21, 2015

suricata-ids.org

Suricata Ubuntu PPA updated to 2.0.6

We have updated the official Ubuntu PPA to Suricata 2.0.6. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at January 21, 2015 12:06 PM

Suricata 2.0.5 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.5 release is now available.

Download it here: Suricata-2.0.5-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at January 21, 2015 12:05 PM

Suricata 2.0.6 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.6 release is now available.

Download it here: Suricata-2.0.6-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at January 21, 2015 12:03 PM

January 20, 2015

Security Onion

Suricata 2.0.6

Suricata 2.0.6 was recently released:
http://suricata-ids.org/2015/01/15/suricata-2-0-6-available/

I've packaged Suricata 2.0.6 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.6-0ubuntu0securityonion1

Issues Resolved

Issue 673: Suricata 2.0.6
https://code.google.com/p/security-onion/issues/detail?id=673

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:
    sudo rule-update


Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Commercial Support
Need training and/or commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at January 20, 2015 08:55 AM

January 16, 2015

suricata-ids.org

Suricata 2.0.6 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.6. This release fixes a number of important issues in the 2.0 series. The most important part is the fixing of evasion issues, therefore upgrading is highly recommended!

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.6.tar.gz

Changes

  • Bug #1364: evasion issues
  • Bug #1337: output-json: duplicate logging
  • Bug #1325: tls detection leads to tcp stream reassembly sequence gaps (IPS)
  • Bug #1192: Suricata does not compile on OS X/Clang due to redefinition of string functions
  • Bug #1183: pcap: cppcheck warning

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Martin Küchler

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at January 16, 2015 03:55 PM

January 15, 2015

Open Information Security Foundation

Suricata 2.0.6 Available!

The OISF development team is pleased to announce Suricata 2.0.6. This release fixes a number of important issues in the 2.0 series. The most important part is the fixing of evasion issues, therefore upgrading is highly recommended!

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.6.tar.gz

Changes

  • Bug #1364: evasion issues
  • Bug #1337: output-json: duplicate logging
  • Bug #1325: tls detection leads to tcp stream reassembly sequence gaps (IPS)
  • Bug #1192: Suricata does not compile on OS X/Clang due to redefinition of string functions
  • Bug #1183: pcap: cppcheck warning

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Martin Küchler

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at January 15, 2015 09:44 AM

January 08, 2015

Victor Julien

Suricata has been added to Debian Backports

Thanks to the hard work of Arturo Borrero Gonzalez, Suricata has just been added to the openlogo-100Debian ‘backports’ repository. This allows users of Debian stable to run up to date versions of Suricata.

The ‘Backports’ repository makes the Suricata and libhtp packages from Debian Testing available to ‘stable’ users. As ‘testing’ is currently in a freeze, it may take a bit of time before 2.0.5 and libhtp 0.5.16 appear.

Anyway, here is how to use it.

Install

First add backports repo to your sources:

# echo "deb http://http.debian.net/debian wheezy-backports main" > /etc/apt/sources.list.d/backports.list
# apt-get update

As explained here http://backports.debian.org/Instructions/, this will not affect your normal packages.

To prove this, check:

# apt-get install suricata -s
Conf libhtp1 (0.2.6-2 Debian:7.7/stable [amd64])
Conf suricata (1.2.1-2 Debian:7.7/stable [amd64])

Not what we want, as that is still the old version.

To install Suricata from backports, we need to specify the repo:

# apt-get install -t wheezy-backports suricata -s
Conf libhtp1 (0.5.15-1~bpo70+1 Debian Backports:/wheezy-backports [amd64])
Conf suricata (2.0.4-1~bpo70+1 Debian Backports:/wheezy-backports [amd64])

Let’s do it!

# apt-get install -t wheezy-backports suricata
...
Setting up suricata (2.0.4-1~bpo70+1) ...
[FAIL] suricata disabled, please adjust the configuration to your needs ... failed!
[FAIL] and then set RUN to 'yes' in /etc/default/suricata to enable it. ... failed!

Suricata 2.0.4 is now installed, but it’s not yet running.
To see what features have been compiled in, run:

# suricata --build-info
This is Suricata version 2.0.4 RELEASE

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         yes
  NFLOG support:                           no
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  Prelude support:                         yes
  PCRE jit:                                yes
  LUA support:                             yes
  libluajit:                               yes
  libgeoip:                                no
  Non-bundled htp:                         yes
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

It has Luajit enabled, libjansson for the JSON output, NFQ and AF_PACKET IPS modes, NSS for MD5 checksums and unix sockets. Quite a good feature set.

Run

To get it running, we need a few more steps:

Edit /etc/default/suricata:

1. Change RUN=no to RUN=yes
2. Change LISTENMODE to “af-packet”:

Now lets start it.

# service suricata start
Starting suricata in IDS (af-packet) mode... done.

And confirm that it’s running.

# ps aux|grep suricata
root     20295  1.8  4.1 200212 42544 ?        Ssl  00:50   0:00 /usr/bin/suricata -c /etc/suricata/suricata-debian.yaml --pidfile /var/run/suricata.pid --af-packet -D

Check if we’re seeing traffic:

# tail /var/log/suricata/stats.log -f|grep capture
capture.kernel_packets    | RxAFPeth01                | 406
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 411
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 417
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 587
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0
capture.kernel_packets    | RxAFPeth01                | 593
capture.kernel_drops      | RxAFPeth01                | 0
capture.kernel_packets    | RxAFPeth11                | 0
capture.kernel_drops      | RxAFPeth11                | 0

Logging

As the init script starts Suricata in daemon mode, we need to enable logging to file:

Edit /etc/suricata/suricata-debian.yaml and go to the “logging:” section, there change the “file” portion to look like:

  - file:
      enabled: yes
      filename: /var/log/suricata/suricata.log

Note: in the YAML indentation matters, so make sure it’s exactly right.

Rules

Oinkmaster is automatically installed, so lets use that:

First create the rules directory:

mkdir /etc/suricata/rules/

Open /etc/oinkmaster.conf in your editor and add:

url = https://rules.emergingthreats.net/open/suricata-2.0/emerging.rules.tar.gz

Then run:

# oinkmaster -C /etc/oinkmaster.conf -o /etc/suricata/rules
Loading /etc/oinkmaster.conf
Downloading file from https://rules.emergingthreats.net/open/suricata-2.0/emerging.rules.tar.gz... done.
...

Edit /etc/suricata/suricata-debian.yaml and change “default-rule-path” to:

default-rule-path: /etc/suricata/rules

Finally, restart to load the new rules:

# service suricata restart

Validate

Now that Suricata is running with rules, lets see if it works:

# wget http://www.testmyids.com
--2015-01-08 01:21:30--  http://www.testmyids.com/
Resolving www.testmyids.com (www.testmyids.com)... 82.165.177.154

This should trigger a specific rule:

# tail /var/log/suricata/fast.log 
01/08/2015-01:21:30.870346  [**] [1:2100498:7] GPL ATTACK_RESPONSE id check returned root [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 82.165.177.154:80 -> 192.168.122.181:59190

Success! :)

Thanks

Thanks to Arturo Borrero Gonzalez for taking on this work for us. Also many thanks for Pierre Chifflier for maintaining the Suricata and libhtp packages in Debian.


by inliniac at January 08, 2015 12:34 AM

January 05, 2015

Security Onion

Suricata 2.0.5

Suricata 2.0.5 was recently released:
http://www.openinfosecfoundation.org/index.php/component/content/article/1-latest-news/201-suricata-205-available

I've packaged Suricata 2.0.5 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.5-0ubuntu0securityonion1

Issues Resolved

Issue 655: Suricata 2.0.5
https://code.google.com/p/security-onion/issues/detail?id=655

Updating
The new package is now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:

sudo rule-update

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Commercial Support
Need training and/or commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

Thanks!

by Doug Burks (noreply@blogger.com) at January 05, 2015 10:15 AM

December 24, 2014

suricata-ids.org

Suricata Ubuntu PPA updated to 2.0.5

We have updated the official Ubuntu PPA to Suricata 2.0.5. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at December 24, 2014 07:02 PM

December 23, 2014

Victor Julien

Profiling Suricata with JEMALLOC

JEMALLOC is a memory allocation library: http://www.canonware.com/jemalloc/

It offers many interesting things for a tool like Suricata. Ken Steele of EZchip (formerly Tilera) made me aware of it. In Ken’s testing it helps performance.

Install

wget http://www.canonware.com/download/jemalloc/jemalloc-3.6.0.tar.bz2
tar xvfj jemalloc-3.6.0.tar.bz2
cd jemalloc-3.6.0
./configure --prefix=/opt/jemalloc/
make
sudo make install

Then use it by preloading it:

LD_PRELOAD=/opt/jemalloc/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

I haven’t benchmarked this, but if you’re running a high performance setup it may certainly be worth a shot.

Profiling

The library comes with many interesting profiling and debugging features.

make clean
./configure --prefix=/opt/jemalloc-prof/ --enable-prof
make
sudo make install

Start Suricata like this:

LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

Now we don’t see any change as we need to tell jemalloc what we want.

Exit stats

Dumps a lot of stats to the screen.

MALLOC_CONF=stats_print:true LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v

Memory leak checks

MALLOC_CONF=prof_leak:true,lg_prof_sample:0 LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ./src/suricata -c suricata.yaml -l tmp/ -r ~/sync/pcap/sandnet.pcap -S emerging-all.rules -v
[... suricata output ...]
<jemalloc>: Leak summary: 2011400 bytes, 4523 objects, 645 contexts
<jemalloc>: Run pprof on "jeprof.22760.0.f.heap" for leak detail

Then use the pprof tool that comes with jemalloc to inspect the dumped stats.

$ /opt/jemalloc-prof/bin/pprof --show_bytes ./src/suricata jeprof.22760.0.f.heap
Using local file ./src/suricata.
Using local file jeprof.22760.0.f.heap.
Welcome to pprof!  For help, type 'help'.
(pprof) top
Total: 2011400 B
1050112  52.2%  52.2%  1050112  52.2% PacketGetFromAlloc
600064  29.8%  82.0%   600064  29.8% SCProfilePacketStart
103936   5.2%  87.2%   103936   5.2% SCACCreateDeltaTable
65536   3.3%  90.5%    66192   3.3% pcap_fopen_offline
35520   1.8%  92.2%    35520   1.8% ConfNodeNew
26688   1.3%  93.6%    26688   1.3% __GI___strdup
20480   1.0%  94.6%    20480   1.0% MemBufferCreateNew
20480   1.0%  95.6%    20480   1.0% _TmSlotSetFuncAppend
14304   0.7%  96.3%    14304   0.7% pcre_compile2
14064   0.7%  97.0%    25736   1.3% SCPerfRegisterQualifiedCounter

So it seems we don’t properly clean up our packet pools yet.

Create a PDF of this info:

$ /opt/jemalloc-prof/bin/pprof --show_bytes --pdf ./src/suricata jeprof.22760.0.f.heap > jemalloc.pdf

Dumping stats during runtime

Dump stats after every 16MiB of allocations (lg_prof_interval:24, means every 2^24 bytes, so 16MiB)

I’ve done this in a separate directory since it dumps many files.

$ mkdir jemalloc-profile
$ cd jemalloc-profile/
$ MALLOC_CONF="prof:true,prof_prefix:victor.out,lg_prof_interval:24" LD_PRELOAD=/opt/jemalloc-prof/lib/libjemalloc.so ../src/suricata -c ../suricata.yaml -l ../tmp/ -r ~/sync/pcap/sandnet.pcap -S ../emerging-all.rules -v

Then you should see new *.heap files appear, many during startup. But after some time it should slow down.

You can then visualize the diff between two dumps:

$ /opt/jemalloc-prof/bin/pprof --show_bytes --pdf ../src/suricata --base victor.out.24159.150.i150.heap victor.out.24159.200.i200.heap > jemalloc.pdf

This creates a PDF of the 200th dump taking the 150th dump as a baseline. As we dump every ~16MiB, this covers about 50 * 16 = 800MiB worth of allocations.

Further reading

http://www.canonware.com/jemalloc/

https://github.com/jemalloc/jemalloc/wiki

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

Many thanks to Ken Steele for pointing me to the lib and providing me with some good examples.


by inliniac at December 23, 2014 03:50 PM

December 20, 2014

Victor Julien

Crossing the Streams in Suricata

At it’s core, Suricata is a packet processor. It reads packets and pushes them through a configurable pipeline. The 2nd most important processing unit in Suricata is the flow. In Suricata we use the term flow for the bidirectional flows of packets with the same 5 tuple (proto, src ip, dst ip, sp, dp. Vlans can be added as well). In fact, much of Suricata’s threading effort revolves around the flow. In the 2 main runmodes, autofp and workers, flow based load balancing makes sure that a all packets of a single flow always go through the same threading pipeline. In workers this means one single thread, in autofp 2: the capture thread and a stream/detect/output thread.

Flows are the central unit for out ‘app layer’ parsing. Protocol parsers like HTTP don’t even have access to the original packet. It all runs on top of the stream engine, which tracks TCP flows in … our flow structure.

Another place where the flow is crucial is in many of the rules. Rules extensively use the concept of ‘flowbits’. This allows one rule to ‘flag’ a flow, and then another to check this flag. In Emerging Threats many hundreds of rules use this logic.

Ever since we started Suricata, we’ve been talking about what some called ‘global flowbits’. A bit of a strange and contradictory name, but pretty much rule writers wanted the logic of flowbits, but then applied to other units as well. So a few weeks ago I (finally) decided to check if I could quickly implement ‘hostbits’. As Suricata already has a scalable ‘host table’, it was easy add the storage of ‘bits’ there. In a few hours I had the basics working and made it public: see this pull request.

Although I got some nice feedback, I was mostly interested in what the ET folks would think, since they would be the main consumers. While presenting the work I also mentioned the xbits ideas by Michael Rash and the response was “wow, do we have ip_pair tracking now?”. Ehh, no, just ip/host based… “Ah well, I guess that is nice too”. Not exactly the response I hoped for :)

IP pair tracking is not something Suricata already did. But as the need was clear I decided to have a look at it. Turned out it was quite simple to do. The IPPair tracker is much like the Host tracking. It’s only done on demand, which sets it apart from the Flow tracking which is done unconditionally. In this case only the new keyword is making use of the IP Pair storage.

So, what I have implemented is pretty much ‘xbits’. It supports tracking by ‘ip_src’, ‘ip_dst’ and ‘ip_pair’. It uses the syntax as suggested by Michael Rash:

xbits:<set|unset|isset|isnotset|toggle>,<bitname>,\
      track <ip_src|ip_dst|ip_pair>,expire <seconds>

It’s only lightly tested, so I would appreciate testing feedback!

You’ll find the code here in PR 1275 at github. This should normally end up in Suricata 2.1, which will come out early next year.


by inliniac at December 20, 2014 11:37 PM

December 12, 2014

Open Information Security Foundation

Suricata 2.0.5 Available!

The OISF development team is pleased to announce Suricata 2.0.5. This release fixes a number of important issues in the 2.0 series.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.5.tar.gz

Changes

  • Bug #1190: http_header keyword not matching when SYN|ACK and ACK missing
  • Bug #1246: EVE output Unix domain socket not working
  • Bug #1272: Segfault in libhtp 0.5.15
  • Bug #1298: Filestore keyword parsing issue
  • Bug #1303: improve stream ‘bad window update’ detection
  • Bug #1304: improve stream handling of bad SACK values
  • Bug #1305: fix tcp session reuse for ssh/ssl sessions
  • Bug #1307: byte_extract, within combination not working
  • Bug #1326: pcre pkt/flowvar capture broken for non-relative matches
  • Bug #1329: Invalid rule being processed and loaded
  • Bug #1330: Flow memuse bookkeeping error (2.0.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Jason Ish — Endace/Emulex
  • Ken Steele — Tilera
  • lessyv
  • Tom DeCanio — FireEye
  • Andreas Herz
  • Matt Carothers
  • Duane Howard
  • Edward Fjellskål
  • Giuseppe Longo

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at December 12, 2014 11:22 AM

suricata-ids.org

Suricata 2.0.5 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.5. This release fixes a number of important issues in the 2.0 series.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.5.tar.gz

Changes

  • Bug #1190: http_header keyword not matching when SYN|ACK and ACK missing
  • Bug #1246: EVE output Unix domain socket not working
  • Bug #1272: Segfault in libhtp 0.5.15
  • Bug #1298: Filestore keyword parsing issue
  • Bug #1303: improve stream ‘bad window update’ detection
  • Bug #1304: improve stream handling of bad SACK values
  • Bug #1305: fix tcp session reuse for ssh/ssl sessions
  • Bug #1307: byte_extract, within combination not working
  • Bug #1326: pcre pkt/flowvar capture broken for non-relative matches
  • Bug #1329: Invalid rule being processed and loaded
  • Bug #1330: Flow memuse bookkeeping error (2.0.x)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Jason Ish — Endace/Emulex
  • Ken Steele — Tilera
  • lessyv
  • Tom DeCanio — FireEye
  • Andreas Herz
  • Matt Carothers
  • Duane Howard
  • Edward Fjellskål
  • Giuseppe Longo

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at December 12, 2014 11:21 AM

December 07, 2014

Peter Manev

Suricatasc unix socket interaction for Suricata IDS/IPS



Suricatasc is a unix socket interaction  script that is automatically installed when one compiles/installs Suricata IDS/IPS. An in depth description, prerequisites and how to documentation is located here - https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Interacting_via_Unix_Socket

However  lets look at a quick usage example - that can come very handy in certain situations.

Once you have unix socket command enabled in suricata.yaml :

unix-command:
    enabled: yes
    #filename: custom.socket # use this to specify an alternate file

the traditional way to use the script would be type suricatasc and hit Enter (on the machine running Suricata):




However you can also use it directly as a command line parameter for example :
root@suricata:~# suricatasc -c version

like so:


NOTE:
You need to quote commands involving interfaces:
root@debian64:~# suricatasc -c "iface-stat eth0"



Very handy when you want quick interaction and info from the currently running Suricata IDS/IPS.




by Peter Manev (noreply@blogger.com) at December 07, 2014 11:34 AM

Suricata - disable detection mode


There is a trick for using Suricata IDS/IPS that has come in handy - in my experience that I thought  would share and might be useful to others.

My HW was CPU 1x E5-2680 with 64 GB RAM ,  NIC a  82599EB 10-Gigabit SFI/SFP+ and mirroring about 9,5Gbps:





So I wanted to get a better understanding of what is the max log output from Suricata in that particular environment without putting detection into consideration. Just pure event logging and profiling - DNS,HTTP,TLS,SSH, File transactions, SMTP all of these.

Suricata offers just that - with a really low need for HW (having in mind we are looking at 9,5Gbps). What i did was compile suricata with the "--disable-detection" switch. What it does it simply disables detection(alerts) in Suricata  - however every other logging/parsing capability is preserved ( DNS,HTTP,TLS,SSH, File transactions, SMTP). So i downloaded a fresh copy:

git clone git://phalanx.openinfosecfoundation.org/oisf.git && cd oisf/ &&  git clone https://github.com/ironbee/libhtp.git -b 0.5.x

then

./autogen.sh && ./configure --disable-detection   &&  make clean && make && make install &&  ldconfig

enabled all JSON outputs in the eve-log section in suricata.yaml. I confirmed that detection was disabled:


and started Suricata.

Careful what you asked for :) - I was getting 10-15K logs per second :

 root@suricata:/var/log/suricata# tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
1
6531
13860
10704
10877
10389
10664
10205
9996
14798
15427
14223

And the HW(CPU/RAM/HDD) usage was not much at all:


As you can see - 30-40% CPU with a third of RAM  usage.

 I used this command to count logs per second  -
tail -f  eve.json |perl -e 'while (<>) {$l++;if (time > $e) {$e=time;print "$l\n";$l=0}}'
more about similar commands and counting Suricata logs you could find here:


So this can came in handy in a few scenarious:
  • you can  actually do a  "profiling" run on that particular set up in order to size up a SIEM specification 
  • and/or you can size up your prod IDS/IPS deployment needs
  • you can also just feed all those logs info to an existing log analysis system

Thanks

by Peter Manev (noreply@blogger.com) at December 07, 2014 04:53 AM

December 05, 2014

suricata-ids.org

Suricata Ubuntu PPA updated to 2.1beta2

We have updated the official Ubuntu PPA to Suricata 2.1beta2. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at December 05, 2014 01:40 PM

Suricata 2.1beta2 Windows Installer Available

The Windows MSI installer of the Suricata 2.1beta2 release is now available.

Download it here: suricata-2.1beta2-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at December 05, 2014 01:37 PM

November 11, 2014

Victor Julien

SMTP file extraction in Suricata

In 2.1beta2 the long awaited SMTP file extraction support for Suricata finally appeared. It has been a long development cycle. Originally started by BAE Systems, it was picked up by Tom Decanio of FireEye Forensics Group (formerly nPulse Technologies) followed by a last round of changes from my side. But it’s here now.

It contains:

  • a MIME decoder
  • updates to the SMTP parser to use the MIME decoder for extracting files
  • SMTP JSON log, integrated with EVE
  • SMTP message URL extraction and logging

As it uses the Suricata file handling API, it shares almost everything with the existing file handling for HTTP. The rule keyword work and the various logs work automatically with SMTP as well.

Trying it out

To enable the file extraction, make sure that the MIME decoder is enabled:

app-layer:
  protocols:
    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

        # Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

        # Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

        # Extract URLs and save in state data structure
        extract-urls: yes

Like with HTTP, SMTP depends on the stream engine working correctly. So this page applies https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction, although of course the HTTP specific settings are irrelevant to SMTP.

Troubleshooting (SMTP) file extraction issues should always start here: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams#File-Extraction-and-Logging-Issues

Logging

Enabling the SMTP logging is simple, just add ‘smtp’ to the list of types in your EVE config, like so:

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert:
            # payload: yes           # enable dumping payload in Base64
            # payload-printable: yes # enable dumping payload in printable (lossy) format
            # packet: yes            # enable dumping of packet (without stream segments)
            # http: yes              # enable dumping of http fields
        - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional http fields to be included in eve-log
            # the example below adds three additional fields when uncommented
            #custom: [Accept-Encoding, Accept-Language, Authorization]
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: no   # force logging magic on all logged files
            force-md5: no     # force logging of md5 checksums
        #- drop
        - smtp
        - ssh
        # bi-directional flows
        #- flow
        # uni-directional flows
        #- newflow

URLs

As a bonus, the MIME decoder also extracts URL’s from the SMTP message body (not attachments) and logs them in the SMTP log. This should make it easy to post process them. Currently only ‘HTTP’ URLS are extracted, starting with ‘http://‘. So HTTPS/FTP or URLs that don’t have the protocol prefix aren’t logged.

Testing

Naturally, if you’re using SMTP over TLS or have STARTTLS enabled, as you should at least on public networks, none of this will work.

Please help us test this feature!


by inliniac at November 11, 2014 11:15 AM

November 06, 2014

Open Information Security Foundation

Suricata 2.1beta2 Available!

The OISF development team is proud to announce Suricata 2.1beta2. This is the second beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta2.tar.gz

New Features

  • Feature #549: Extract file attachments from emails
  • Feature #1312: Lua output support
  • Feature #899: MPLS over Ethernet support
  • Feature #383: Stream logging

Improvements

  • Feature #1263: Lua: Access to Stream Payloads
  • Feature #1264: Lua: access to TCP quad / Flow Tuple
  • Feature #707: ip reputation files – network range inclusion availability (cidr)

Bugs

  • Bug #1048: PF_RING/DNA config – suricata.yaml
  • Bug #1230: byte_extract, within combination not working
  • Bug #1257: Flow switch is missing from the eve-log section in suricata.yaml
  • Bug #1259: AF_PACKET IPS is broken in 2.1beta1
  • Bug #1260: flow logging at shutdown broken
  • Bug #1279: BUG: NULL pointer dereference when suricata was debug mode.
  • Bug #1280: BUG: IPv6 address vars issue
  • Bug #1285: Lua – http.request_line not working (2.1)
  • Bug #1287: Lua Output has dependency on eve-log:http
  • Bug #1288: Filestore keyword in wrong place will cause entire rule not to trigger
  • Bug #1294: Configure doesn’t use –with-libpcap-libraries when testing PF_RING library
  • Bug #1301: suricata yaml – PF_RING load balance per hash option
  • Bug #1308: http_header keyword not matching when SYN|ACK and ACK missing (master)
  • Bug #1311: EVE output Unix domain socket not working (2.1)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Tom Decanio — FireEye
  • Ken Steele — Tilera
  • Giuseppe Longo — Emerging Threats & Ntop
  • David Abarbanel — BAE Systems
  • Jason Ish — Endace/Emulex
  • Mats Klepsland
  • Duarte Silva
  • Bill Meeks
  • Anoop Saldanha
  • lessyv

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at November 06, 2014 10:28 AM

suricata-ids.org

Suricata 2.1beta2 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.1beta2. This is the second beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta2.tar.gz

New Features

  • Feature #549: Extract file attachments from emails
  • Feature #1312: Lua output support
  • Feature #899: MPLS over Ethernet support
  • Feature #383: Stream logging

Improvements

  • Feature #1263: Lua: Access to Stream Payloads
  • Feature #1264: Lua: access to TCP quad / Flow Tuple
  • Feature #707: ip reputation files – network range inclusion availability (cidr)

Bugs

  • Bug #1048: PF_RING/DNA config – suricata.yaml
  • Bug #1230: byte_extract, within combination not working
  • Bug #1257: Flow switch is missing from the eve-log section in suricata.yaml
  • Bug #1259: AF_PACKET IPS is broken in 2.1beta1
  • Bug #1260: flow logging at shutdown broken
  • Bug #1279: BUG: NULL pointer dereference when suricata was debug mode.
  • Bug #1280: BUG: IPv6 address vars issue
  • Bug #1285: Lua – http.request_line not working (2.1)
  • Bug #1287: Lua Output has dependency on eve-log:http
  • Bug #1288: Filestore keyword in wrong place will cause entire rule not to trigger
  • Bug #1294: Configure doesn’t use –with-libpcap-libraries when testing PF_RING library
  • Bug #1301: suricata yaml – PF_RING load balance per hash option
  • Bug #1308: http_header keyword not matching when SYN|ACK and ACK missing (master)
  • Bug #1311: EVE output Unix domain socket not working (2.1)

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Tom Decanio — FireEye
  • Ken Steele — Tilera
  • Giuseppe Longo — Emerging Threats & Ntop
  • David Abarbanel — BAE Systems
  • Jason Ish — Endace/Emulex
  • Mats Klepsland
  • Duarte Silva
  • Bill Meeks
  • Anoop Saldanha
  • lessyv

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at November 06, 2014 10:25 AM

October 29, 2014

suricata-ids.org

Suricata Ubuntu PPA updated to 2.0.4

We have updated the official Ubuntu PPA to Suricata 2.0.4. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at October 29, 2014 02:42 PM

Suricata 2.0.4 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.4 release is now available.

Download it here: Suricata-2.0.4-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at October 29, 2014 02:35 PM

October 26, 2014

Peter Manev

Suricata ids/ips - dropping privileges



This tutorial is intended for Linux (Debian/Ubuntu).

Install the prerequisite packages in order to compile Suricata. I add/enable some optional features so in my case I usually do:
apt-get -y install libpcre3 libpcre3-dbg libpcre3-dev \
build-essential autoconf automake libtool libpcap-dev libnet1-dev \
libyaml-0-2 libyaml-dev zlib1g zlib1g-dev make flex bison \
libmagic-dev

For Eve (all JSON output):
apt-get install libjansson-dev libjansson4
For MD5 support(file extraction):
apt-get install libnss3-dev libnspr4-dev
For GeoIP:
apt-get install libgeoip1 libgeoip-dev
For nfqueue(ips mode):
apt-get install libnetfilter-queue-dev libnetfilter-queue1 libnfnetlink-dev libnfnetlink0
For the dropping privileges part you can simply do:
apt-get install libcap-ng0 libcap-ng-dev

OR get the latest libcap-ng version form here:
http://people.redhat.com/sgrubb/libcap-ng/
like so:

wget http://people.redhat.com/sgrubb/libcap-ng/libcap-ng-0.7.4.tar.gz
tar -zxf libcap-ng-0.7.4.tar.gz
cd libcap-ng-0.7.4
./configure && make && make install
cd ..

Let's fetch and compile Suricata:
wget http://www.openinfosecfoundation.org/download/suricata-2.0.4.tar.gz
tar -xzf suricata-2.0.4.tar.gz 
cd suricata-2.0.4
 One liner... one of my favorite:

./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --disable-gccmarch-native \
--enable-geoip --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ \
--enable-nfqueue \
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include \
--with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr && \
make clean && make && make install-full && ldconfig


Above we enable some other features like :
(
you can do like this
root@IDS:~/suricata-2.0.4# ./configure --help
to see what each option is for
)

but this line -
--with-libcap_ng-libraries=/usr/local/lib --with-libcap_ng-includes=/usr/local/include
is the one you need to compile and enable dropping privileges with Suricata.


Then you can run Suri like so
/usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Make sure the log directory has the right permissions to allow the user "logstash" to write to it.
After you start Suricata  - you should see something similar:
root@IDS:~# ls -lh /var/log/suricata/
total 77M
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 certs
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 core
-rw-r----- 1 logstash logstash  18M Oct 26 10:48 eve.json
-rw-r----- 1 logstash logstash 806K Oct 26 10:48 fast.log
drwxr-xr-x 2 logstash logstash 4.0K Oct 15 13:06 files
drwxr-xr-x 2 logstash logstash 4.0K Oct 26 06:26 StatsByDate
-rw-r--r-- 1 root     root      58M Oct 26 10:48 stats.log
-rw-r--r-- 1 root     root     1.1K Oct 26 09:15 suricata-start.log
root@IDS:~#
Notice the user logstash ownership.

root@IDS:~# ps aux |grep suricata
logstash  2189 11.0 10.6 420448 219972 ?       Ssl  09:15  13:04 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash
root@IDS:~#
Now you have the user logstash running (not as root) Suricata IDS/IPS.




by Peter Manev (noreply@blogger.com) at October 26, 2014 09:38 AM

October 24, 2014

Peter Manev

Suricata - preparing 10Gbps network cards for IDPS and file extraction


OS used/tested for this tutorial - Debian Wheezy and/or Ubuntu LTS 12.0.4
With 3.2.0 and 3.5.0 kernel level respectively with Suricata 2.0dev at the moment of this writing.



This article consists of the following major 3 sections:
  • Network card drivers and tuning
  • Kernel specific tunning
  • Suricata.yaml configuration  (file extraction specific)

Network and system  tools:
apt-get install ethtool bwm-ng iptraf htop

Network card drivers and tuning

Our card is Intel 82599EB 10-Gigabit SFI/SFP+


rmmod ixgbe
sudo modprobe ixgbe FdirPballoc=3
ifconfig eth3 up
then (we disable irqbalance and make sure it does not enable itself during reboot)
 killall irqbalance
 service irqbalance stop

 apt-get install chkconfig
 chkconfig irqbalance off
Get the Intel network driver form here (we will use them in a second) - https://downloadcenter.intel.com/default.aspx

 Download to your directory of choice then unzip,compile and install:
 tar -zxf ixgbe-3.18.7.tar.gz
 cd /home/pevman/ixgbe-3.18.7/src
 make clean && make && make install
Set irq affinity - do not forget to change eth3  below with the name of the network interface you are using:
 cd ../scripts/
 ./set_irq_affinity  eth3


 You should see something like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ./set_irq_affinity  eth3
no rx vectors found on eth3
no tx vectors found on eth3
eth3 mask=1 for /proc/irq/101/smp_affinity
eth3 mask=2 for /proc/irq/102/smp_affinity
eth3 mask=4 for /proc/irq/103/smp_affinity
eth3 mask=8 for /proc/irq/104/smp_affinity
eth3 mask=10 for /proc/irq/105/smp_affinity
eth3 mask=20 for /proc/irq/106/smp_affinity
eth3 mask=40 for /proc/irq/107/smp_affinity
eth3 mask=80 for /proc/irq/108/smp_affinity
eth3 mask=100 for /proc/irq/109/smp_affinity
eth3 mask=200 for /proc/irq/110/smp_affinity
eth3 mask=400 for /proc/irq/111/smp_affinity
eth3 mask=800 for /proc/irq/112/smp_affinity
eth3 mask=1000 for /proc/irq/113/smp_affinity
eth3 mask=2000 for /proc/irq/114/smp_affinity
eth3 mask=4000 for /proc/irq/115/smp_affinity
eth3 mask=8000 for /proc/irq/116/smp_affinity
root@suricata:/home/pevman/ixgbe-3.18.7/scripts#
Now we have the latest drivers installed (at the time of this writing) and we have run the affinity script:
   *-network:1
       description: Ethernet interface
       product: 82599EB 10-Gigabit SFI/SFP+ Network Connection
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:04:00.1
       logical name: eth3
       version: 01
       serial: 00:e0:ed:19:e3:e1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list ethernet physical fibre
       configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=3.18.7 duplex=full firmware=0x800000cb latency=0 link=yes multicast=yes port=fibre promiscuous=yes
       resources: irq:37 memory:fbc00000-fbc1ffff ioport:e000(size=32) memory:fbc40000-fbc43fff memory:fa700000-fa7fffff memory:fa600000-fa6fffff



We need to disable all offloading on the network card in order for the IDS to be able to see the traffic as it is supposed to be (without checksums,tcp-segmentation-offloading and such..) Otherwise your IDPS would not be able to see all "natural" network traffic the way it is supposed to and will not inspect it properly.

This would influence the correctness of ALL outputs including file extraction. So make sure all offloading features are OFF !!!

When you first install the drivers and card your offloading settings might look like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
root@suricata:/home/pevman/ixgbe-3.18.7/scripts#

So we disable all of them, like so (and we load balance the UDP flows for that particular network card):

ethtool -K eth3 tso off
ethtool -K eth3 gro off
ethtool -K eth3 ufo off
ethtool -K eth3 lro off
ethtool -K eth3 gso off
ethtool -K eth3 rx off
ethtool -K eth3 tx off
ethtool -K eth3 sg off
ethtool -K eth3 rxvlan off
ethtool -K eth3 txvlan off
ethtool -N eth3 rx-flow-hash udp4 sdfn
ethtool -N eth3 rx-flow-hash udp6 sdfn
ethtool -C eth3 rx-usecs 1 rx-frames 0
ethtool -C eth3 adaptive-rx off

Your output should look something like this:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 tso off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 gro off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 lro off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 gso off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 rx off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 tx off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 sg off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 rxvlan off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -K eth3 txvlan off
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -N eth3 rx-flow-hash udp4 sdfn
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -N eth3 rx-flow-hash udp6 sdfn
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -n eth3 rx-flow-hash udp6
UDP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -n eth3 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -C eth3 rx-usecs 0 rx-frames 0
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -C eth3 adaptive-rx off

Now we doublecheck and run ethtool again to verify that the offloading is OFF:
root@suricata:/home/pevman/ixgbe-3.18.7/scripts# ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off

Ring parameters on the network card:

root@suricata:~# ethtool -g eth3
Ring parameters for eth3:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:            4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512


We can increase that to the max Pre-set RX:

root@suricata:~# ethtool -G eth3 rx 4096

Then we  have a look again:

root@suricata:~# ethtool -g eth3
Ring parameters for eth3:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             512

Making network changes permanent across reboots


On Ubuntu for example you can do:
root@suricata:~# crontab -e

Add the following:
# add cronjob at reboot - disbale network offload
@reboot /opt/tmp/disable-network-offload.sh

and your disable-network-offload.sh script (in this case under /opt/tmp/ ) will contain the following:



#!/bin/bash
ethtool -K eth3 tso off
ethtool -K eth3 gro off
ethtool -K eth3 ufo off
ethtool -K eth3 lro off
ethtool -K eth3 gso off
ethtool -K eth3 rx off
ethtool -K eth3 tx off
ethtool -K eth3 sg off
ethtool -K eth3 rxvlan off
ethtool -K eth3 txvlan off
ethtool -N eth3 rx-flow-hash udp4 sdfn
ethtool -N eth3 rx-flow-hash udp6 sdfn
ethtool -C eth3 rx-usecs 1 rx-frames 0
ethtool -C eth3 adaptive-rx off


with:
chmod 755 disable-network-offload.sh



Kernel specific tunning


Certain adjustments in parameters in the kernel can help as well :

sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.core.rmem_max = 16777216
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.optmem_max=16777216


Making kernel changes permanent across reboots


example:
echo 'net.core.netdev_max_backlog =250000' >> /etc/sysctl.conf

reload the changes:
sysctl -p

OR for all the above adjustments:

echo 'net.core.netdev_max_backlog=250000' >> /etc/sysctl.conf
echo 'net.core.rmem_max = 16777216' >> /etc/sysctl.conf
echo 'net.core.rmem_max=16777216' >> /etc/sysctl.conf
echo 'net.core.rmem_default=16777216' >> /etc/sysctl.conf
echo 'net.core.optmem_max=16777216' >> /etc/sysctl.conf
sysctl -p


Suricata.yaml configuration  (file extraction specific)

As of Suricata 1.2  - it is possible to detect and extract/store over 5000 types of files from HTTP sessions.

Specific file extraction instructions can also be found in the official page documentation.

The following libraries are needed on the system running Suricata :
apt-get install libnss3-dev libnspr4-dev

Suricata also needs to be compiled with file extraction enabled (not covered here).

In short in the suriacta.yaml, those are the sections below that can be tuned/configured and affect the file extraction and logging:
(the bigger the mem values the better on a busy link )


  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh


For file store to disk/extraction:
   - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      force-md5: yes     # force logging of md5 checksums
      #waldo: file.waldo # waldo file to store the file_id across runs


 stream:
  memcap: 32mb
  checksum-validation: no      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 128mb
    depth: 1mb                  # reassemble 1mb into a stream
  
depth: 1mb , would mean that in one tcp reassembled flow, the max size of the file that can be extracted is just about 1mb.

Both stream.memcap and reassembly.memcap (if reassembly is needed) must be big enough to accommodate the whole file on the fly in traffic that needs to be extracted PLUS any other stream and reassembly tasks that the engine needs to do while inspecting the traffic on a particular link.

 app-layer:
  protocols:
....
....
     http:
      enabled: yes
      # memcap: 64mb

The default limit for mem usage for http is 64mb   , that could be increased , ex - memcap: 4GB -  since HTTP is present everywhere and a low memcap on a busy HTTP link would limit the inspection and extraction size ability.

       libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 3072
           response-body-limit: 3072

The default values above control how far the HTTP request and response body is tracked and also limit file inspection. This should be set to a much higher value:

        libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 1gb
           response-body-limit: 1gb

 or 0 (which would mean unlimited):

       libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 0
           response-body-limit: 0

and then of course you would need a rule loaded(example):
alert http any any -> any any (msg:"PDF file Extracted"; filemagic:"PDF document"; filestore; sid:11; rev:11;)



That's it.
























by Peter Manev (noreply@blogger.com) at October 24, 2014 09:37 AM

October 03, 2014

suricata-ids.org

Get Trained January 26 and 27 in San Jose, CA!

Join us for this dynamic, hands-on, 2-day Suricata training event! Developers and security professionals will walk-away with not only a greater proficiency in Suricata’s core technology; but will have the unique opportunity to bring questions, challenges, and new ideas directly to Suricata’s development team.

This training session will take place on January 26 and 27 at the Tilera HQ in San Jose, CA. It will be given by Suricata expert Peter Manev, and OISF president and Emerging Threats CTO Matt Jonkman.

Some of topics that will be covered over the course of the 2-days include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Advanced Tuning
  • Integration with Other Tools

You can register through eventbrite here. More info on the Suricata Training Program can be found here.

This event is generously hosted by our long time supporters: Tilera.

tilera_logo_pms361_plain

We hope to see you there!

by inliniac at October 03, 2014 09:10 AM

September 30, 2014

Security Onion

Suricata 2.0.4

Suricata 2.0.4 was recently released:
http://www.openinfosecfoundation.org/index.php/component/content/article/1-latest-news/198-suricata-204-available

I've packaged Suricata 2.0.4 and it has been tested by David Zawdie (thanks!).

The new package version is:
securityonion-suricata - 2.0.4-0ubuntu0securityonion1

Issues Resolved

Issue 600: Suricata 2.0.4
https://code.google.com/p/security-onion/issues/detail?id=600

Updating
The new packages are now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

This update will back up each of your existing suricata.yaml files to suricata.yaml.bak.  You'll then need to do the following:

  • re-apply any local customizations to suricata.yaml
  • update ruleset and restart Suricata as follows:
    sudo rule-update


Screenshots

Update Process
sudo rule-update

rule-update restarts Suricata

Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Training
Only 16 seats left for the 3-day Security Onion class in Richmond VA!
https://security-onion-class-20141020.eventbrite.com/

Commercial Support
Need commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

We especially need help in answering support questions on the mailing list:
http://groups.google.com/group/security-onion

We also need help testing new packages:
http://groups.google.com/group/security-onion-testing

Thanks!

by Doug Burks (noreply@blogger.com) at September 30, 2014 07:54 AM

September 29, 2014

Victor Julien

Suricata Training Tour

After a lot of preparations, it’s finally going to happen: official Suricata trainings!

In the next couple of months I’ll be doing at least 3 sessions: a home match (Amsterdam), a workshop in Luxembourg and a session at DeepSec. Next to this, we’re planning various US based sessions on the East coast and West coast.

I’m really looking forward to doing these sessions. Other than the official content, there will be plenty of room for questions and discussions.

Hope to see you soon! :)


by inliniac at September 29, 2014 09:25 AM

suricata-ids.org

Get Trained at DeepSec in Vienna

DeepSecLogoJoin us for this dynamic, hands-on, 2-day training session. Developers and security professionals will walk-away with not only a greater proficiency in Suricata’s core technology; but will have the unique opportunity to bring questions, challenges, and new ideas directly to Suricata’s lead developers.

This training session will take place on November 18 and 19 at the DeepSec conference in Vienna . It will be given by Suricata lead developer Victor Julien, OISF president and Emerging Threats CTO Matt Jonkman, Suricata developer Eric Leblond and Suricata expert Peter Manev.

Some of topics that will be covered at this course include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Integration with Other Tools

You can register at the DeepSec conference registration page here.

More info on the Suricata Training Program can be found here.

We hope to see you there!

by inliniac at September 29, 2014 09:16 AM

Open Information Security Foundation

Announcing the Suricata Training Program

The OISF team is proud to announce the start of the Suricata training program. In this program, we’ll be delivering 1 and 2 day user trainings for Suricata.

Some of topics that will be covered over the course of the 2-days include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Advanced Tuning
  • Integration with Other Tools


This dynamic, hands-on, 2 day Suricata training will be delivered by the OISF development and support team.  So apart of the great content on how to install, use and troubleshoot Suricata, you will also have the great opportunity to talk in-depth about Suricata with it’s creators.

Proceeds of the trainings go straight into supporting Suricata’s development, so not only will you learn a great deal, you’ll actually be supporting Suricata’s development by taking this training.


We’re kicking off with 3 training sessions in Europe in the last quarter of 2014. For early 2015, we’re planning to do a number of US trainings. Keep an eye on this space for updates. Also, dedicated on-site training options are available.




Amsterdam, October 13 and 14: 2 day training

This training session will take place on October 13 and 14 in down town Amsterdam. It will be given by Suricata lead developer Victor Julien, and OISF president and Emerging Threats CTO Matt Jonkman. Also in the room: master rule writer William Metcalf.

You can register through eventbrite here: https://www.eventbrite.com/e/suricata-training-event-tickets-13264631871

This event is generously hosted by our friends from Intelworks.

Luxembourg, October 20: 1 day workshop

This workshop will take place on October 20 in the conference hotel of the excellent Hack.lu conference. It will be given by Suricata lead developer Victor Julien, Suricata developer Eric Leblond and Suricata expert Peter Manev.

This event is generously hosted by our friends from Hack.lu. You can register through eventbrite here:
https://www.eventbrite.com/e/suricata-workshop-hacklu-tickets-13329929177

A registration / ticket for the Hack.lu conference is NOT required for this event. Of course, we do highly recommend the conference!


DeepSec - Vienna, November 18 and 19: 2 day training event

This training session will take place on November 18 and 19 at the DeepSec conference. It will be given by Victor Julien, Eric Leblond, Peter Manev and Matt Jonkman.

The event is part of the DeepSec conference, so registrations/bookings go through: https://deepsec.net/register.html

See also http://blog.deepsec.net/?p=1893


Trainings are tracked on their own page here: http://suricata-ids.org/training/. For questions or more info, please contact us at oisf-team@openinfosecfoundation.org!

by Victor Julien (postmaster@inliniac.net) at September 29, 2014 08:43 AM

September 25, 2014

suricata-ids.org

Get Trained at Hack.lu in Luxembourg

Join us for this dynamic, hands-on, full day  Suricata workshop! Developers and security professionals will walk-away with not only a greater proficiency in Suricata’s core technology; but will have the unique opportunity to bring questions, challenges, and new ideas directly to Suricata’s lead developers.

This workshop will take place on October 20 in the conference hotel of the excellent Hack.lu conference. It will be given by Suricata lead developer Victor Julien, Suricata developer Eric Leblond and Suricata expert Peter Manev.

Some of topics that will be covered at this course include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Integration with Other Tools

You can register through eventbrite here: https://www.eventbrite.com/e/suricata-workshop-hacklu-tickets-13329929177240pxlogohacklu2014.

More info on the Suricata Training Program can be found here.

This event is generously hosted by our friends from Hack.lu.

A registration / ticket for the Hack.lu conference is NOT required for this event. Of course, we do highly recommend the conference!

We hope to see you there!

by inliniac at September 25, 2014 03:45 PM

September 23, 2014

suricata-ids.org

Get Trained in Amsterdam!

Join us for this dynamic, hands-on, 2-day Suricata training event! Developers and security professionals will walk-away with not only a greater proficiency in Suricata’s core technology; but will have the unique opportunity to bring questions, challenges, and new ideas directly to Suricata’s lead developers.

This training session will take place on October 13 and 14 in down town Amsterdam. It will be given by Suricata lead developer Victor Julien, and OISF president and Emerging Threats CTO Matt Jonkman.

Some of topics that will be covered over the course of the 2-days include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Advanced Tuning
  • Integration with Other Tools

You can register through eventbrite here: https://www.eventbrite.com/e/suricata-training-event-tickets-13264631871. More info on the Suricata Training Program can be found here.

This event is generously hosted by our friends from Intelworks.

We hope to see you there!

by inliniac at September 23, 2014 06:05 PM

Announcing the Suricata Training Program

The OISF team is proud to announce the start of the Suricata training program. In this program, we’ll be delivering 1 and 2 day user trainings for Suricata.

Some of topics that will be covered over the course of the 2-days include:

  • Compiling, Installing, and Configuring Suricata
  • Performance Factors, Rules and Rulesets
  • Capture Methods and Performance
  • Event / Data Outputs and Capture Hardware
  • Troubleshooting Common Problems
  • Advanced Tuning
  • Integration with Other Tools

This dynamic, hands-on, 2 day Suricata training will be delivered by the OISF development and support team.  So apart of the great content on how to install, use and troubleshoot Suricata, you will also have the great opportunity to talk in-depth about Suricata with it’s creators.

Proceeds of the trainings go straight into supporting Suricata’s development, so not only will you learn a great deal, you’ll actually be supporting Suricata’s development by taking this training.

We’re kicking off with 3 training sessions in Europe in the last quarter of 2014. For early 2015, we’re planning to do a number of US trainings. Keep an eye on this space for updates. Also, dedicated on-site training options are available.

Trainings are tracked on their own page here. For questions or more info, please contact us!

by inliniac at September 23, 2014 05:04 PM

Open Information Security Foundation

Suricata 2.0.4 Available!

The OISF development team is pleased to announce Suricata 2.0.4. This release fixes a number of important issues in the 2.0 series.

This update fixes a bug in the SSH parser, where a malformed banner could lead to evasion of SSH rules and missing log entries. In some cases it may also lead to a crash. Bug discovered and reported by Steffen Bauch.

Additionally, this release also addresses a new IPv6 issue that can lead to evasion. Bug discovered by Rafael Schaefer working with ERNW GmbH.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.4.tar.gz

Changes

  • Bug #1276: ipv6 defrag issue with routing headers
  • Bug #1278: ssh banner parser issue
  • Bug #1254: sig parsing crash on malformed rev keyword
  • Bug #1267: issue with ipv6 logging
  • Bug #1273: Lua – http.request_line not working
  • Bug #1284: AF_PACKET IPS mode not logging drops and stream inline issue

Security

  • CVE-2014-6603

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at September 23, 2014 11:21 AM

suricata-ids.org

Suricata 2.0.4 Available!

Photo by Eric Leblond

The OISF development team is pleased to announce Suricata 2.0.4. This release fixes a number of important issues in the 2.0 series.

This update fixes a bug in the SSH parser, where a malformed banner could lead to evasion of SSH rules and missing log entries. In some cases it may also lead to a crash. Bug discovered and reported by Steffen Bauch.

Additionally, this release also addresses a new IPv6 issue that can lead to evasion. Bug discovered by Rafael Schaefer working with ERNW GmbH.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.4.tar.gz

Changes

  • Bug #1276: ipv6 defrag issue with routing headers
  • Bug #1278: ssh banner parser issue
  • Bug #1254: sig parsing crash on malformed rev keyword
  • Bug #1267: issue with ipv6 logging
  • Bug #1273: Lua – http.request_line not working
  • Bug #1284: AF_PACKET IPS mode not logging drops and stream inline issue

Security

  • CVE-2014-6603

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at September 23, 2014 11:20 AM

August 26, 2014

Security Onion

New PF_RING, Snort, Suricata, Bro packages

New versions of our PF_RING, Snort, Suricata, and Bro packages are now available!  The new package versions are as follows:

securityonion-bro - 2.3-0ubuntu0securityonion10
securityonion-bro-scripts - 20121004-0ubuntu0securityonion26
securityonion-daq - 2.0.2-0ubuntu0securityonion5
securityonion-elsa-extras - 20131117-1ubuntu0securityonion43
securityonion-pfring-daq - 20121107-0ubuntu0securityonion7
securityonion-pfring-devel - 20121107-0ubuntu0securityonion7
securityonion-pfring-ld - 20120827-0ubuntu0securityonion7
securityonion-pfring-module - 20121107-0ubuntu0securityonion23
securityonion-pfring-userland - 20140805-0ubuntu0securityonion3
securityonion-snort - 2.9.6.2-0ubuntu0securityonion7
securityonion-suricata - 2.0.3-0ubuntu0securityonion2

These new packages have been tested by the following (thanks!):
Ronny Vaningh
Andrea De Pasquale
Pete Nelson
Pietro Delsante
David Zawdie
Heine Lysemose
Eddy Simons

Issues Resolved

Issue 535: PF_RING 6.0.2 SVN
https://code.google.com/p/security-onion/issues/detail?id=535

Issue 462: Snort 2.9.6.2
https://code.google.com/p/security-onion/issues/detail?id=462

Issue 567: Snort Daq 2.0.2
https://code.google.com/p/security-onion/issues/detail?id=567

Issue 465: Suricata 2.0.3
https://code.google.com/p/security-onion/issues/detail?id=465

Issue 445: Bro 2.3
https://code.google.com/p/security-onion/issues/detail?id=445

Issue 484: securityonion-bro-scripts: update APT1 scripts with Seth's changes for certificate matching
https://code.google.com/p/security-onion/issues/detail?id=484

Issue 414: Bro script should lookup interface in /etc/nsm/sensortab to obtain sensorname
https://code.google.com/p/security-onion/issues/detail?id=414

Issue 577: ELSA: update parsers for Bro 2.3 log changes
https://code.google.com/p/security-onion/issues/detail?id=577

Updating
The new packages are now available in our stable repo.  Please see the following page for full update instructions:
https://code.google.com/p/security-onion/wiki/Upgrade

These updates will do the following:

  • back up your Bro configuration
  • back up each of your existing snort.conf files to snort.conf.bak
  • back up each of your existing suricata.yaml files to suricata.yaml.bak

You'll then need to do the following:
  • re-apply any local customizations to the Bro/Snort/Suricata config
  • restart Bro as follows:
sudo nsm_sensor_ps-restart --only-bro
  • update ruleset and restart Snort/Suricata as follows:
sudo rule-update

Screenshots
Run "sudo soup" which first installs the new PF_RING kernel module

DKMS compiles the new kernel module

Soup then installs the remaining packages

Bro, Snort, and Suricata notify you that config files have been updated and you'll need to add back any local customizations

After adding back any local Bro customizations, restart Bro using "sudo nsm_sensor_ps-restart --only-bro"

After adding back any local snort.conf or suricata.yaml customizations, run "sudo rule-update" to download the latest ruleset for the new IDS engine

rule-update then restarts Barnyard2 and the IDS engine



Feedback
If you have any questions or problems, please use our security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Conference
Less than 30 seats left for the Security Onion conference in Augusta GA! Reserve your seat today!
https://securityonionconference2014.eventbrite.com

Commercial Support/Training
Need training and/or commercial support?  Please see:
http://securityonionsolutions.com

Help Wanted
If you and/or your organization have found value in Security Onion, please consider giving back to the community by joining one of our teams:
https://code.google.com/p/security-onion/wiki/TeamMembers

We especially need help in answering support questions on the mailing list:
http://groups.google.com/group/security-onion

We also need help testing new packages:
http://groups.google.com/group/security-onion-testing

Thanks!

by Doug Burks (noreply@blogger.com) at August 26, 2014 02:02 PM

August 24, 2014

Peter Manev

Suricata - more data for your alerts


As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of packet data and information in a standard JSON output logging capability supplementing further the alert logging output.

This guide makes use of Suricata and ELK - Elasticsearch, Logstash, Kibana.
You can install all of them following the guide HERE
 ...or you can download and try out SELKS  and use directly.


After everything is in place, we need to open the suricata.yaml and make the following editions in the eve.json section:

 # "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert:

            payload: yes           # enable dumping payload in Base64
            payload-printable: yes # enable dumping payload in printable (lossy) format
            packet: yes            # enable dumping of packet (without stream segments)
            http: yes              # enable dumping of http fields
       
You can start Suricata and let it inspect traffic for some time in order to generate alert log data.
Then navigate to your Kibana web interface, find an alert record/log and you could see the usefulness of the extra data yourself.

Some examples though :) :






Lets kick it up  notch.....

We want to search through -
  1. all the generated alerts that have 
  2. a printable payload data 
  3. that have the following string: uid=0(root)
 Easy, here is the query:
 
payload_printable:"uid=0\(root\)"
You should enter it like this in Kibana:



Well what do you know - we got what we were looking for:




Some more useful reading on the Lucene Query Syntax (you should at least have a look :) ):
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

http://www.solrtutorial.com/solr-query-syntax.html

http://lucene.apache.org/core/2_9_4/queryparsersyntax.html






by Peter Manev (noreply@blogger.com) at August 24, 2014 04:03 AM

Suricata - Flows, Flow Managers and effect on performance



As of Suricata 2.1beta1  - Suricata IDS/IPS provides the availability of high performance/advanced tuning for custom thread configuration for the IDS/IPS engine management threads.

Aka ..these
[27521] 20/7/2014 -- 01:46:19 - (tm-threads.c:2206) <Notice>  (TmThreadWaitOnThreadInit) -- all 16 packet processing threads, 3 management threads initialized, engine started.


These 3 management threads initialized above are flow manager (1), counter/stats related threads (2x)

So ... in the default suricata.yaml setting we have:

flow:
  memcap: 64mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  #managers: 1 # default to one flow manager
  #recyclers: 1 # default to one flow recycler thread

and we can choose accordingly of how many threads we would like to dedicate for the management tasks within the engine itself.
The recyclers threads offload part of the flow managers work and if enabled do flow/netflow logging.

Good !
What does this has to do with performance?

Suricata IDS/IPS is powerful, flexible and scalable - so be careful what you wish for.
The examples below demonstrate the effect on a 10Gbps Suricata IDS sensor.

Example 1


suricata.yaml config - >
flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30
  prune-flows: 50000
  managers: 2 # default is 1

CPU usage ->

 2 flow management threads use 8% CPU each

 Example 2



suricata.yaml config - >
flow:
  memcap: 4gb
  hash-size: 15728640
  prealloc: 8000000
  emergency-recovery: 30
  managers: 2 # default is 1

 CPU usage ->

2 flow management threads use 39% CPU each as compared to Example 1 !!


So a 4 fold increase in memcap, 8 fold increase in prealloc and 15 fold increase on hash-size settings leads to about 3 fold increase in RAM consumption and 5 fold on CPU consumption  - in terms of flow management thread usage.

It would be very rare that you would need the settings in Example 2 - you need huge traffic for that...

So how would you know when to tune/adjust those settings in suricata.yaml? It is recommended that you always keep an eye on your stats.log and make sure you do not enter emergency clean up mode:



it should always be 0

Some additional reading on flows and flow managers -
http://blog.inliniac.net/2014/07/28/suricata-flow-logging/




by Peter Manev (noreply@blogger.com) at August 24, 2014 04:01 AM

Suricata - filtering tricks for the fileinfo output with eve.json


As of Suricata 2.0  - Suricata IDS/IPS provides the availability of a standard JSON output logging capability. This guide makes use of Suricata and ELK - Elasticsearch, Logstash, Kibana.

You can install all of them following the guide HERE
 ...or you can download and try out SELKS  and use directly.

Once you have the installation in place and have the Kibana web interface up and running you can make use of the following fileinfo filters (tricks :).
You can enter the queries like so:



 fileinfo.magic:"PE32" -fileinfo.filename:*exe
will show you all "PE32 executable" executables that were seen transferred that have no exe extension in their file name:




 Alternatively
fileinfo.magic:"pdf" -fileinfo.filename:*pdf


will show you all "PDF document version......" files that were transferred that have no PDF extension in their file name.

You can explore further :)






by Peter Manev (noreply@blogger.com) at August 24, 2014 03:47 AM

August 23, 2014

Peter Manev

Suricata IDS/IPS - HTTP custom header logging


As a continuation of the article HERE- some more screenshots from the ready to use template....

For the Elasticsearch/Logstash/Kibana users there is a ready to use template that you could download from here - "HTTP-Extended-Custom"
https://github.com/pevma/Suricata-Logstash-Templates














by Peter Manev (noreply@blogger.com) at August 23, 2014 06:11 AM

August 20, 2014

suricata-ids.org

Suricata Ubuntu PPA updated to 2.1beta1

We have updated the official Ubuntu PPA to Suricata 2.1beta1. To use this PPA read our docs here.

If you’re using this PPA, updating is as simple as:

apt-get update && apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at August 20, 2014 12:31 PM

Suricata 2.1beta1 Windows Installer Available

The Windows MSI installer of the Suricata 2.1beta1 release is now available.

Download it here: suricata-2.1beta1-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at August 20, 2014 12:15 PM

Suricata Ubuntu PPA updated to 2.0.3

We have updated the official Ubuntu PPA to Suricata 2.0.3. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at August 20, 2014 12:05 PM

Suricata 2.0.3 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.3 release is now available.

Download it here: Suricata-2.0.3-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at August 20, 2014 12:01 PM

August 12, 2014

Open Information Security Foundation

Suricata 2.1beta1 Available!

The OISF development team is proud to announce Suricata 2.1beta1. This is the first beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta1.tar.gz

New Features

  • Feature #1248: flow/connection logging
  • Feature #1155 & #1208: Log packet payloads in eve alerts

Improvements

  • Optimization #1039: Packetpool should be a stack
  • Optimization #1241: pcap recording: record per thread
  • Feature #1258: json: include HTTP info with Alert output
  • AC matcher start up optimizations
  • BM matcher runtime optimizations by Ken Steele

Removals

  • ‘pcapinfo’ output was removed. Suriwire now works with the JSON ‘eve’ output

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Matt Carothers
  • Alexander Gozman
  • Giuseppe Longo
  • Duarte Silva
  • sxhlinux

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at August 12, 2014 02:54 PM

suricata-ids.org

Suricata 2.1beta1 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.1beta1. This is the first beta release for the upcoming 2.1 version. It should be considered a development snapshot for the 2.1 branch.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.1beta1.tar.gz

New Features

  • Feature #1248: flow/connection logging
  • Feature #1155 & #1208: Log packet payloads in eve alerts

Improvements

  • Optimization #1039: Packetpool should be a stack
  • Optimization #1241: pcap recording: record per thread
  • Feature #1258: json: include HTTP info with Alert output
  • AC matcher start up optimizations
  • BM matcher runtime optimizations by Ken Steele

Removals

  • ‘pcapinfo’ output was removed. Suriwire now works with the JSON ‘eve’ output

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Matt Carothers
  • Alexander Gozman
  • Giuseppe Longo
  • Duarte Silva
  • sxhlinux

Known issues & missing features

In a beta release like this things may not be as polished yet. So please handle with care. That said, if you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at August 12, 2014 02:33 PM

August 10, 2014

Peter Manev

HTTP Header fields extended logging with Suricata IDPS


With the release of Suricata 2.0.1 there is availability and option to do extended custom HTTP header fields logging through the JSON output module.

For the Elasticsearch/Logstash/Kibana users there is a ready to use template that you could download from here -
https://github.com/pevma/Suricata-Logstash-Templates

So what does this mean?

Well besides the standard http logging in the eve.json you also get 47 additional HTTP fields logged, mainly these:
accept
accept-charset
accept-encoding
accept-language
accept-datetime
authorization
cache-control
cookie
from
max-forwards
origin
pragma
proxy-authorization
range
te
via
x-requested-with
dnt
x-forwarded-proto
accept-range
age
allow
connection
content-encoding
content-language
content-length
content-location
content-md5
content-range
content-type
date
etags
last-modified
link
location
proxy-authenticate
referrer
refresh
retry-after
server
set-cookie
trailer
transfer-encoding
upgrade
vary
warning
www-authenticate

What they are and what they mean/affect you could read more about here:
http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

You can choose any combination of those fields above or all of them. What you need to do is simply add those to the existing logging in suricata.yaml's eve section. To add all of them if found in the HTTP traffic you could do like so:
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional http fields to be included in eve-log
            # the example below adds three additional fields when uncommented
            #custom: [Accept-Encoding, Accept-Language, Authorization]
            custom: [accept, accept-charset, accept-encoding, accept-language,
            accept-datetime, authorization, cache-control, cookie, from,
            max-forwards, origin, pragma, proxy-authorization, range, te, via,
            x-requested-with, dnt, x-forwarded-proto, accept-range, age,
            allow, connection, content-encoding, content-language,
            content-length, content-location, content-md5, content-range,
            content-type, date, etags, last-modified, link, location,
            proxy-authenticate, referrer, refresh, retry-after, server,
            set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
            www-authenticate]
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh
Then you just start Suricata.

What is the benefit?

You can log and search/filter/select through any or all of those 60 or so http header fields. JSON is a standard format - so depending on what you are using for DB and/or search engine, you could get very easy interesting and very helpful statistics that would help your security teams.

Some possible stats using Elasticsearch and Kibana
(how to set up Elasticsearch, Logstash and Kibana with Suricata)-











by Peter Manev (noreply@blogger.com) at August 10, 2014 06:36 AM

August 08, 2014

Open Information Security Foundation

Suricata 2.0.3 Available!

The OISF development team is proud to announce Suricata 2.0.3. This release fixes a number of issues in the 2.0 series. Most importantly, this release addresses a number of IPv6 issues that can lead to evasion. Bugs discovered by Rafael Schaefer working with ERNW GmbH.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.3.tar.gz

Changes

  • Bug #1236: fix potential crash in http parsing
  • Bug #1244: ipv6 defrag issue
  • Bug #1238: Possible evasion in stream-tcp-reassemble.c
  • Bug #1221: lowercase conversion table missing last value
  • Support #1207: Cannot compile on CentOS 5 x64 with –enable-profiling
  • Updated bundled libhtp to 0.5.15

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Rafael Schaefer working with ERNW GmbH
  • Antonios Atlasis working with ERNW GmbH
  • Alexander Gozman
  • sxhlinux
  • Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at August 08, 2014 09:40 AM

suricata-ids.org

Suricata 2.0.3 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.0.3. This release fixes a number of issues in the 2.0 series. Most importantly, this release addresses a number of IPv6 issues that can lead to evasion. Bugs discovered by Rafael Schaefer working with ERNW GmbH.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.3.tar.gz

Changes

  • Bug #1236: fix potential crash in http parsing
  • Bug #1244: ipv6 defrag issue
  • Bug #1238: Possible evasion in stream-tcp-reassemble.c
  • Bug #1221: lowercase conversion table missing last value
  • Support #1207: Cannot compile on CentOS 5 x64 with –enable-profiling
  • Updated bundled libhtp to 0.5.15

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Rafael Schaefer working with ERNW GmbH
  • Antonios Atlasis working with ERNW GmbH
  • Alexander Gozman
  • sxhlinux
  • Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at August 08, 2014 09:39 AM

August 01, 2014

Security Onion

PF_RING, Snort, and Suricata packages have reached Release Candidate status!

Our new PF_RING/Snort/Suricata packages have reached Release Candidate status!  Since these packages are critical components, I'd like to do one final phase of testing before promoting to stable.  If at all possible, please try installing on some of your production sensors so that we can get some real world testing before promoting to stable.

Join the discussion here:
https://groups.google.com/d/topic/security-onion-testing/mKVn-GAPaIg/discussion

by Doug Burks (noreply@blogger.com) at August 01, 2014 11:42 AM

July 28, 2014

Victor Julien

Suricata Flow Logging

Pretty much from the start of the project, Suricata has been able to track flows. In Suricata the term ‘flow’ means the bidirectional flow of packets with the same 5 tuple. Or 7 tuple when vlan tags are counted as well.

Such a flow is created when the first packet comes in and is stored in the flow hash. Each new packet does a hash look-up and attaches the flow to the packet. Through the packet’s flow reference we can access all that is stored in the flow: TCP session, flowbits, app layer state data, protocol info, etc.

When a flow hasn’t seen any packets in a while, a separate thread times it out. This ‘Flow Manager’ thread constantly walks the hash table and looks for flows that are timed out. The time a flow is considered ‘active’ depends on the protocol, it’s state and the configuration settings.

In Suricata 2.1, flows will optionally be logged when they time out. This logging is available through a new API, with an implementation for ‘Eve’ JSON output already developed. Actually, 2 implementations:

  1. flow — logs bidirectional records
  2. netflow — logs unidirectional records

As the flow logging had to be done at flow timeout, the Flow Manager had to drive it. Suricata 2.0 and earlier had a single Flow Manager thread. This was hard coded, and in some cases it was clearly a bottleneck. It wasn’t uncommon to see this thread using more CPU than the packet workers.

So adding more tasks to the Flow Manager, especially something as expensive as output, was likely going to make things worse. To address this, 2 things are now done:

  1. multiple flow manager support
  2. offloading of part of the flow managers tasks to a new class of management threads

The multiple flow managers simply divide up the hash table. Each thread manages it’s own part of it. The new class of threads is called ‘Flow Recycler’. It takes care of the actual flow cleanup and recycling. This means it’s taking over a part of the old Flow Manager’s tasks. In addition, if enabled, these threads are tasked with performing the actual flow logging.

As the flow logging follows the ‘eve’ format, passing it into Elasticsearch, Logstash and Kibana (ELK) is trivial. If you already run such a setup, the only thing that is need is enabling the feature in your suricata.yaml.

kibana-flow

kibana-netflowThe black netflow dashboard is available here: http://www.inliniac.net/files/NetFlow.json

Many thanks to the FireEye Forensics Group (formerly nPulse Technologies) for funding this work.


by inliniac at July 28, 2014 10:05 PM

Peter Manev

Granularity in advance memory tuning for segments and http processing with Suricata


Just recently(at the time of this writing) in the dev branch of Suricata IDPS (git) were introduced a few new config  options for the suricata.yaml

stream:
  memcap: 32mb
  checksum-validation: yes      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    randomize-chunk-range: 10
    raw: yes
    chunk-prealloc: 250 #size 4KB
    segments:
      - size: 4
        prealloc: 256
      - size: 16
        prealloc: 512
      - size: 112
        prealloc: 512
      - size: 248
        prealloc: 512
      - size: 512
        prealloc: 512
      - size: 768
        prealloc: 1024
      - size: 1448
        prealloc: 1024
      - size: 65535
        prealloc: 128


and under the app layer protocols section (in suricata.yaml) ->

    http:
      enabled: yes
      # memcap: 64mb



Stream segments memory preallocation - config option

This first one gives you an advance granular control over your memory consuption in terms or preallocating memory for segmented packets that are going through the stream reassembly engine ( of certain size)

The patch's info:
commit b5f8f386a37f61ae0c1c874b82f978f34394fb91
Author: Victor Julien <victor@inliniac.net>
Date:   Tue Jan 28 13:48:26 2014 +0100

    stream: configurable segment pools
   
    The stream reassembly engine uses a set of pools in which preallocated
    segments are stored. There are various pools each with different packet
    sizes. The goal is to lower memory presure. Until now, these pools were
    hardcoded.
   
    This patch introduces the ability to configure them fully from the yaml.
    There can be at max 256 of these pools.

In other words to speed things up in Suricata, you could do some traffic profiling with the iptraf tool (apt-get install iptraf , then select "Statistical breakdowns", then select "By packet size", then the appropriate interface):

So partly based on the pic above (one also should determine the packet breakdown from TCP perspective) you could do some adjustments in the default config section in suricata.yaml:

segments:
      - size:4
        prealloc: 256
      - size:74
        prealloc: 65535
      - size: 112
        prealloc: 512
      - size: 248
        prealloc: 512
      - size: 512
        prealloc: 512
      - size: 768
        prealloc: 1024
      - size: 1276
        prealloc: 65535
      - size: 1425
        prealloc: 262140
      - size: 1448
        prealloc: 262140
      - size: 9216 
        prealloc: 65535 
      - size: 65535
        prealloc: 9216


Make sure you calculate your memory, this all falls under the stream reassembly memcap set in the yaml. So naturally it would have to be  big enough to accommodate those changes :).
For example the changes in bold above would need 1955 MB of RAM from the stream reassembly value set in the suricata.yaml. So for example if the values is set like so:
stream:
  memcap: 2gb
  checksum-validation: yes      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 4gb
it will use 1955MB for prealloc segment  packets and there will be roughly 2gb left for the other reassembly tasks - like for example allocating segments and chunks that were not  prealloc in the  settings.

If you would like to be exact - you can run Suricata with the -v switch to enable verbosity, thus giving you an exact picture of what your segments numbers are (for example: run it for a 24 hrs and after you stop it with kill -15 pid_of_suricata):

(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 4 had a peak use of 96199 segments, more than the prealloc setting of 256
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 16 had a peak use of 28743 segments, more than the prealloc setting of 512
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 112 had a peak use of 96774 segments, more than the prealloc setting of 512
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 248 had a peak use of 25833 segments, more than the prealloc setting of 512
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 512 had a peak use of 24354 segments, more than the prealloc setting of 512
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 768 had a peak use of 30954 segments, more than the prealloc setting of 1024
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 1448 had a peak use of 139742 segments, more than the prealloc setting of 1024
(stream-tcp-reassemble.c:502) <Info> (StreamTcpReassembleFree) -- TCP segment pool of size 65535 had a peak use of 139775 segments, more than the prealloc setting of 128
(stream.c:182) <Info> (StreamMsgQueuesDeinit) -- TCP segment chunk pool had a peak use of 21676 chunks, more than the prealloc setting of 250
So then you can adjust the values accordingly for all segments.


HTTP memcap option

In suricata .yaml you can set an explicit limit for the http usage of memory in the inspection engine.

    http:
      enabled: yes
      memcap: 4gb

Those two config option do add some more powerful ways of fine tuning the already highly flexible Suricata IDPS capabilities.


Of course when setting memcaps in the suricata.yaml you would have to make sure you have the total available RAM on your server/machine....otherwise funny things happen :)

by Peter Manev (noreply@blogger.com) at July 28, 2014 01:46 AM

July 08, 2014

suricata-ids.org

Suricata 2.0.2 Windows Installer Available

The Windows MSI installer of the Suricata 2.0.2 release is now available.

Download it here: Suricata-2.0.2-1-32bit.msi

After downloading, double click the file to launch the installer. The installer is now signed.

If you have a previous version installed, please remove that first.

by fleurixx at July 08, 2014 03:32 PM

Suricata Ubuntu PPA updated to 2.0.2

We have updated the official Ubuntu PPA to Suricata 2.0.2. To use this PPA read our docs here.

To install Suricata through this PPA, enter:
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

If you’re already using this PPA, updating is as simple as:
sudo apt-get update && sudo apt-get upgrade

The PPA Ubuntu packages have IPS mode through NFQUEUE enabled.

by fleurixx at July 08, 2014 03:19 PM

June 30, 2014

Anoop Saldanha

Passing a OpenCL cl_mem device address from host to the device, but not as a kernel argument. Pointers for Suricata OpenCL porters.


This post is not specific to Suricata, but rather a generic one, that can help most devs who write OpenCL code + the ones who want to implement OpenCL support inside suricata.  Have been seeing quite a few attempts on porting suricata's CUDA support to use OpenCL.  Before we experimented with CUDA, we had given OpenCL a shot back in the early OpenCL days, when the drivers were in it's infancy and had a ton of bugs, and we, a ton of segvs, leaving us with no clue as to where the bug was - the driver or the code.  The driver might be a lot stabler today, of course.

Either ways, supporting OpenCL in suricata  should be a pretty straightforward task, but there's one issue that needs to be kept in mind while carrying out this port.  Something most folks who contacted me during their port, got stuck at.  And also a question a lot of OpenCL devs have on passing a memory object as a part of a byte stream, structure and not as a kernel argument.

Let's get to the topic at hand.  I will use the example of suricata to explain the issue.

What's the issue?

Suricata buffers a payload and along with the payload, specifies a gpu memory address(cl_mem) that points to the pattern matching state table that the corresponding payload should be matched against.  With CUDA the memory address we are buffering is of type "CUdeviceptr", that is allocated using the call cuMemAlloc().  The value stored inside CUdeviceptr is basically an address from the gpu address space(not a handle).  You can test this by writing a simple program like the one I have below for OpenCL.  You can also check this article that confirms the program's findings.

With OpenCL, cl_mem is defined to be a handle against an address in the gpu address space.  I would have expected Nvidia'a OpenCL implementation to show a behaviour that was similar to it's cuda library, i.e. the handle being nothing but an address in the gpu address space, but it isn't the case(probably has something to do do with the size of cl_mem?).  We can't directly pass the cl_mem handle value as the device address.  We will need to extract the device address out for a particular cl_mem handle, and pass this retrieved value instead.

Here is a sample program -

==get_address.cu==

__kernel void get_address(__global ulong *c)
{
    *c = (ulong)c;
}

==get_address.c==

unsigned long get_address(cl_kernel kernel_address,
                                            cl_command_queue command_queue,
                                            cl_mem dst_mem)
{
    unsigned long result_address = 0;

    BUG_ON(clSetKernelArg(kernel_address, 0,
                                             sizeof(dst_mem), &dst_mem) < 0);

    BUG_ON(clEnqueueNDRangeKernel(command_queue,
                                                                kernel_address,
                                                                1,
                                                                NULL,
                                                                &global_work_size,
                                                                &local_work_size,
                                                                0, NULL,
                                                                 NULL) < 0);
    BUG_ON(clEnqueueReadBuffer(command_queue,
                                                        dst_mem,
                                                        CL_TRUE,
                                                        0,
                                                        sizeof(result_address),
                                                        &result_address,
                                                        0, NULL,
                                                         NULL) < 0);
    return result_address;
}

* Untested code.  Code written keeping in mind a 64 bit hardware on the gpu and the cpu.

Using the above get_address() function should get you the gpu address for a cl_mem instance, and the returned value is what should be passed to the gpu as the address, in place of CUDA's CUdeviceptr.  It's sort of a hack, but it should work.

Another question that pops up in my head is, would the driver change the memory allocated against a handle?  Any AMD/Nvidia driver folks can answer this?

Any alternate solutions(apart from passing all of it as kernel arguments :) ) welcome.


by poona (noreply@blogger.com) at June 30, 2014 12:03 AM

June 26, 2014

Eric Leblond

pshitt: collect passwords used in SSH bruteforce

Introduction

I’ve been playing lately on analysis SSH bruteforce caracterization. I was a bit frustrated of just getting partial information:

  • ulogd can give information about scanner settings
  • suricata can give me information about software version
  • sshd server logs shows username
But having username without having the password is really frustrating.

So I decided to try to get them. Looking for a SSH server honeypot, I did find kippo but it was going too far for me by providing a fake shell access. So I’ve decided to build my own based on paramiko.

pshitt, Passwords of SSH Intruders Transferred to Text, was born. It is a lightweight fake SSH server that collect authentication data sent by intruders. It basically collects username and password and writes the extracted data to a file in JSON format. For each authentication attempt, pshitt is dumping a JSON formatted entry:

{"username": "admin", "src_ip": "116.10.191.236", "password": "passw0rd", "src_port": 36221, "timestamp": "2014-06-26T10:48:05.799316"}
The data can then be easily imported in Logstash (see pshitt README) or Splunk.

The setup

As I want to really connect to the box running ssh with a regular client, I needed a setup to automatically redirect the offenders and only them to pshitt server. A simple solution was to used DOM. DOM parses Suricata EVE JSON log file in which Suricata gives us the software version of IP connecting to the SSH server. If DOM sees a software version containing libssh, it adds the originating IP to an ipset set. So, the idea of our honeypot setup is simple:
  • Suricata outputs SSH software version to EVE
  • DOM adds IP using libssh to the ipset set
  • Netfilter NAT redirects all IP off the set to pshitt when they try to connect to our ssh server
Getting the setup in place is really easy. We first create the set:
ipset create libssh hash:ip
then we start DOM so it adds all offenders to the set named libssh:
cd DOM
./dom -f /usr/local/var/log/suricata/eve.json -s libssh
A more accurate setup for dom can be the following. If you know that your legitimate client are only based on OpenSSH then you can run dom to put in the list all IP that do not (-i) use an OpenSSH client (-m OpenSSh):
./dom -f /usr/local/var/log/suricata/eve.json -s libssh -vvv -i -m OpenSSH
If we want to list the elements of the set, we can use:
ipset list libssh
Now, we can start pshitt:
cd pshitt
./pshitt
And finally we redirect the connection coming from IP of the libssh set to the port 2200:
iptables -A PREROUTING -m set --match-set libssh src -t nat -i eth0 -p tcp -m tcp --dport 22 -j REDIRECT --to-ports 2200

Some results

Here’s an extract of the most used passwords when trying to get access to the root account: real root passwords And here’s the same thing for the admin account attempt: Root passwords Both data show around 24 hours of attempts on an anonymous box.

Conclusion

Thanks to paramiko, it was really fast to code pshitt. I’m now collecting data and I think that they will help to improve the categorization of SSH bruteforce tools.

by Regit at June 26, 2014 08:41 AM

June 25, 2014

Open Information Security Foundation

Suricata 2.0.2 Available!

The OISF development team is proud to announce Suricata 2.0.2. This release fixes a number of issues in the 2.0 series.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.2.tar.gz

Notable changes

  • IP defrag issue leading to evasion. Bug discovered by Antonios Atlasis working with ERNW GmbH
  • Support for NFLOG as a capture method. Nice work by Giuseppe Longo
  • DNS TXT parsing and logging. Funded by Emerging Threats
  • Log rotation through SIGHUP. Created by Jason Ish of Endace/Emulex

All closed tickets

  • Feature #781: IDS using NFLOG iptables target
  • Feature #1158: Parser DNS TXT data parsing and logging
  • Feature #1197: liblua support
  • Feature #1200: sighup for log rotation
  • Bug #1098: http_raw_uri with relative pcre parsing issue
  • Bug #1175: unix socket: valgrind warning
  • Bug #1189: abort() in 2.0dev (rev 6fbb955) with pf_ring 5.6.3
  • Bug #1195: nflog: cppcheck reports memleaks
  • Bug #1206: ZC pf_ring not working with Suricata 2.0.1 (or latest git)
  • Bug #1211: defrag issue
  • Bug #1212: core dump (after a while) when app-layer.protocols.http.enabled = yes
  • Bug #1214: Global Thresholds (sig_id 0, gid_id 0) not applied correctly if a signature has event vars
  • Bug #1217: Segfault in unix-manager.c line 529 when using –unix-socket and sending pcap files to be analized via socket

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Jason Ish — Endace/Emulex
  • Tom Decanio — nPulse
  • Antonios Atlasis working with ERNW GmbH
  • Alessandro Guido
  • Mats Klepsland
  • @rmkml
  • Luigi Sandon
  • Christie Bunlon
  • @42wim
  • Jeka Pats
  • Noam Meltzer
  • Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by Victor Julien (postmaster@inliniac.net) at June 25, 2014 04:27 PM

suricata-ids.org

Suricata 2.0.2 Available!

Photo by Eric Leblond

The OISF development team is proud to announce Suricata 2.0.2. This release fixes a number of issues in the 2.0 series.

Download

Get the new release here: http://www.openinfosecfoundation.org/download/suricata-2.0.2.tar.gz

Notable changes

  • IP defrag issue leading to evasion. Bug discovered by Antonios Atlasis working with ERNW GmbH
  • Support for NFLOG as a capture method. Nice work by Giuseppe Longo
  • DNS TXT parsing and logging. Funded by Emerging Threats
  • Log rotation through SIGHUP. Created by Jason Ish of Endace/Emulex

All closed tickets

  • Feature #781: IDS using NFLOG iptables target
  • Feature #1158: Parser DNS TXT data parsing and logging
  • Feature #1197: liblua support
  • Feature #1200: sighup for log rotation
  • Bug #1098: http_raw_uri with relative pcre parsing issue
  • Bug #1175: unix socket: valgrind warning
  • Bug #1189: abort() in 2.0dev (rev 6fbb955) with pf_ring 5.6.3
  • Bug #1195: nflog: cppcheck reports memleaks
  • Bug #1206: ZC pf_ring not working with Suricata 2.0.1 (or latest git)
  • Bug #1211: defrag issue
  • Bug #1212: core dump (after a while) when app-layer.protocols.http.enabled = yes
  • Bug #1214: Global Thresholds (sig_id 0, gid_id 0) not applied correctly if a signature has event vars
  • Bug #1217: Segfault in unix-manager.c line 529 when using –unix-socket and sending pcap files to be analized via socket

Special thanks

We’d like to thank the following people and corporations for their contributions and feedback:

  • Ken Steele — Tilera
  • Jason Ish — Endace/Emulex
  • Tom Decanio — nPulse
  • Antonios Atlasis working with ERNW GmbH
  • Alessandro Guido
  • Mats Klepsland
  • @rmkml
  • Luigi Sandon
  • Christie Bunlon
  • @42wim
  • Jeka Pats
  • Noam Meltzer
  • Ivan Ristic

Known issues & missing features

If you encounter issues, please let us know! As always, we are doing our best to make you aware of continuing development and items within the engine that are not yet complete or optimal. With this in mind, please notice the list we have included of known items we are working on.  See issues for an up to date list and to report new issues. See Known_issues for a discussion and time line for the major issues.

About Suricata

Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

by inliniac at June 25, 2014 04:25 PM

June 24, 2014

Peter Manev

Suricata IDPS - getting the best out of undersized servers with BPFs on heavy traffic links


How to inspect 3-4Gbps with Suricata IDPS with 20K rules loaded on 4CPUs(2.5GHz) and 16G RAM server while having minimal drops - less than 1%

Impossible? ... definitely not...
Improbable? ...not really

Setup


3,2-4Gbps of mirrored traffic
4 X CPU  - E5420  @ 2.50GHz (4 , NOT 8 with hyper-threading, just 4)
16GB RAM
Kernel Level -  3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Network Card - 82599EB 10-Gigabit SFI/SFP+ Network Connection
with driver=ixgbe driverversion=3.17.3
Suricata version 2.0dev (rev 896b614) - some commits above 2.0.1
20K rules - ETPro ruleset 




between 400-700K pps







If you want to run Suricata on that HW with about 20 000 rules inspecting 3-4Gbps traffic with minimal drops -  it is just not possible. There are not enough CPUs , not enough RAM....

Sometimes funding is tough, convincing management to buy new/more HW could be difficult for a particular endeavor/test and a number of other reasons...

So what can you do?

BPF


Suricata can utilize BPF (Berkeley Packet Filter) when running inspection. It allows to select and filter the type of traffic you would want Suricata to inspect.


There are three ways you can use BPF filter with Suricata:

  • On the command line
suricata -c /etc/suricata/suricata.yaml -i eth0 -v dst port 80

  • From suricata.yaml
Under each respective runmode in suricata.yaml (afpacket,pfring,pcap) - bpf-filter: port 80 or udp

  • From a file
suricata -c /etc/suricata/suricata.yaml -i eth0 -v -F bpf.file

Inside the bpf.file you would have your BPF filter.


The examples above would filter only the traffic that has dest port 80 and would pass it too Suricata for inspection.

BPF - The tricky part



It DOES make a difference when using BPF if you have VLANs in the mirrored traffic.

Please read here before you continue further - http://taosecurity.blogspot.se/2008/12/bpf-for-ip-or-vlan-traffic.html


The magic


If you want to -
extract all client data , TCP SYN|FIN flags to preserve session state and server response headers
the BPF filter (thanks to Cooper Nelson (UCSD) who shared the filter on our (OISF) mailing list) would look like this:


(port 53 or 443 or 6667) or (tcp dst port 80 or (tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or tcp[((tcp[12:1] & 0xf0) >>
2):4] = 0x48545450)))



That would inspect traffic on ports 53 (DNS) , 443(HTTPS), 6667 (IRC) and 80 (HTTP)

NOTE: the filter above is  for NON VLAN traffic !

Now the same filter for VLAN present traffic would look like this below:

((ip and port 53 or 443 or 6667) or ( ip and tcp dst port 80 or (ip
and tcp src port 80 and (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))
or
((vlan and port 53 or 443 or 6667) or ( vlan and tcp dst port 80 or
(vlan and tcp src port 80 and (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0
or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))

BPF - my particular case

I did some traffic profiling on the sensor and it could be summed up like this (using iptraf):


you can see that the traffic on port 53 (DNS) is just as much as the one on http. I was facing some tough choices...

The bpf filter that I made for this particular case was:

(
(ip and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 445 or 587 or 6667)
or ( ip and tcp dst port 80 or (ip and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))
or
((vlan and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 445 or 587 or 6667)
or ( vlan and tcp dst port 80 or (vlan and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)))
)

That would filter MIXED (both VLAN and NON VLAN) traffic on ports
  • 20/21 (FTP)
  • 25 (SMTP) 
  • 80 (HTTP) 
  • 110 (POP3)
  • 161 (SNMP)
  • 443(HTTPS)
  • 445 (Microsoft-DS Active Directory, Windows shares)
  • 587 (MSA - SNMP)
  • 6667 (IRC) 

and pass it to Suricata for inspection.

I had to drop the DNS - I am not saying this is right to do, but tough times call for tough measures. I had a seriously undersized server (4 cpu 2,5 Ghz 16GB RAM) and traffic between 3-4Gbps

How it is actually done


Suricata
root@snif01:/home/pmanev# suricata --build-info
This is Suricata version 2.0dev (rev 896b614)
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON
SIMD support: SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.6.3, C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         yes
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  DAG enabled:                             no
  Napatech enabled:                        no
  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  Prelude support:                         no
  PCRE jit:                                no
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                yes
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix (--prefix):          /usr/local
  Configuration directory (--sysconfdir):  /usr/local/etc/suricata/
  Log directory (--localstatedir) :        /usr/local/var/log/suricata/

  Host:                                    x86_64-unknown-linux-gnu
  GCC binary:                              gcc
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no

In suricata .yaml

#max-pending-packets: 1024
max-pending-packets: 65534
...
# Runmode the engine should use. Please check --list-runmodes to get the available
# runmodes for each packet acquisition method. Defaults to "autofp" (auto flow pinned
# load balancing).
#runmode: autofp
runmode: workers
.....
.....
af-packet:
  - interface: eth2
    # Number of receive threads (>1 will enable experimental flow pinned
    # runmode)
    threads: 4
    # Default clusterid.  AF_PACKET will load balance packets based on flow.
    # All threads/processes that will participate need to have the same
    # clusterid.
    cluster-id: 98
    # Default AF_PACKET cluster type. AF_PACKET can load balance per flow or per hash.
    # This is only supported for Linux kernel > 3.1
    # possible value are:
    #  * cluster_round_robin: round robin load balancing
    #  * cluster_flow: all packets of a given flow are send to the same socket
    #  * cluster_cpu: all packets treated in kernel by a CPU are send to the same socket
    cluster-type: cluster_cpu
    # In some fragmentation case, the hash can not be computed. If "defrag" is set
    # to yes, the kernel will do the needed defragmentation before sending the packets.
    defrag: yes
    # To use the ring feature of AF_PACKET, set 'use-mmap' to yes
    use-mmap: yes
    # Ring size will be computed with respect to max_pending_packets and number
    # of threads. You can set manually the ring size in number of packets by setting
    # the following value. If you are using flow cluster-type and have really network
    # intensive single-flow you could want to set the ring-size independantly of the number
    # of threads:
    ring-size: 200000
    # On busy system, this could help to set it to yes to recover from a packet drop
    # phase. This will result in some packets (at max a ring flush) being non treated.
    #use-emergency-flush: yes
    # recv buffer size, increase value could improve performance
    # buffer-size: 100000
    # Set to yes to disable promiscuous mode
    # disable-promisc: no
    # Choose checksum verification mode for the interface. At the moment
    # of the capture, some packets may be with an invalid checksum due to
    # offloading to the network card of the checksum computation.
    # Possible values are:
    #  - kernel: use indication sent by kernel for each packet (default)
    #  - yes: checksum validation is forced
    #  - no: checksum validation is disabled
    #  - auto: suricata uses a statistical approach to detect when
    #  checksum off-loading is used.
    # Warning: 'checksum-validation' must be set to yes to have any validation
    #checksum-checks: kernel
    # BPF filter to apply to this interface. The pcap filter syntax apply here.
    #bpf-filter: port 80 or udp
....
....  
detect-engine:
  - profile: high
  - custom-values:
      toclient-src-groups: 2
      toclient-dst-groups: 2
      toclient-sp-groups: 2
      toclient-dp-groups: 3
      toserver-src-groups: 2
      toserver-dst-groups: 4
      toserver-sp-groups: 2
      toserver-dp-groups: 25
  - sgh-mpm-context: auto
 ...
flow-timeouts:

  default:
    new: 5 #30
    established: 30 #300
    closed: 0
    emergency-new: 1 #10
    emergency-established: 2 #100
    emergency-closed: 0
  tcp:
    new: 5 #60
    established: 60 # 3600
    closed: 1 #30
    emergency-new: 1 # 10
    emergency-established: 5 # 300
    emergency-closed: 0 #20
  udp:
    new: 5 #30
    established: 60 # 300
    emergency-new: 5 #10
    emergency-established: 5 # 100
  icmp:
    new: 5 #30
    established: 60 # 300
    emergency-new: 5 #10
    emergency-established: 5 # 100
....
....
stream:
  memcap: 4gb
  checksum-validation: no      # reject wrong csums
  midstream: false
  prealloc-sessions: 50000
  inline: no                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 8gb
    depth: 12mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
...
...
default-rule-path: /etc/suricata/et-config/
rule-files:
 - trojan.rules  
 - malware.rules
 - local.rules
 - activex.rules
 - attack_response.rules
 - botcc.rules
 - chat.rules
 - ciarmy.rules
 - compromised.rules
 - current_events.rules
 - dos.rules
 - dshield.rules
 - exploit.rules
 - ftp.rules
 - games.rules
 - icmp_info.rules
 - icmp.rules
 - imap.rules
 - inappropriate.rules
 - info.rules
 - misc.rules
 - mobile_malware.rules ##
 - netbios.rules
 - p2p.rules
 - policy.rules
 - pop3.rules
 - rbn-malvertisers.rules
 - rbn.rules
 - rpc.rules
 - scada.rules
 - scada_special.rules
 - scan.rules
 - shellcode.rules
 - smtp.rules
 - snmp.rules

...
....
libhtp:

         default-config:
           personality: IDS

           # Can be specified in kb, mb, gb.  Just a number indicates
           # it's in bytes.
           request-body-limit: 12mb
           response-body-limit: 12mb

           # inspection limits
           request-body-minimal-inspect-size: 32kb
           request-body-inspect-window: 4kb
           response-body-minimal-inspect-size: 32kb
           response-body-inspect-window: 4kb

           # decoding
           double-decode-path: no
           double-decode-query: no


Create the BFP file (you can put it anywhere)
touch /home/pmanev/test/bpf-filter


The  bpf-filter should look like this:


root@snif01:/var/log/suricata# cat /home/pmanev/test/bpf-filter
(
(ip and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 445 or 587 or 6667)
or ( ip and tcp dst port 80 or (ip and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))
or
((vlan and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 445 or 587 or 6667)
or ( vlan and tcp dst port 80 or (vlan and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)))
)
root@snif01:/var/log/suricata#


Start Suricata like this:
suricata -c /etc/suricata/peter-yaml/suricata-afpacket.yaml --af-packet=eth2 -D -v -F /home/pmanev/test/bpf-filter

Like this I was able to achieve inspection on 3,2-4Gbps with about 20K rules with 1% drops.




In the suricata.log:
root@snif01:/var/log/suricata# more suricata.log
[1274] 21/6/2014 -- 19:36:35 - (suricata.c:1034) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev 896b614)
[1274] 21/6/2014 -- 19:36:35 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 4
......
[1275] 21/6/2014 -- 19:36:46 - (detect.c:452) <Info> (SigLoadSignatures) -- 46 rule files processed. 20591 rules successfully loaded, 8 rules failed
[1275] 21/6/2014 -- 19:36:47 - (detect.c:2591) <Info> (SigAddressPrepareStage1) -- 20599 signatures processed. 827 are IP-only rules, 6510 are inspecting packet payload, 15650 inspect ap
plication layer, 0 are decoder event only
.....
.....
[1275] 21/6/2014 -- 19:37:17 - (runmode-af-packet.c:150) <Info> (ParseAFPConfig) -- Going to use command-line provided bpf filter '( (ip and port 20 or 21 or 22 or 25 or 110 or 161 or 44
3 or 445 or 587 or 6667)  or ( ip and tcp dst port 80 or (ip and tcp src port 80 and  (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)))) or ((vl
an and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 445 or 587 or 6667)  or ( vlan and tcp dst port 80 or (vlan and tcp src port 80 and  (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))) ) '

.....
.....
[1275] 22/6/2014 -- 01:45:34 - (stream.c:182) <Info> (StreamMsgQueuesDeinit) -- TCP segment chunk pool had a peak use of 6674 chunks, more than the prealloc setting of 250
[1275] 22/6/2014 -- 01:45:34 - (host.c:245) <Info> (HostPrintStats) -- host memory usage: 825856 bytes, maximum: 16777216
[1275] 22/6/2014 -- 01:45:35 - (detect.c:3890) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[1275] 22/6/2014 -- 01:45:35 - (util-device.c:190) <Notice> (LiveDeviceListClean) -- Stats for 'eth2':  pkts: 2820563520, drop: 244696588 (8.68%), invalid chksum: 0

That gave me about 9% drops...... I further adjusted the filter (after realizing I could drop 445 Windows Shares for the moment from inspection).

The new filter was like so:

root@snif01:/var/log/suricata# cat /home/pmanev/test/bpf-filter
(
(ip and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 587 or 6667)
or ( ip and tcp dst port 80 or (ip and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))
or
((vlan and port 20 or 21 or 22 or 25 or 110 or 161 or 443 or 587 or 6667)
or ( vlan and tcp dst port 80 or (vlan and tcp src port 80 and
(tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or
tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)))
)
root@snif01:/var/log/suricata#
Notice - I removed port 445.

So with that filter I was able to do 0.95% drops with 20K rules:

[16494] 22/6/2014 -- 10:13:10 - (suricata.c:1034) <Notice> (SCPrintVersion) -- This is Suricata version 2.0dev (rev 896b614)
[16494] 22/6/2014 -- 10:13:10 - (util-cpu.c:170) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 4
...
...
[16495] 22/6/2014 -- 10:13:20 - (detect.c:452) <Info> (SigLoadSignatures) -- 46 rule files processed. 20591 rules successfully loaded, 8 rules failed
[16495] 22/6/2014 -- 10:13:21 - (detect.c:2591) <Info> (SigAddressPrepareStage1) -- 20599 signatures processed. 827 are IP-only rules, 6510 are inspecting packet payload, 15650 inspect application layer, 0 are decoder event only
...
...
[16495] 23/6/2014 -- 01:45:32 - (host.c:245) <Info> (HostPrintStats) -- host memory usage: 1035520 bytes, maximum: 16777216
[16495] 23/6/2014 -- 01:45:32 - (detect.c:3890) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
[16495] 23/6/2014 -- 01:45:32 - (util-device.c:190) <Notice> (LiveDeviceListClean) -- Stats for 'eth2':  pkts: 6550734692, drop: 62158315 (0.95%), invalid chksum: 0




 So with that BPF filter we have ->

Pros 


I was able to inspect with a lot of rules(20K) a lot of traffic (4Gbps peak) with an undersized and minimal HW (4 CPU 16GB RAM ) sustained with less then 1% drops

Cons

  • Not inspecting DNS
  • Making an assumption that all HTTP traffic is using port 80. (Though in my case 99.9% of the http traffic was on port 80)
  • This is an advanced BPF filter , requires a good chunk of knowledge in order to understand/implement/re-edit


 Simple and efficient


In the case where you have a network or a device that generates a lot of false positives and you are sure you can disregard any traffic from that device  - you could do a filter like this:

(ip and not host 1.1.1.1 ) or (vlan and not host 1.1.1.1)

for a VLAN and non VLAN traffic mixed. If you are sure there is no VLAN traffic you could just do that:
ip and not host 1.1.1.1

Then  you can simply start Suricata like so:
suricata -c /etc/suricata/peter-yaml/suricata-afpacket.yaml --af-packet=eth2 -D -v \(ip and not host 1.1.1.1 \) or \(vlan and not host 1.1.1.1\)

or like this respectively (to the two examples above):
suricata -c /etc/suricata/peter-yaml/suricata-afpacket.yaml --af-packet=eth2 -D -v ip and not host 1.1.1.1






by Peter Manev (noreply@blogger.com) at June 24, 2014 11:26 AM