Planet Suricata

by Peter Manev (noreply@blogger.com) at December 12, 2013 09:29 AM · Introduction

checking for pfring_open in -lpfring... no

   ERROR! --enable-pfring was passed but the library was not found or version is >4, go get it
   from http://www.ntop.org/PF_RING.html
Events: 32K cycles                                                                                                                                                                                                                            
 28.41%  suricata            [.] SCACSearch
 19.86%  libc-2.15.so        [.] tolower
 17.83%  suricata            [.] SigMatchSignaturesBuildMatchArray
  6.11%  suricata            [.] SigMatchSignaturesBuildMatchArrayAddSignature
  2.06%  suricata            [.] tolower@plt
  1.70%  libpthread-2.15.so  [.] pthread_mutex_trylock
  1.17%  suricata            [.] StreamTcpGetFlowState
  1.10%  libc-2.15.so        [.] __memcpy_ssse3_back
  0.90%  libpthread-2.15.so  [.] pthread_mutex_lock
./configure --enable-pfring --enable-luajit CFLAGS="-fno-omit-frame-pointer"
make
make install

sudo perf record -a --call-graph -p 9366
sudo perf report --call-graph --stdio
     0.94%  Suricata-Main  libpthread-2.15.so     [.] pthread_mutex_lock
            |
            --- pthread_mutex_lock
               |
               |--48.69%-- FlowHandlePacket
               |          |
               |          |--53.04%-- DecodeUDP
               |          |          |
               |          |          |--95.84%-- DecodeIPV4
               |          |          |          |
               |          |          |          |--99.97%-- DecodeVLAN
               |          |          |          |          DecodeEthernet
               |          |          |          |          DecodePfring
               |          |          |          |          TmThreadsSlotVarRun
               |          |          |          |          TmThreadsSlotProcessPkt
               |          |          |          |          ReceivePfringLoop
               |          |          |          |          TmThreadsSlotPktAcqLoop
               |          |          |          |          start_thread
               |          |          |           --0.03%-- [...]
               |          |          |
               |          |           --4.16%-- DecodeIPV6
               |          |                     |
               |          |                     |--97.59%-- DecodeTunnel
               |          |                     |          |
               |          |                     |          |--99.18%-- DecodeTeredo
               |          |                     |          |          DecodeUDP
               |          |                     |          |          DecodeIPV4
               |          |                     |          |          DecodeVLAN
               |          |                     |          |          DecodeEthernet
               |          |                     |          |          DecodePfring
               |          |                     |          |          TmThreadsSlotVarRun
               |          |                     |          |          TmThreadsSlotProcessPkt
               |          |                     |          |          ReceivePfringLoop
               |          |                     |          |          TmThreadsSlotPktAcqLoop
               |          |                     |          |          start_thread
               |          |                     |          |
               |          |                     |           --0.82%-- DecodeIPV4
               |          |                     |                     DecodeVLAN
               |          |                     |                     DecodeEthernet
               |          |                     |                     DecodePfring
               |          |                     |                     TmThreadsSlotVarRun
               |          |                     |                     TmThreadsSlotProcessPkt
               |          |                     |                     ReceivePfringLoop
               |          |                     |                     TmThreadsSlotPktAcqLoop
               |          |                     |                     start_thread
               |          |                     |
               |          |                      --2.41%-- DecodeIPV6
               |          |                                DecodeTunnel
               |          |                                DecodeTeredo
               |          |                                DecodeUDP
               |          |                                DecodeIPV4
               |          |                                DecodeVLAN
               |          |                                DecodeEthernet
               |          |                                DecodePfring
               |          |                                TmThreadsSlotVarRun
               |          |                                TmThreadsSlotProcessPkt
               |          |                                ReceivePfringLoop
               |          |                                TmThreadsSlotPktAcqLoop
               |          |                                start_thread

profiling:
  # per keyword profiling
  keywords:
    enabled: yes
    filename: keyword_perf.log
    append: yes
--------------------------------------------------------------------------
Date: 11/7/2013 -- 15:13:11
--------------------------------------------------------------------------
Stats for: total
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00    
content          1274592063  534328   196738   312321      2385.00     2424.00     2362.00    
pcre             56626031    11149    824      254562      5079.00     12234.00    4507.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
isdataat         580764      558      556      2427        1040.00     1040.00     1017.00    
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         1677209     874      870      9873        1919.00     1923.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: packet
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flow             38276182    22842    22842    63987       1675.00     1675.00     0.00       
dsize            2212029     2062     2061     3711        1072.00     1072.00     789.00     
flowbits         351171      294      290      5526        1194.00     1198.00     884.00     
itype            1653        2        1        1386        826.00      267.00      1386.00    
icode            27383781    93827    2        25545       291.00      1021.00     291.00     
flags            192751968   245519   189709   255639      785.00      753.00      892.00     
--------------------------------------------------------------------------
Stats for: packet/stream payload
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1203990910  512902   183628   312321      2347.00     2365.00     2337.00    
pcre             28087301    6598     54       254562      4256.00     12279.00    4190.00    
byte_test        153287955   128254   32109    67989       1195.00     1658.00     1040.00    
byte_jump        3676404     2041     2041     15939       1801.00     1801.00     0.00       
isdataat         578172      556      554      2427        1039.00     1039.00     1017.00    
byte_extract     143091      78       78       7743        1834.00     1834.00     0.00       
--------------------------------------------------------------------------
Stats for: http uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          44775802    13102    8351     60993       3417.00     3257.00     3698.00    
pcre             18284421    3646     97       61338       5014.00     8916.00     4908.00    
isdataat         2592        2        2        1725        1296.00     1296.00     0.00       
urilen           6149297     6142     1099     28299       1001.00     1395.00     915.00     
--------------------------------------------------------------------------
Stats for: http raw uri
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
pcre             9534        2        0        4953        4767.00     0.00        4767.00    
--------------------------------------------------------------------------
Stats for: http client body
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          1556904     441      181      58476       3530.00     2874.00     3986.00    
pcre             63924       6        6        17358       10654.00    10654.00    0.00       
--------------------------------------------------------------------------
Stats for: http headers
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23688244    7631     4348     31098       3104.00     3311.00     2829.00    
pcre             9998970     859      667      71904       11640.00    12727.00    7862.00    
--------------------------------------------------------------------------
Stats for: http stat code
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          80052       39       20       3699        2052.00     2199.00     1898.00    
--------------------------------------------------------------------------
Stats for: http method
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          476334      203      201      27240       2346.00     2351.00     1846.00    
--------------------------------------------------------------------------
Stats for: http cookie
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
content          23817       10       9        2763        2381.00     2384.00     2358.00    
pcre             181881      38       0        13095       4786.00     0.00        4786.00    
--------------------------------------------------------------------------
Stats for: post-match
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
flowbits         1326038     580      580      9873        2286.00     2286.00     0.00       
--------------------------------------------------------------------------
Stats for: threshold
--------------------------------------------------------------------------
Keyword          Ticks       Checks   Matches  Max Ticks   Avg         Avg Match   Avg No Match
---------------- ----------- -------- -------- ----------- ----------- ----------- ----------- 
threshold        355324491   190574   409      72276       1864.00     3625.00     1860.00
input {
  file { 
    path => "/home/eric/builds/suricata/var/log/suricata/files-json.log" 
    codec =>   json 
    # This format tells logstash to expect 'logstash' json events from the file.
    #format => json_event 
  }
}

output { 
  stdout { codec => rubydebug }
  elasticsearch { embedded => true }
}

#geoip part
filter {
  if [srcip] {
    geoip {
      source => "srcip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}

You must specify 'database => ...' in your geoip filter"
#geoip part
filter {
  if [srcip] {
    geoip {
      source => "srcip"
      target => "geoip"
      database => "/path/to/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }
}

java -jar /home/eric/builds/logstash/logstash-1.2.2-flatjar.jar agent -f /home/eric/builds/logstash/logstash.conf --log /home/eric/builds/logstash/log/logstash-indexer.out -- web

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    MAGIC HERE
}'

curl -XGET 'http://localhost:9200/_all/_mapping'

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "logs" : {
         "properties": {
            "@timestamp":{"type":"date",
            "format":"dateOptionalTime"},
            "@version":{"type":"string"},
            "dp":{"type":"long"},
            "dstip":{"type":"ip"},
            "filename":{"type":"string"},
            "geoip":{
               "properties":{
                  "area_code":{"type":"long"},
                  "city_name":{"type":"string", "index":"not_analyzed"},
                  "continent_code":{"type":"string"},
                  "coordinates":{"type":"string"},
                  "country_code2":{"type":"string"},
                  "country_code3":{"type":"string"},
                  "country_name":{"type":"string", "index":"not_analyzed"},
                  "dma_code":{"type":"long"},
                  "ip":{"type":"string"},
                  "latitude":{"type":"double"},
                  "longitude":{"type":"double"},
                  "postal_code":{"type":"string"},
                  "real_region_name":{"type":"string", "index":"not_analyzed"},
                  "region_name":{"type":"string", "index":"not_analyzed"},
                  "timezone":{"type":"string"}
               }
            },
            "host":{"type":"string"},
            "http_host":{"type":"string"},
            "http_referer":{"type":"string"},
            "http_uri":{"type":"string"},
            "http_user_agent":{"type":"string", "index":"not_analyzed", "omit_norms":true, "index_options":"docs"},
            "ipver":{"type":"long"},
            "magic":{"type":"string", "index":"not_analyzed", "omit_norms":true, "index_options":"docs"},
            "md5":{"type":"string"},
            "path":{"type":"string"},
            "protocol":{"type":"long"},
            "size":{"type":"long"},
            "sp":{"type":"long"},
            "srcip":{"type":"ip"},
            "state":{"type":"string"},
            "stored":{"type":"boolean"},
            "tags":{"type":"string"},
            "timestamp":{"type":"string"}
      }
    }
  }
}'

["192.168.42.24" TO "192.168.42.45"]

curl -XGET 'http://localhost:9200/_all/_mapping'
{"logstash-2013.10.27":{"logs":{"properties":

curl -XDELETE 'http://localhost:9200/logstash-2013.10.27'
curl -XPUT 'http://localhost:9200/logstash-2013.10.27'

curl -XGET 'http://localhost:9200/logstash-2013.10.27/_refresh'

    a = ScFlowintIncr(0)

    function init (args)
        local needs = {}
        needs["http.request_headers"] = tostring(true)
        needs["flowint"] = {"cnt_incr"}
        return needs
    end

    function match(args)
        a = ScFlowintIncr(0);
        if a == 23 then
            return 1
        end

        return 0
    end
    return 0

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2434188332   59.71  82249    795      711777      29595.35    7683.20     29809.22   
  2        2            1        0        1015328580   24.91  82249    795      154398      12344.57    3768.66     12428.27   
  3        3            1        0        626858067    15.38  82249    795      160731      7621.47     3439.91     7662.28    

alert http any any -> any any (msg:"LUAJIT HTTP flowvar match"; luajit:lua_flowvar_cnt.lua; flow:to_server; sid:1;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint match"; luajit:lua_flowint_cnt.lua; flow:to_server; sid:2;)
alert http any any -> any any (msg:"LUAJIT HTTP flowint incr match"; luajit:lua_flowint_incr_cnt.lua; flow:to_server; sid:3;)

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowint"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowintGet(0);
    if a then
        ScFlowintSet(0, a + 1)
    else
        ScFlowintSet(0, 1)
    end 

    a = ScFlowintGet(0);
    if a == 23 then
        return 1
    end 

    return 0
end 

return 0

function init (args)
    local needs = {}
    needs["http.request_headers"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end

function match(args)
    a = ScFlowvarGet(0);
    if a then
        a = tostring(tonumber(a)+1)
        ScFlowvarSet(0, a, #a)
    else
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end 

    if tonumber(a) == 23 then
        return 1
    end

    return 0
end

return 0

   Num      Rule         Gid      Rev      Ticks        %      Checks   Matches  Max Ticks   Avg Ticks   Avg Match   Avg No Match
  -------- ------------ -------- -------- ------------ ------ -------- -------- ----------- ----------- ----------- -------------- 
  1        1            1        0        2392221879   70.56  82249    795      834993      29085.12    6964.14     29301.02   
  2        2            1        0        998297994    29.44  82249    795      483810      12137.51    4019.44     12216.74   

function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"cnt"}
    return needs
end
    needs["flowvar"] = {"cnt", "somevar", "anothervar" }
function match(args)
    a = ScFlowvarGet(0);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(0, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(0, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end
function init (args)
    local needs = {}
    needs["http.request_headers.raw"] = tostring(true)
    needs["flowvar"] = {"blah", "cnt"}
    return needs
end

local var_cnt = 1

function match(args)
    a = ScFlowvarGet(var_cnt);
    if a then
        print ("We have an A: " .. (a))
        a = tostring(tonumber(a)+1)
        print ("A incremented to: " .. (a))
        ScFlowvarSet(var_cnt, a, #a)
    else
        print "Init A to 1"
        a = tostring(1)
        ScFlowvarSet(var_cnt, a, #a)
    end

    print ("A is " .. (a))
    if tonumber(a) == 23 then
        print "Match!"
        return 1
    end

    return 0
end
-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 101
-> SYN: SEQ 10
<- SYN/ACK 1: ACK 11, SEQ 100
<- SYN/ACK 2: ACK 11, SEQ 1000
-> ACK: SEQ 11, ACK 1001
    UpdateState(p);
    ssn->state = TCP_SYN_RECV;
    if (p != ssn) {
        QueueState(p);
    if (ssn->queue_len) {
        q = QueueFindState(p);
        if (q)
            UpdateState(q);
    }
    UpdateState(p);
    ssn->state = TCP_ESTABLISHED;
stream:
  max-synack-queued: 5
  alert tcp any any -> any any (msg:"SURICATA STREAM 3way handshake \
      excessive different SYN/ACKs"; stream-event:3whs_synack_flood; \
      sid:2210055; rev:1;)
af-packet:
  - interface: wlan0
    # In some fragmentation case, the hash can not be computed. If "defrag" is set
    # to yes, the kernel will do the needed defragmentation before sending the packets.
    defrag: no

Planet Suricata

December 12, 2013

Introduction

Part One - PF_RING

Compile and install PF_RING

Compile and install Suricata with PF_RING enabled

PF_RING - suricata.yaml tune up and configuration

Run it

On a side note

December 11, 2013

Introduction

Part Two - DNA

Compile

Configure

Configuration in suricata.yaml

Start Suricata with DNA

Introduction

Chapter I - Preparation

System's HW

System's OS

Suricata

Network Traffic

Packages installation

Installation and configuration

Suricata

Network card drivers and tuning

December 10, 2013

Introduction

Chapter III - AF_PACKET

AF_PACKET - suricata.yaml tune up and configuration

Run it

Results

November 18, 2013

Introduction

Getting function call graph in perf

November 07, 2013

October 28, 2013

Introduction

Geoip support

Logstash indexing and mapping

October 15, 2013

October 10, 2013

October 09, 2013

September 30, 2013

September 27, 2013

September 26, 2013

September 25, 2013

September 24, 2013

Fixes

Special thanks

Security

Known issues & missing features

About Suricata

Fixes

Special thanks

Security

Known issues & missing features

About Suricata

September 20, 2013

September 04, 2013

July 28, 2013

July 26, 2013

Fixes

Special thanks

Known issues & missing features

About Suricata

Fixes

Special thanks

Known issues & missing features

About Suricata

July 19, 2013

July 18, 2013

New features

Improvements

Fixes

Special thanks

Known issues & missing features

About Suricata

New features

Improvements