This package provides limiting mechanisms for Zeek's intelligence framework. This encompasses matching thresholds on per item basis and a heavy hitter suppression.
The scripts are available as package for the Bro/Zeek Package Manager and can be installed using the following command:
bro-pkg install intel-limiter
To enable per item thresholds and heavy hitter suppression make sure the package is loaded:
bro-pkg load intel-limiter
With the package loaded, intel items might specify a new meta data field
meta.matching_threshold, which denotes the number of matches that must be seen to trigger a match. By default (e.g. if the intel file does not specify the additional meta data) the threshold is set to one, which can be adapted using the
default_matching_threshold option. The following example shows an intel file that contains two IPs, which will cause a match after three and five hits.
#fields indicator indicator_type meta.source meta.desc meta.matching_threshold 10.0.0.23 Intel::ADDR source_a This host is bad 3 10.0.0.42 Intel::ADDR source_b This host is bad 5
Once a match is triggered, the internal match counter will be reset and the next match is triggered after the threshold is reached again. The threshold counters are managed per item. If two items that are obtained from different sources share the same indicator, thresholds do not interfere. For example, if thresholds of two and four are specified for a single indicator, every fourth time the indicator is seen a hit containing all meta data will be generated, while every second reporting will cause a match containing only the meta data of the item that defines the threshold of two.
Note: As Bro 2.6 does not pass modified hook parameters along the chain, hits will be reported in the correct frequency but contain the meta data of all items.
Heavy Hitter Suppression
To enable heavy hitter suppression, a heavy hitter interval and threshold have to be specified. The option
heavy_hitter_interval defines the time window in which at most
heavy_hitter_threshold hits might occur. If more hits are observed in that time window, the indicator is removed. By default the threshold is set to zero which disables suppression. When removing an heavy hitter, a reporter warning is generated to inform the user about the removal.
item-threshold.bro implements per item thresholds and is loaded by default. Per item thresholds are realized using two additional meta data fields:
matching_thresholddefines the number of reports required to trigger a match
recent_matchesimplements a counter for recent matches
To prevent matching of an item before the specified threshold is reached, the script handles the
extend_match hook of the intelligence framework and checks the number of recent hits.
For further details on the intel framework see the corresponding blog post.
heaver-hitter.bro implements heavy hitter suppression simply by tracking hits per indicator in a table. Note that item removal and reporting a warning is a very simple strategy. In more complex environments one would want to report heavy hitters using a dedicated feedback channel, e.g. implemented with broker.
Thanks to @mavam (Tenzir), who brought up the idea for heavy hitter suppression during Zeek Workshop Europe 2019.