|
 |
Per-user and per-port connection and bandwidth/data transfer statistics from firewall logs
Introduction
You can use the logs generated by Clavister Firewall, either FWLogger
logs or syslog logs, to collect and display per-user statistics about
connection and/or bandwidth/data transfer on a periodic basis.
This requires that all log data about connections that you are interested
in are sent to the log receiver in question, i.e. you need to enable
logging for all "Allow" and "NAT" rules that you want statistics for.
Prerequisites
- Clavister Firewall logs, stored by a Clavister FWLogger or generic syslog receiver.
- The "toptalkers.zip" archive, which contains:
- "toptalkers.awk", the statistics collection script
Note that it requires gawk v3.1 or newer (included in the .zip for windows users).
- "fwlogqry.exe", Clavister FWLogQry (also available on v8.10+ install CDs)
- "example.bat", example commands for win32 / fwlogger
- "example.sh", example commands for unix / syslog
- The "gawk", "sort" and "head" utilities. Native win32 versions of
these utilities are included in the above package. Any recent
version of unix/linux should already have them.
Unix / syslog procedure
Below are the relevant excerpts from "example.sh". At a minimum,
you will have to point "logfile" at the right syslog file.
You may also have to adjust the "grep" command to extract only
entries from the relevant firewall, in case the syslog file
contains logs from multiple firewalls.
This script can be periodically run just before midnight to extract
the past day's activity and output "top talkers" reports.
# Which log file do we examine?
logfile=/var/log/firewall
# What interface do you want statistics for?
iface=int
# How many top talkers do you want to list?
topnum=10
# What to display? Connections, Received, Sent and/or Data totals? Users? Ports?
set display=crsdup
# Format a date string like the standard BSD syslog daemon does.
now=`date '+%b %e'`;
echo Analyzing '"'$now'"' from $logfile...
# Extract today's records from log file and hand to toptalkers.awk
grep "^$now .*FW:.*conn=close" $logfile | gawk -f toptalkers.awk $iface $topnum $display
|
Windows / FWLogger procedure
Below are the relevant excerpts from "example.bat". At a minimum,
you will need to change "fwloggerdir" and "fwname".
This script can be periodically run after midnight to extract
the past day's activity and output "top talkers" reports.
REM Where does your fwlogger live? UNC paths are OK.
set fwloggerdir=d:\fwlogger
REM What is the name of your firewall?
set fwname=myfw
REM What interface do you want statistics for?
set iface=int
REM How many top talkers do you want to list?
set topnum=10
REM What to display? Connections, Received, Sent and/or Data totals? Users? Ports?
set display=crsdup
REM We don't want the standard windows sort; it can't do the work we want.
set sortcmd=.\sort.exe
REM Now run the fwlogqry command and pipe the output to toptalkers.awk
REM - This LQL expression could be much simpler, but fwlogqry is faster at
REM finding the relevant information than awk string searching is.
fwlogqry --userootdir %fwloggerdir% "select connevent,recvif,srcip,destiface,
(contd..) destip,ipproto,srcport,destport,usernames,origsent,termsent
(contd..) from "%fwname%" last full days 1 where connevent=closed and
(contd..) (recvif="%iface%" or destiface="%iface%")" |
(contd..) .\gawk -f toptalkers.awk %iface% %topnum% %display%
|
Example output
This is example output with "topnum=20" and "display=cd" in the above scripts.
The IP addresses have been obfuscated in this example.
Top 20 talkers -- connections
-----------------------------
403 833 ___.12.34.187
392 908 ___.12.33.33
376 752 ___.12.34.136
374 330 ___.12.32.250
248 061 ___.12.34.138
243 120 ___.12.32.238
175 185 ___.12.36.227
144 835 ___.12.37.98
144 310 ___.0.0.1
143 997 ___.12.38.232
142 409 ___.13.79.50
139 379 ___.12.32.48
130 473 ___.13.79.178
95 785 ___.12.33.10
94 080 ___.13.80.72
85 819 ___.12.37.67
81 341 ___.12.33.43
80 754 ___.12.32.55
77 645 ___.12.32.162
77 482 ___.12.32.208
Top 20 talkers -- data
---------------------------
11 330 933K ___.12.33.186 ( 8 719 485 sent + 2 611 447 recvd)
10 584 339K ___.12.33.212 ( 4 758 913 sent + 5 825 425 recvd)
8 349 053K ___.12.38.95 ( 7 272 085 sent + 1 076 967 recvd)
7 777 013K ___.12.34.192 ( 7 402 950 sent + 374 062 recvd)
7 675 384K ___.12.32.82 ( 6 134 604 sent + 1 540 780 recvd)
6 767 399K ___.12.33.44 ( 1 476 365 sent + 5 291 034 recvd)
6 479 599K ___.12.33.238 ( 3 090 665 sent + 3 388 933 recvd)
6 185 717K ___.12.33.15 ( 5 959 723 sent + 225 993 recvd)
5 950 243K ___.12.38.169 ( 4 169 761 sent + 1 780 481 recvd)
5 947 764K ___.12.34.187 ( 5 065 392 sent + 882 372 recvd)
5 816 955K ___.16.111.11 ( 4 885 593 sent + 931 361 recvd)
5 729 528K ___.12.32.131 ( 5 252 624 sent + 476 903 recvd)
5 685 686K ___.13.79.82 ( 208 559 sent + 5 477 127 recvd)
5 479 972K ___.12.34.136 ( 3 945 619 sent + 1 534 353 recvd)
5 221 463K ___.12.32.55 ( 4 037 023 sent + 1 184 439 recvd)
5 177 882K ___.12.32.230 ( 4 785 071 sent + 392 811 recvd)
5 100 944K ___.12.32.194 ( 2 029 310 sent + 3 071 634 recvd)
4 849 091K ___.12.33.98 ( 1 043 938 sent + 3 805 153 recvd)
4 550 656K ___.12.34.201 ( 3 711 943 sent + 838 712 recvd)
4 165 671K ___.12.32.243 ( 2 906 234 sent + 1 259 436 recvd)
|
Revision history
v1.0 2003-08-29 First public release
v1.1 2003-09-01 Added display of user names (if available) through the "u" switch (on by default)
v1.2 2003-09-08 Added per-port statistics through the "p" switch (on by default)
|
