after some days on not much used server, proftpd under systrace freezes. it produces no output to the logs, it does not respond to packets, but it is litening on its sockets; systrace does not log anything. this may be problem with locking. (?) XXX investigate this fstat output: USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root fstat 7758 root / 2 drwxr-xr-x 512 r root fstat 7758 wd / 86534 drwxr-xr-x 512 r root fstat 7758 text /usr 155603 -r-xr-sr-x 13056 r root fstat 7758 0 / 65643 crw--w---- ttyp1 rw root fstat 7758 1 /tmp 21135 -rw-r--r-- 0 w root fstat 7758 2 / 65643 crw--w---- ttyp1 rw root fstat 7758 3 / 65155 crw-r----- mem r root fstat 7758 4 / 65151 crw-r----- kmem r root fstat 7758 5 / 21871 -rw-r--r-- 964 r root fstat 7758 6 / 22290 -rw------- 40960 r root tcpdump 7749 root / 2 drwxr-xr-x 512 r root tcpdump 7749 wd / 86534 drwxr-xr-x 512 r root tcpdump 7749 text /usr 64907 -r-xr-xr-x 344276 r root tcpdump 7749 0 / 65643 crw--w---- ttyp1 rw root tcpdump 7749 1 / 65643 crw--w---- ttyp1 rw root tcpdump 7749 2 / 65643 crw--w---- ttyp1 rw root tcpdump 7749 3 / 64930 crw------- bpf0 r root tcpdump 7749 4 /tmp 21130 -rw-r--r-- 0 w root bash 7729 root / 2 drwxr-xr-x 512 r root bash 7729 wd / 86534 drwxr-xr-x 512 r root bash 7729 text /usr 57622 -rwxr-xr-x 817300 r root bash 7729 0 / 65643 crw--w---- ttyp1 rw root bash 7729 1 / 65643 crw--w---- ttyp1 rw root bash 7729 2 / 65643 crw--w---- ttyp1 rw root bash 7729 255 / 65643 crw--w---- ttyp1 rw malyl bash 7642 root / 2 drwxr-xr-x 512 r malyl bash 7642 wd /home 86604 drwxr-xr-x 1024 r malyl bash 7642 text /usr 57622 -rwxr-xr-x 817300 r malyl bash 7642 0 / 65643 crw--w---- ttyp1 rw malyl bash 7642 1 / 65643 crw--w---- ttyp1 rw malyl bash 7642 2 / 65643 crw--w---- ttyp1 rw malyl bash 7642 255 / 65643 crw--w---- ttyp1 rw malyl sshd 7641 root / 2 drwxr-xr-x 512 r malyl sshd 7641 wd / 2 drwxr-xr-x 512 r malyl sshd 7641 text /usr 64904 -r-xr-xr-x 152388 r malyl sshd 7641 0 / 65419 crw-rw-rw- null rw malyl sshd 7641 1 / 65419 crw-rw-rw- null rw malyl sshd 7641 2 / 65419 crw-rw-rw- null rw malyl sshd 7641 3* local stream d50c5320 <-> d4da1000 malyl sshd 7641 4* pipe d4d8c260 <-> d4d8d480 0 rw malyl sshd 7641 5* internet stream tcp d2e51fc0 malyl sshd 7641 6* pipe d4d8d480 <-> d4d8c260 0 rw malyl sshd 7641 7 / 65644 crw-rw-rw- ptyp1 rw malyl sshd 7641 8 / 65644 crw-rw-rw- ptyp1 rw malyl sshd 7641 9 / 65644 crw-rw-rw- ptyp1 rw root sshd 7638 root / 2 drwxr-xr-x 512 r root sshd 7638 wd / 2 drwxr-xr-x 512 r root sshd 7638 text /usr 64904 -r-xr-xr-x 152388 r root sshd 7638 0 / 65419 crw-rw-rw- null r root sshd 7638 1 / 65419 crw-rw-rw- null rw root sshd 7638 2 / 65419 crw-rw-rw- null rw root sshd 7638 3 / 65644 crw-rw-rw- ptyp1 rw root sshd 7638 4* local stream d4da1000 <-> d50c5320 root sshd 7638 5* internet stream tcp d2e51fc0 root bash 4370 root / 2 drwxr-xr-x 512 r root bash 4370 wd /home 65139 drwxr-xr-x 1536 r root bash 4370 text /usr 57622 -rwxr-xr-x 817300 r root bash 4370 0 / 65631 crw--w---- ttyp0 rw root bash 4370 1 / 65631 crw--w---- ttyp0 rw root bash 4370 2 / 65631 crw--w---- ttyp0 rw root bash 4370 255 / 65631 crw--w---- ttyp0 rw lada bash 4357 root / 2 drwxr-xr-x 512 r lada bash 4357 wd /home 65139 drwxr-xr-x 1536 r lada bash 4357 text /usr 57622 -rwxr-xr-x 817300 r lada bash 4357 0 / 65631 crw--w---- ttyp0 rw lada bash 4357 1 / 65631 crw--w---- ttyp0 rw lada bash 4357 2 / 65631 crw--w---- ttyp0 rw lada bash 4357 255 / 65631 crw--w---- ttyp0 rw lada sshd 4356 root / 2 drwxr-xr-x 512 r lada sshd 4356 wd / 2 drwxr-xr-x 512 r lada sshd 4356 text /usr 64904 -r-xr-xr-x 152388 r lada sshd 4356 0 / 65419 crw-rw-rw- null rw lada sshd 4356 1 / 65419 crw-rw-rw- null rw lada sshd 4356 2 / 65419 crw-rw-rw- null rw lada sshd 4356 3* local stream d559b1e0 <-> d5590aa0 lada sshd 4356 4* pipe d4d8d340 <-> d4d8d200 0 rw lada sshd 4356 5* internet stream tcp d2e85500 lada sshd 4356 6* pipe d4d8d200 <-> d4d8d340 0 rw lada sshd 4356 7 / 65632 crw-rw-rw- ptyp0 rw lada sshd 4356 8 / 65632 crw-rw-rw- ptyp0 rw lada sshd 4356 9 / 65632 crw-rw-rw- ptyp0 rw root sshd 4353 root / 2 drwxr-xr-x 512 r root sshd 4353 wd / 2 drwxr-xr-x 512 r root sshd 4353 text /usr 64904 -r-xr-xr-x 152388 r root sshd 4353 0 / 65419 crw-rw-rw- null r root sshd 4353 1 / 65419 crw-rw-rw- null rw root sshd 4353 2 / 65419 crw-rw-rw- null rw root sshd 4353 3 / 65632 crw-rw-rw- ptyp0 rw root sshd 4353 4* local stream d5590aa0 <-> d559b1e0 root sshd 4353 5* internet stream tcp d2e85500 root getty 215 root / 2 drwxr-xr-x 512 r root getty 215 wd / 2 drwxr-xr-x 512 r root getty 215 text /usr 22848 -r-xr-xr-x 21308 r root getty 215 0 / 65626 crw------- ttyv7 rw root getty 215 1 / 65626 crw------- ttyv7 rw root getty 215 2 / 65626 crw------- ttyv7 rw root getty 214 root / 2 drwxr-xr-x 512 r root getty 214 wd / 2 drwxr-xr-x 512 r root getty 214 text /usr 22848 -r-xr-xr-x 21308 r root getty 214 0 / 65625 crw------- ttyv6 rw root getty 214 1 / 65625 crw------- ttyv6 rw root getty 214 2 / 65625 crw------- ttyv6 rw root getty 213 root / 2 drwxr-xr-x 512 r root getty 213 wd / 2 drwxr-xr-x 512 r root getty 213 text /usr 22848 -r-xr-xr-x 21308 r root getty 213 0 / 65621 crw------- ttyv5 rw root getty 213 1 / 65621 crw------- ttyv5 rw root getty 213 2 / 65621 crw------- ttyv5 rw root getty 212 root / 2 drwxr-xr-x 512 r root getty 212 wd / 2 drwxr-xr-x 512 r root getty 212 text /usr 22848 -r-xr-xr-x 21308 r root getty 212 0 / 65412 crw------- ttyv4 rw root getty 212 1 / 65412 crw------- ttyv4 rw root getty 212 2 / 65412 crw------- ttyv4 rw root getty 211 root / 2 drwxr-xr-x 512 r root getty 211 wd / 2 drwxr-xr-x 512 r root getty 211 text /usr 22848 -r-xr-xr-x 21308 r root getty 211 0 / 65411 crw------- ttyv3 rw root getty 211 1 / 65411 crw------- ttyv3 rw root getty 211 2 / 65411 crw------- ttyv3 rw root getty 210 root / 2 drwxr-xr-x 512 r root getty 210 wd / 2 drwxr-xr-x 512 r root getty 210 text /usr 22848 -r-xr-xr-x 21308 r root getty 210 0 / 65410 crw------- ttyv2 rw root getty 210 1 / 65410 crw------- ttyv2 rw root getty 210 2 / 65410 crw------- ttyv2 rw root getty 209 root / 2 drwxr-xr-x 512 r root getty 209 wd / 2 drwxr-xr-x 512 r root getty 209 text /usr 22848 -r-xr-xr-x 21308 r root getty 209 0 / 65405 crw------- ttyv1 rw root getty 209 1 / 65405 crw------- ttyv1 rw root getty 209 2 / 65405 crw------- ttyv1 rw root getty 208 root / 2 drwxr-xr-x 512 r root getty 208 wd / 2 drwxr-xr-x 512 r root getty 208 text /usr 22848 -r-xr-xr-x 21308 r root getty 208 0 / 65404 crw------- ttyv0 rw root getty 208 1 / 65404 crw------- ttyv0 rw root getty 208 2 / 65404 crw------- ttyv0 rw cvsupd cvsupd 183 root / 2 drwxr-xr-x 512 r cvsupd cvsupd 183 wd / 2 drwxr-xr-x 512 r cvsupd cvsupd 183 text /usr 101735 -rwxr-xr-x 890916 r cvsupd cvsupd 183 0 / 65419 crw-rw-rw- null rw cvsupd cvsupd 183 1 / 65419 crw-rw-rw- null rw cvsupd cvsupd 183 2 / 65419 crw-rw-rw- null rw cvsupd cvsupd 183 3* internet stream tcp d2e34500 cvsupd cvsupd 183 4* pipe d4d8df20 <-> d4d8de80 0 rw cvsupd cvsupd 183 5* pipe d4d8de80 <-> d4d8df20 0 rw cvsupd cvsupd 183 6* local dgram d4da1c80 <-> d4da1f00 root proftpd 173 root / 2 drwxr-xr-x 512 r root proftpd 173 wd / 2 drwxr-xr-x 512 r root proftpd 173 text /usr 43658 -r-xr-xr-x 364456 r root proftpd 173 0* internet stream tcp d2e4a880 root proftpd 173 1 / 86775 -rw-r--r-- 21664 rw root systrace 172 root / 2 drwxr-xr-x 512 r root systrace 172 wd / 2 drwxr-xr-x 512 r root systrace 172 text / 22304 -r-xr-xr-x 326948 r root systrace 172 0 - - none - root systrace 172 1 - - none - root systrace 172 2 - - none - root systrace 172 4* local dgram d4ffdf00 <-> d4da1f00 root snmpd 169 root / 2 drwxr-xr-x 512 r root snmpd 169 wd / 2 drwxr-xr-x 512 r root snmpd 169 text /usr 101260 -rwxr-xr-x 17416 r root snmpd 169 0 - - none - root snmpd 169 1 - - none - root snmpd 169 2 - - none - root snmpd 169 3 /var/log 3 -rw-r--r-- 2186257 w root snmpd 169 4 / 65155 crw-r----- mem r root snmpd 169 5 / 65151 crw-r----- kmem r root snmpd 169 6* internet dgram udp d2834d40 root snmpd 169 7 / 65155 crw-r----- mem r root snmpd 169 8 / 65151 crw-r----- kmem r root snmpd 169 9 / 86616 -rw-r--r-- 65536 r smmsp sendmail 154 root / 2 drwxr-xr-x 512 r smmsp sendmail 154 wd / 86589 drwxrwx--- 512 r smmsp sendmail 154 text /usr 22965 -r-xr-sr-x 586520 r smmsp sendmail 154 0 / 65419 crw-rw-rw- null r smmsp sendmail 154 1 / 65419 crw-rw-rw- null w smmsp sendmail 154 2 / 65419 crw-rw-rw- null w smmsp sendmail 154 3* local dgram d4da1d20 <-> d4da1f00 root sendmail 151 root / 2 drwxr-xr-x 512 r root sendmail 151 wd / 86578 drwxr-xr-x 512 r root sendmail 151 text /usr 22965 -r-xr-sr-x 586520 r root sendmail 151 0 / 65419 crw-rw-rw- null r root sendmail 151 1 / 65419 crw-rw-rw- null w root sendmail 151 2 / 65419 crw-rw-rw- null w root sendmail 151 3* internet stream tcp d2e34bc0 root sendmail 151 4* local dgram d4da1dc0 <-> d4da1f00 root sendmail 151 5* internet stream tcp d2e34980 root sshd 148 root / 2 drwxr-xr-x 512 r root sshd 148 wd / 2 drwxr-xr-x 512 r root sshd 148 text /usr 64904 -r-xr-xr-x 152388 r root sshd 148 0 / 65419 crw-rw-rw- null rw root sshd 148 1 / 65419 crw-rw-rw- null rw root sshd 148 2 / 65419 crw-rw-rw- null rw root sshd 148 3* internet6 stream tcp d2e33e40 root sshd 148 4* internet stream tcp d2e34e00 root cron 146 root / 2 drwxr-xr-x 512 r root cron 146 wd / 86547 drwxr-x--- 512 r root cron 146 text /usr 70345 -r-xr-xr-x 30280 r root cron 146 0 / 65419 crw-rw-rw- null rw root cron 146 1 / 65419 crw-rw-rw- null rw root cron 146 2 / 65419 crw-rw-rw- null rw root cron 146 3 / 86723 -rw-r--r-- 4 rw root syslogd 137 root / 2 drwxr-xr-x 512 r root syslogd 137 wd / 2 drwxr-xr-x 512 r root syslogd 137 text /usr 74670 -r-xr-xr-x 29924 r root syslogd 137 1 / 65419 crw-rw-rw- null rw root syslogd 137 2 / 65419 crw-rw-rw- null rw root syslogd 137 3* local dgram d4da1f00 root syslogd 137 4* internet6 dgram udp d2834ec0 root syslogd 137 5* internet dgram udp d2834e00 root syslogd 137 6 / 65420 crw------- klog r root syslogd 137 7 / 65024 crw------- console w root syslogd 137 8 /var/log 10 -rw-rw-r-- 1171840 w root syslogd 137 9 /var/log 5 -rw-r--r-- 0 w root syslogd 137 10 /var/log 6 -rw-r--r-- 143901565 w root syslogd 137 11 /var/log 44 -rw-r--r-- 58 w root syslogd 137 12 /var/log 8 -rw-r--r-- 0 w root syslogd 137 13 /var/log 14 -rw------- 36776 w root syslogd 137 14 /var/log 33 -rw-r----- 93403 w root ipmon 87 root / 2 drwxr-xr-x 512 r root ipmon 87 wd / 2 drwxr-xr-x 512 r root ipmon 87 text / 127 -r-xr-xr-x 184004 r root ipmon 87 0* local dgram d4da1e60 <-> d4da1f00 root ipmon 87 2 / 65419 crw-rw-rw- null rw root ipmon 87 3 / 65652 crw------- ipl r root init 1 root / 2 drwxr-xr-x 512 r root init 1 wd / 2 drwxr-xr-x 512 r root init 1 text / 129 -r-x------ 280656 r netstat -m output: 100/1696/131072 mbufs in use (current/peak/max): 99 mbufs allocated to data 1 mbufs allocated to packet headers 97/1350/32768 mbuf clusters in use (current/peak/max) 3124 Kbytes allocated to network (3% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines netstat -tan output: Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 48 195.250.155.36.22 195.122.204.155.2342 ESTABLISHED tcp4 0 0 195.250.155.36.21 195.122.207.242.3976 CLOSE_WAIT tcp4 6 0 195.250.155.36.21 212.20.108.177.3362 CLOSE_WAIT tcp4 0 0 195.250.155.36.21 195.250.155.26.2275 CLOSE_WAIT tcp4 0 0 195.250.155.36.21 213.45.28.28.2299 CLOSE_WAIT tcp4 0 0 195.250.155.36.21 195.122.207.242.3894 CLOSE_WAIT tcp4 0 0 *.21 *.* LISTEN tcp4 0 30913 195.250.155.36.20 80.143.57.187.3518 FIN_WAIT_1 tcp4 0 0 195.250.155.36.22 195.122.204.153.63635 ESTABLISHED tcp4 0 0 *.5999 *.* LISTEN tcp4 0 0 *.587 *.* LISTEN tcp4 0 0 *.25 *.* LISTEN tcp4 0 0 *.22 *.* LISTEN tcp46 0 0 *.22 *.* LISTEN udp4 0 0 *.161 *.* udp4 0 0 *.514 *.* udp6 0 0 *.514 *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr d4da1000 stream 0 0 0 d50c5320 0 0 d50c5320 stream 0 0 0 d4da1000 0 0 d5590aa0 stream 0 0 0 d559b1e0 0 0 d559b1e0 stream 0 0 0 d5590aa0 0 0 d4da1e60 dgram 0 0 0 d4da1f00 0 d4ffdf00 d4ffdf00 dgram 0 0 0 d4da1f00 0 d4da1c80 d4da1c80 dgram 0 0 0 d4da1f00 0 d4da1d20 d4da1d20 dgram 0 0 0 d4da1f00 0 d4da1dc0 d4da1dc0 dgram 0 0 0 d4da1f00 0 0 d4da1f00 dgram 0 0 d4d9c140 0 d4da1e60 0 /var/run/log ps output: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 7753 0.0 0.1 424 212 p1 R+ 12:48AM 0:00.00 ps auxwww root 1 0.0 0.0 552 72 ?? ILs 25Feb04 0:00.27 /sbin/init -- root 2 0.0 0.0 0 0 ?? DL 25Feb04 0:00.00 (taskqueue) root 3 0.0 0.0 0 0 ?? DL 25Feb04 0:08.79 (pagedaemon) root 4 0.0 0.0 0 0 ?? DL 25Feb04 0:00.27 (vmdaemon) root 5 0.0 0.0 0 0 ?? DL 25Feb04 0:27.79 (bufdaemon) root 6 0.0 0.0 0 0 ?? DL 25Feb04 0:44.54 (vnlru) root 7 0.0 0.0 0 0 ?? DL 25Feb04 4:45.89 (syncer) root 87 0.0 0.1 928 164 ?? Ss 25Feb04 0:18.36 /sbin/ipmon -Ds root 137 0.0 0.2 1184 476 ?? Ss 25Feb04 0:28.15 /usr/sbin/syslogd -s root 146 0.0 0.1 1024 240 ?? Is 25Feb04 0:07.97 /usr/sbin/cron root 148 0.0 0.1 2596 316 ?? Is 25Feb04 0:01.26 /usr/sbin/sshd root 151 0.0 0.4 3056 944 ?? Ss 25Feb04 0:25.19 sendmail: accepting connections (sendmail) smmsp 154 0.0 0.2 2928 620 ?? Is 25Feb04 0:00.37 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail) root 169 0.0 0.6 7200 1456 con- I 25Feb04 8:52.27 /usr/local/sbin/snmpd -p 161 root 172 0.0 0.2 732 432 ?? Is 25Feb04 43:16.54 systrace -aUi /usr/local/libexec/proftpd root 173 0.0 0.6 2224 1476 ?? Is 25Feb04 1:23.24 proftpd: (accepting connections) (proftpd) cvsupd 183 0.0 0.4 2344 1044 ?? Ss 25Feb04 0:36.39 /usr/local/sbin/cvsupd -C 16 -l @daemon root 208 0.0 0.0 952 0 v0 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv0 root 209 0.0 0.0 952 0 v1 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv1 root 210 0.0 0.0 952 0 v2 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv2 root 211 0.0 0.0 952 0 v3 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv3 root 212 0.0 0.0 952 0 v4 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv4 root 213 0.0 0.0 952 0 v5 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv5 root 214 0.0 0.0 952 0 v6 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv6 root 215 0.0 0.0 952 0 v7 IWs+ - 0:00.00 /usr/libexec/getty Pc ttyv7 root 4353 0.0 0.6 5296 1532 ?? I 10:27PM 0:00.03 sshd: lada [priv] (sshd) lada 4356 0.0 0.6 5292 1600 ?? I 10:27PM 0:00.21 sshd: lada@ttyp0 (sshd) lada 4357 0.0 0.3 1100 752 p0 Is 10:27PM 0:00.01 -bash (bash) root 4370 0.0 0.3 1100 760 p0 I+ 10:27PM 0:00.08 su (bash) root 7638 0.0 0.6 5292 1528 ?? I 12:44AM 0:00.02 sshd: malyl [priv] (sshd) malyl 7641 0.0 0.6 5292 1600 ?? S 12:44AM 0:00.05 sshd: malyl@ttyp1 (sshd) malyl 7642 0.0 0.3 1080 780 p1 Is 12:44AM 0:00.01 -bash (bash) root 7729 0.0 0.3 1112 816 p1 S 12:45AM 0:00.03 -su (bash) root 7749 0.0 0.4 3160 1092 p1 T 12:47AM 0:00.01 tcpdump -n -X -s 1500 -w /tmp/malyl/tcpdump.out root 0 0.0 0.0 0 0 ?? DLs 25Feb04 0:00.01 (swapper) ipfstat output: IP states added: 149186 TCP 213009 UDP 288272 ICMP 202080082 hits 897743 misses 0 maximum 0 no memory 22 bkts in use 22 active 501281 expired 149164 closed