Bug in SteppingStone analyzer causes segmentation faults

[matthias]

I ran into a series of segfaults when activating the stepping stone analyzer. Examining the core dump is unfortunately not very clueful, here are two example sessions:

rogram terminated with signal 11, Segmentation fault.
#0  SteppingStoneEndpoint (this=0x408306c8, e=0x40ecb1e8, m=0x0) at SteppingStone.h:85
85              int NextID()                    { return endp_cnt++; }
(gdb) bt 5
#0  SteppingStoneEndpoint (this=0x408306c8, e=0x40ecb1e8, m=0x0) at SteppingStone.h:85
#1  0x08189ae5 in SteppingStone_Analyzer::Init (this=0x3e6ad5d8) at SteppingStone.cc:175
#2  0x080716fb in Analyzer::InitChildren (this=0x3f0ca928) at Analyzer.cc:297
#3  0x080abdca in DPM::BuildInitialAnalyzerTree (this=0x8300c40, proto=TRANSPORT_TCP, conn=0x43d475f4, data=0x937ca34 "\\�") at DPM.cc:372
#4  0x0817c884 in NetSessions::NewConn (this=0x937d1e8, k=0x42998f58, t=1229040633.3851731, id=0xbfc41aa0, data=0x937ca34 "\\�", proto=6) at Sessions.cc:1196
(More stack frames follow...)
Missing separate debuginfos, use: debuginfo-install e2fsprogs.i386 gcc.i386 glibc.i686 keyutils.i386 krb5.i386 libpcap.i386 libselinux.i386 libtermcap.i386 openssl.i686 zlib.i386
(gdb) up
#1  0x08189ae5 in SteppingStone_Analyzer::Init (this=0x3e6ad5d8) at SteppingStone.cc:175
175             orig_endp = new SteppingStoneEndpoint(TCP()->Orig(), stp_manager);
(gdb) l
170     void SteppingStone_Analyzer::Init()
171             {
172             TCP_ApplicationAnalyzer::Init();
173
174             assert(TCP());
175             orig_endp = new SteppingStoneEndpoint(TCP()->Orig(), stp_manager);
176             resp_endp = new SteppingStoneEndpoint(TCP()->Resp(), stp_manager);
177             }
178
179     void SteppingStone_Analyzer::DeliverPacket(int len, const u_char* data,

Program terminated with signal 11, Segmentation fault.
#0  Dictionary::MemoryAllocation (this=0x93ce9c0) at Dict.cc:557
557                                     size += padded_sizeof(DictEntry) + pad_size((*chain)[j]->len);
(gdb) bt 5
#0  Dictionary::MemoryAllocation (this=0x93ce9c0) at Dict.cc:557
#1  0x0817b871 in NetSessions::MemoryAllocation (this=0x93ce8f8) at Sessions.cc:1463
#2  0x08188723 in ProfileLogger::Log (this=0x93ee900) at Stats.cc:121
#3  0x0804e59d in terminate_bro () at main.cc:272
#4  0x0804fd05 in main (argc=0, argv=0xbfe2fb04) at main.cc:1054
Missing separate debuginfos, use: debuginfo-install e2fsprogs.i386 gcc.i386 glibc.i686 keyutils.i386 krb5.i386 libpcap.i386 libselinux.i386 libtermcap.i386 openssl.i686 zlib.i386
(gdb) l
552             for ( int i = 0; i < num_buckets; ++i )
553                     if ( tbl[i] )
554                             {
555                             PList(DictEntry)* chain = tbl[i];
556                             loop_over_list(*chain, j)
557                                     size += padded_sizeof(DictEntry) + pad_size((*chain)[j]->len);
558                             size += chain->MemoryAllocation();
559                             }
560
561             size += pad_size(num_buckets * sizeof(PList(DictEntry)*));
(gdb) p *this
$1 = {_vptr.Dictionary = 0x8248820, tbl = 0x93ca7e0, num_buckets = 17, num_entries = 0, max_num_entries = 0, den_thresh = 3, thresh_entries = 51, tbl2 = 0x0, num_buckets2 = 0, num_entries2 = 0, max_num_entries2 = 17, 
  den_thresh2 = 3.4584595208887258e-323, thresh_entries2 = 0, tbl_next_ind = 1, order = 0x0, delete_func = 0x8145230 <bro_obj_delete_func(void*)>, cookies = {<BaseList> = {entry = 0x0, chunk_size = 10, max_entries = 0, 
      num_entries = 0}, <No data fields>}}
(gdb) up
#1  0x0817b871 in NetSessions::MemoryAllocation (this=0x93ce8f8) at Sessions.cc:1463
1463                    + fragments.MemoryAllocation() - padded_sizeof(fragments)
(gdb) l
1458                            (tcp_conns.Length() * pad_size(12))
1459                    + udp_conns.MemoryAllocation() - padded_sizeof(udp_conns) -
1460                            (udp_conns.Length() * pad_size(12))
1461                    + icmp_conns.MemoryAllocation() - padded_sizeof(icmp_conns) -
1462                            (icmp_conns.Length() * pad_size(12))
1463                    + fragments.MemoryAllocation() - padded_sizeof(fragments)
1464                    // FIXME: MemoryAllocation() not implemented for rest.
1465                    ;
1466            }
(gdb)

For the first session, I extracted the connection during the crash and re-run Bro only on that. This time though, there was no crash.

The workaround that works reliably is turning the stepping stone analyzer off.

[anonymous]

I could not get this to reproduce using the current trunk.

[vern]

Replying to anonymous:

I could not get this to reproduce using the current trunk.

To clarify - I can reproduce the second problem (crash inside terminate_bro()). However, I don't think this has anything to do with stepping-stone detection, but rather probably relates to  other problems that manifest upon shut-down.

[seth]

I think 2 years is long enough to let this ticket stand in it's current condition. Closing now.