varnish process segfaulting out caused by the wurfl library

Questions about our enterprise "in the network" WURFL components
kirsten_sekhaolelo
Posts: 18
Joined: Thu May 29, 2014 10:38 am

varnish process segfaulting out caused by the wurfl library

Postby kirsten_sekhaolelo » Thu Jul 03, 2014 3:45 am

A few days after installing WURFL varnish module we experienced downtime. It was caused by the varnish process segfaulting out, this in turn was caused by the wurfl library. Please review the following extract from syslog and advise on a course of action :

Jul 2 07:04:58 lb1 kernel: [5839868.213998] show_signal_msg: 12 callbacks suppressed
Jul 2 07:04:58 lb1 kernel: [5839868.214004] varnishd[29974]: segfault at 18 ip 00007faee096cd12 sp 00007fae447e9f28 error 4 in libstdc++.so.6.0.19[7faee08f8000+e6000]
Jul 2 07:05:11 lb1 varnishd[29184]: Child (29185) not responding to CLI, killing it.
Jul 2 07:05:32 lb1 varnishd[29184]: message repeated 15 times: [ Child (29185) not responding to CLI, killing it.]
Jul 2 07:05:32 lb1 varnishd[29184]: Child (29185) died signal=11 (core dumped)
Jul 2 07:05:32 lb1 varnishd[29184]: Child cleanup complete
Jul 2 07:05:34 lb1 varnishd[29184]: child (30010) Started
Jul 2 07:05:39 lb1 varnishd[29184]: Child (30010) said Child starts
Jul 2 07:05:39 lb1 varnishd[29184]: Child (30010) said /etc/varnish/wurfl.xml:1: parser error : Extra content at the end of the document
Jul 2 07:05:39 lb1 varnishd[29184]: Child (30010) said
Jul 2 07:05:39 lb1 varnishd[29184]: Child (30010) said ^

Regards,

Kirsten

support-all
Posts: 32
Joined: Mon Oct 08, 2012 2:47 am

Re: varnish process segfaulting out caused by the wurfl libr

Postby support-all » Thu Jul 03, 2014 5:17 am

Hi Kirsten,

can you please post an extended extract of syslog please? I think that the syslog' logs starting from July 2, 06:45:00 would be sufficient.

Thank you,
Andrea

kirsten_sekhaolelo
Posts: 18
Joined: Thu May 29, 2014 10:38 am

Re: varnish process segfaulting out caused by the wurfl libr

Postby kirsten_sekhaolelo » Fri Jul 04, 2014 3:20 am

There are no syslog entries before that time that have anything to do with the WURFL libraries, only normal server noise such as :

Jul 2 03:17:01 lb1 CRON[29546]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 04:17:01 lb1 CRON[29574]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 05:17:01 lb1 CRON[29608]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 06:17:01 lb1 CRON[29636]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 2 06:25:01 lb1 CRON[29645]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))


I have to press some urgency on this now or we may have to look at disabling the WURFL module until it's stable, the frequency of segfaults are increasing daily which would make the site unstable. An extract from today alone :

Jul 4 08:23:22 lb1 varnishd[31880]: Manager got SIGINT
Jul 4 08:23:22 lb1 varnishd[31880]: Stopping Child
Jul 4 08:23:30 lb1 varnishd[34213]: Platform: Linux,3.13.0-24-generic,x86_64,-smalloc,-smalloc,-hcritbit
Jul 4 08:23:30 lb1 varnishd[34213]: child (34215) Started
Jul 4 08:23:37 lb1 varnishd[34213]: Child (34215) said Child starts
Jul 4 08:23:37 lb1 varnishd[34213]: Child (34215) said /etc/varnish/wurfl.xml:1: parser error : Extra content at the end of the document
Jul 4 08:23:37 lb1 varnishd[34213]: Child (34215) said
Jul 4 08:23:37 lb1 varnishd[34213]: Child (34215) said ^
Jul 4 09:17:01 lb1 CRON[34280]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 4 09:18:41 lb1 varnishd[34213]: Manager got SIGINT
Jul 4 09:18:41 lb1 varnishd[34213]: Stopping Child
Jul 4 09:18:45 lb1 varnishd[34412]: Platform: Linux,3.13.0-24-generic,x86_64,-smalloc,-smalloc,-hcritbit
Jul 4 09:18:45 lb1 varnishd[34412]: child (34413) Started
Jul 4 09:18:49 lb1 varnishd[34412]: Child (34413) said Child starts
Jul 4 09:18:49 lb1 varnishd[34412]: Child (34413) said /etc/varnish/wurfl.xml:1: parser error : Extra content at the end of the document
Jul 4 09:18:49 lb1 varnishd[34412]: Child (34413) said
Jul 4 09:18:49 lb1 varnishd[34412]: Child (34413) said ^
Jul 4 09:31:25 lb1 varnishd[34412]: Manager got SIGINT
Jul 4 09:31:25 lb1 varnishd[34412]: Stopping Child
Jul 4 09:31:25 lb1 varnishd[34590]: Platform: Linux,3.13.0-24-generic,x86_64,-smalloc,-smalloc,-hcritbit
Jul 4 09:31:25 lb1 varnishd[34590]: child (34591) Started
Jul 4 09:31:29 lb1 varnishd[34590]: Manager got SIGINT
Jul 4 09:31:29 lb1 varnishd[34590]: Stopping Child
Jul 4 09:31:30 lb1 varnishd[34656]: Platform: Linux,3.13.0-24-generic,x86_64,-smalloc,-smalloc,-hcritbit
Jul 4 09:31:30 lb1 varnishd[34656]: child (34657) Started
Jul 4 09:31:34 lb1 varnishd[34656]: Child (34657) said Child starts
Jul 4 09:31:34 lb1 varnishd[34656]: Child (34657) said /etc/varnish/wurfl.xml:1: parser error : Extra content at the end of the document
Jul 4 09:31:34 lb1 varnishd[34656]: Child (34657) said
Jul 4 09:31:34 lb1 varnishd[34656]: Child (34657) said ^
Jul 4 09:50:11 lb1 varnishd[34656]: Manager got SIGINT
Jul 4 09:50:11 lb1 varnishd[34656]: Stopping Child
Jul 4 09:50:12 lb1 varnishd[34824]: Platform: Linux,3.13.0-24-generic,x86_64,-smalloc,-smalloc,-hcritbit
Jul 4 09:50:13 lb1 varnishd[34824]: child (34825) Started
Jul 4 09:50:17 lb1 varnishd[34824]: Child (34825) said Child starts
Jul 4 09:50:17 lb1 varnishd[34824]: Child (34825) said /etc/varnish/wurfl.xml:1: parser error : Extra content at the end of the document
Jul 4 09:50:17 lb1 varnishd[34824]: Child (34825) said
Jul 4 09:50:17 lb1 varnishd[34824]: Child (34825) said ^


Who is online

Users browsing this forum: No registered users and 2 guests