|
Greg Morehead
|
Problem with slinger crashing.
|
|
Greg Morehead
02/04/2009 8:56 AM
post21394
|
Problem with slinger crashing.
I have a cgi script which is being called every two seconds and producing about 17kb of xml formatted text.
After about an hour or so slinger will simply crash. I restart it and it all works fine again, for a while.
I am running the command slinger -des &. However, I am not seeing any debug output in the log.
I'm running 6.4.0 on a VMWare install.
Any ideas?
|
|
|
|
|
|
Greg Morehead
|
Re: Problem with slinger crashing.
|
|
Greg Morehead
02/05/2009 8:01 AM
post21469
|
Re: Problem with slinger crashing.
Correction; I found the following output in the log:
Feb 04 11:12:37 5 21 0 run fault pid 3399719 tid 11 signal 11 code 1 ip 0xb035a428 usr/sbin/slinger
|
|
|
|
|
|
Thomas Haupt
|
RE: Problem with slinger crashing.
|
|
Thomas Haupt
02/05/2009 10:33 AM
post21485
|
RE: Problem with slinger crashing.
Hi Greg,
can you post the core file (propably in /var/dumps) ?
- Thomas
> -----Original Message-----
> From: Greg Morehead [mailto:community-noreply@qnx.com]
> Sent: 05 February 2009 14:02
> To: ostech-core_os
> Subject: Re: Problem with slinger crashing.
>
> Correction; I found the following output in the log:
>
> Feb 04 11:12:37 5 21 0 run fault pid 3399719 tid 11
> signal 11 code 1 ip 0xb035a428 usr/sbin/slinger
>
>
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post21469
>
>
|
|
|
|
|
|
Thomas Haupt
|
RE: Problem with slinger crashing.
|
|
Thomas Haupt
02/06/2009 11:11 AM
post21615
|
RE: Problem with slinger crashing.
Hey Greg,
I was able to precisely reproduce the fault, same address and all.
Here's what happened:
Before invoking a cgi script or parsing an SSI, slinger sets up a
number of environment variables for the script. One of them is
"SERVER_ROOT", which is the current working directory. This
environment variable is prepared by the line
snprintf(buffer, ENV_BUF, "SERVER_ROOT=%s", getcwd(pathbuf, PATH_MAX
+ 1));
Before executing a CGI script, slinger chdir()s into the cgi-bin
specified in HTTPD_SCRIPTALIAS, so once you ran a script,
slinger's cwd will be the cgi-bin.
Now to crash the server, I
...first requested a script (thus changing slinger working dir),
...removed that directory and
...then requested an SSI.
If you try to run another cgi script when the cgi-bin is gone,
you'll just get an error message, hence the trick using SSI.
I looked at a core file supplied by another customer (one of your
colleagues?) with pretty much the same problem, just under 6.3.2.
According to that, getcwd returned with errno=2, which would be
ENOENT (no such file or directory).
If you can provide your slinger.core, I'll double-check.
Do you think something as described above might have happened?
- Thomas
> -----Original Message-----
> From: Greg Morehead [mailto:community-noreply@qnx.com]
> Sent: 05 February 2009 14:02
> To: ostech-core_os
> Subject: Re: Problem with slinger crashing.
>
> Correction; I found the following output in the log:
>
> Feb 04 11:12:37 5 21 0 run fault pid 3399719 tid 11
> signal 11 code 1 ip 0xb035a428 usr/sbin/slinger
>
>
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post21469
>
>
|
|
|
|
|
|
Greg Morehead
|
Re: RE: Problem with slinger crashing.
|
|
Greg Morehead
02/06/2009 11:23 AM
post21616
|
Re: RE: Problem with slinger crashing.
I've included the core file.
I don't know how the cgi-bin directory could be removed or deleted.
I have set the cgi-bin using the command below in rc.local
export HTTPD_SCRIPTALIAS=/usr/web/cgi
And the /usr/web/cgi directory is always there.
|
|
|
|
|
|
Greg Morehead
|
Re: RE: Problem with slinger crashing.
|
|
Greg Morehead
02/06/2009 11:24 AM
post21617
|
Re: RE: Problem with slinger crashing.
|
|
|
|
|
|
Thomas Haupt
|
Re: RE: Problem with slinger crashing.
|
|
Thomas Haupt
02/06/2009 9:27 PM
post21698
|
Re: RE: Problem with slinger crashing.
Hi Greg,
you're right - there is yet another, more serious problem. It is caused by an HTTP GET request with a long query string
- in your case, the query string by itself is already 520 characters long.
While this doesn't cause a buffer overflow, it invokes a code path in slinger that's obviously supposed to prevent query
strings longer than 256 characters from being passed to the cgi script as a command line argument. Unfortunately, this
code path forgets to append a NULL to the argv[] vector that's later used to run the script...
Below you can see the URL that must have been referred to when the crash occured; try to access it -- it should crash
slinger right away, given I got the IP, path and script name right.
http://10.0.5.195/cgi-bin/TF2000cmd?COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936626477COMMAND=EXPORT_SHMEM_XML&QUAD=0&
rndval=1233936631312COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936633946COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
1233936635949COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936637949COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
1233936640128COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936642449COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
1233936644447COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936646450COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936648583
You might also try the experimental slinger attached (I hope!) to this post, it's got the two bugs fixed that I found so
far.
Cheers,
- Thomas
|
|
|
|
|
|
Greg Morehead
|
Re: RE: Problem with slinger crashing.
|
|
Greg Morehead
02/09/2009 8:40 AM
post21731
|
Re: RE: Problem with slinger crashing.
Thank you very much Thomas.
You were correct, that URL crashed slinger instantly. Whereas the version you sent me did not.
I will also look into and fix the JS code. I'm using a small ajax code snippet which obviously needs a little tlc.
Regards,
Greg
|
|
|
|
|
|
Thomas Haupt
|
RE: RE: Problem with slinger crashing.
|
|
Thomas Haupt
02/09/2009 11:47 AM
post21773
|
RE: RE: Problem with slinger crashing.
Hi Greg,
good to hear.
Were these URLs extreme test cases, or are you actually
expecting to transmit larger quantities of data in your
queries? If so you might consider using POST instead of
GET... slinger is only a rather simple web server, and
as far as i can see, it's not designed to handle infintely
large URLs...
By the way, the fixes are already on their way into the
product. Thanks again for pointing this issue out.
Regards,
- Thomas
> -----Original Message-----
> From: Greg Morehead [mailto:community-noreply@qnx.com]
> Sent: 09 February 2009 14:41
> To: ostech-core_os
> Subject: Re: RE: Problem with slinger crashing.
>
> Thank you very much Thomas.
>
> You were correct, that URL crashed slinger instantly.
> Whereas the version you sent me did not.
>
> I will also look into and fix the JS code. I'm using a small
> ajax code snippet which obviously needs a little tlc.
>
> Regards,
> Greg
>
>
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post21731
>
>
|
|
|
|
|
|
Greg Morehead
|
Re: RE: RE: Problem with slinger crashing.
|
|
Greg Morehead
02/09/2009 1:57 PM
post21794
|
Re: RE: RE: Problem with slinger crashing.
The large URL's are a bug.
However, as I search for the problem I will also investigate using post's instead.
I'm glad to hear my mistakes are making the product more robust. :)
Thanks,
Greg
|
|
|
|
|
|
Todd Peterson(deleted)
|
Re: RE: Problem with slinger crashing.
|
|
Todd Peterson(deleted)
03/26/2013 7:51 PM
post100189
|
Re: RE: Problem with slinger crashing.
Thomas,
We have a similar problem, but on the ARM platform. Could you provide the source code for your experimental version of
slinger? Or the executable for ARM? We get no dump file, since our image does not have a /var/dumps directory. We do not
get a syslog message either. No message in console window either.
Thanks,
Todd
> Hi Greg,
>
> you're right - there is yet another, more serious problem. It is caused by an
> HTTP GET request with a long query string - in your case, the query string by
> itself is already 520 characters long.
>
> While this doesn't cause a buffer overflow, it invokes a code path in slinger
> that's obviously supposed to prevent query strings longer than 256 characters
> from being passed to the cgi script as a command line argument. Unfortunately,
> this code path forgets to append a NULL to the argv[] vector that's later
> used to run the script...
>
> Below you can see the URL that must have been referred to when the crash
> occured; try to access it -- it should crash slinger right away, given I got
> the IP, path and script name right.
>
> http://10.0.5.195/cgi-bin/TF2000cmd?COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
> 1233936626477COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936631312COMMAND=
> EXPORT_SHMEM_XML&QUAD=0&rndval=1233936633946COMMAND=EXPORT_SHMEM_XML&QUAD=0&
>
rndval=1233936635949COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
> 1233936637949COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936640128COMMAND=
> EXPORT_SHMEM_XML&QUAD=0&rndval=1233936642449COMMAND=EXPORT_SHMEM_XML&QUAD=0&
> rndval=
1233936644447COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=
> 1233936646450COMMAND=EXPORT_SHMEM_XML&QUAD=0&rndval=1233936648583
>
> You might also try the experimental slinger attached (I hope!) to this post,
> it's got the two bugs fixed that I found so far.
>
> Cheers,
> - Thomas
|
|
|
|
|
|
Thomas Haupt
|
Re: RE: Problem with slinger crashing.
|
|
Thomas Haupt
03/27/2013 2:15 AM
post100192
|
Re: RE: Problem with slinger crashing.
Hi Todd,
unfortunately, we cannot provide source code any more. The fix I made back then should long be in the product; I checked
it in on Wed, Feb 11 2009, at 16:20:48. It it thus part of all releases at least since 6.5.0.
What version of QNX Neutrino are you running? If it's < 6.5.0, I can probably build you an experimental binary with my
patch applied, if it is >= 6.5.0, then you are probably facing a different bug. In the latter case, I'd need a core file
, or an exact description for how to reproduce the issue, or both.
Kind regards,
Thomas
|
|
|
|
|
|
Todd Peterson(deleted)
|
Re: RE: Problem with slinger crashing.
|
|
Todd Peterson(deleted)
03/27/2013 10:05 AM
post100201
|
Re: RE: Problem with slinger crashing.
We are using 6.4.1. I see in release notes that your bug fixes should be there. Besides, I am sure that it is not the
too long query string bug. I will try the dumper command to see what I get.
Thanks,
Todd
> Hi Todd,
>
> unfortunately, we cannot provide source code any more. The fix I made back
> then should long be in the product; I checked it in on Wed, Feb 11 2009, at 16
> :20:48. It it thus part of all releases at least since 6.5.0.
>
> What version of QNX Neutrino are you running? If it's < 6.5.0, I can probably build you an experimental binary with my
patch applied, if it is >= 6.5.0, then you are
> probably facing a different bug. In the latter case, I'd need a core file, or
> an exact description for how to reproduce the issue, or both.
>
> Kind regards,
> Thomas
|
|
|
|
|
|
Thomas Haupt
|
AW: Re: RE: Problem with slinger crashing.
|
|
Thomas Haupt
03/27/2013 11:06 AM
post100204
|
AW: Re: RE: Problem with slinger crashing.
Hi Todd,
Please do so. I'm rather curious.
Cheers,
Thomas
----- Originalnachricht -----
Von: Todd Peterson [mailto:community-noreply@qnx.com]
Gesendet: Wednesday, March 27, 2013 10:05 AM
An: ostech-core_os <ostech-core_os@community.qnx.com>
Betreff: Re: RE: Problem with slinger crashing.
We are using 6.4.1. I see in release notes that your bug fixes should be there. Besides, I am sure that it is not the
too long query string bug. I will try the dumper command to see what I get.
Thanks,
Todd
> Hi Todd,
>
> unfortunately, we cannot provide source code any more. The fix I made back
> then should long be in the product; I checked it in on Wed, Feb 11 2009, at 16
> :20:48. It it thus part of all releases at least since 6.5.0.
>
> What version of QNX Neutrino are you running? If it's < 6.5.0, I can probably build you an experimental binary with my
patch applied, if it is >= 6.5.0, then you are
> probably facing a different bug. In the latter case, I'd need a core file, or
> an exact description for how to reproduce the issue, or both.
>
> Kind regards,
> Thomas
_______________________________________________
OSTech
http://community.qnx.com/sf/go/post100201
To cancel your subscription to this discussion, please e-mail ostech-core_os-unsubscribe@community.qnx.com
|
|
|
|
|
|
Todd Peterson(deleted)
|
Re: AW: Re: RE: Problem with slinger crashing.
|
|
Todd Peterson(deleted)
03/27/2013 1:35 PM
post100206
|
Re: AW: Re: RE: Problem with slinger crashing.
Thomas,
I think I had a version 6.4.0 of slinger running. md5sum did not match slinger in my 6.4.1 development environment
(28dac151ff397d6d8537d78634552798). It was crashing after about 20 min, it has been running todat for well over an hour.
Thanks for your help.
Todd
P.S. Is there any plan for an https version of slinger? We have been developing one, but it is a pain in the butt
starting from scratch.
> Hi Todd,
>
> Please do so. I'm rather curious.
>
> Cheers,
> Thomas
>
> ----- Originalnachricht -----
> Von: Todd Peterson [mailto:community-noreply@qnx.com]
> Gesendet: Wednesday, March 27, 2013 10:05 AM
> An: ostech-core_os <ostech-core_os@community.qnx.com>
> Betreff: Re: RE: Problem with slinger crashing.
>
> We are using 6.4.1. I see in release notes that your bug fixes should be there
> . Besides, I am sure that it is not the too long query string bug. I will try
> the dumper command to see what I get.
> Thanks,
> Todd
> > Hi Todd,
> >
> > unfortunately, we cannot provide source code any more. The fix I made back
> > then should long be in the product; I checked it in on Wed, Feb 11 2009, at
> 16
> > :20:48. It it thus part of all releases at least since 6.5.0.
> >
> > What version of QNX Neutrino are you running? If it's < 6.5.0, I can probably build you an experimental binary with
my patch applied, if it is >= 6.5.0, then you are
>
> > probably facing a different bug. In the latter case, I'd need a core file,
> or
> > an exact description for how to reproduce the issue, or both.
> >
> > Kind regards,
> > Thomas
>
>
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post100201
> To cancel your subscription to this discussion, please e-mail ostech-core_os-
> unsubscribe@community.qnx.com
|
|
|
|
|
|
Sean Boudreau(deleted)
|
Re: AW: Re: RE: Problem with slinger crashing.
|
|
Sean Boudreau(deleted)
03/27/2013 1:37 PM
post100207
|
Re: AW: Re: RE: Problem with slinger crashing.
Try bozohttp.
Builds from pkgsrc and is tiny.
On Wed, Mar 27, 2013 at 01:35:00PM -0400, Todd Peterson wrote:
> Thomas,
> I think I had a version 6.4.0 of slinger running. md5sum did not match slinger in my 6.4.1 development environment
(28dac151ff397d6d8537d78634552798). It was crashing after about 20 min, it has been running todat for well over an hour.
> Thanks for your help.
> Todd
> P.S. Is there any plan for an https version of slinger? We have been developing one, but it is a pain in the butt
starting from scratch.
>
> > Hi Todd,
> >
> > Please do so. I'm rather curious.
> >
> > Cheers,
> > Thomas
> >
> > ----- Originalnachricht -----
> > Von: Todd Peterson [mailto:community-noreply@qnx.com]
> > Gesendet: Wednesday, March 27, 2013 10:05 AM
> > An: ostech-core_os <ostech-core_os@community.qnx.com>
> > Betreff: Re: RE: Problem with slinger crashing.
> >
> > We are using 6.4.1. I see in release notes that your bug fixes should be there
> > . Besides, I am sure that it is not the too long query string bug. I will try
> > the dumper command to see what I get.
> > Thanks,
> > Todd
> > > Hi Todd,
> > >
> > > unfortunately, we cannot provide source code any more. The fix I made back
> > > then should long be in the product; I checked it in on Wed, Feb 11 2009, at
> > 16
> > > :20:48. It it thus part of all releases at least since 6.5.0.
> > >
> > > What version of QNX Neutrino are you running? If it's < 6.5.0, I can probably build you an experimental binary
with my patch applied, if it is >= 6.5.0, then you are
> >
> > > probably facing a different bug. In the latter case, I'd need a core file,
> > or
> > > an exact description for how to reproduce the issue, or both.
> > >
> > > Kind regards,
> > > Thomas
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> >
> > OSTech
> > http://community.qnx.com/sf/go/post100201
> > To cancel your subscription to this discussion, please e-mail ostech-core_os-
> > unsubscribe@community.qnx.com
>
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post100206
> To cancel your subscription to this discussion, please e-mail ostech-core_os-unsubscribe@community.qnx.com
|
|
|
|
|
|
Todd Peterson(deleted)
|
Re: AW: Re: RE: Problem with slinger crashing.
|
|
Todd Peterson(deleted)
03/27/2013 7:24 PM
post100212
|
Re: AW: Re: RE: Problem with slinger crashing.
The newer version exits as well. No dump produced or diagnostics in syslog. Nothing gets printed to console window. Are
there 'normal' exit conditions? If so, could these be instrumented to print a message of some sort? For now, I wrote a
script that checks if slinger is running every 5 secs and restarts if it isn't. Ugly way to do it.
> Thomas,
> I think I had a version 6.4.0 of slinger running. md5sum did not match slinger
> in my 6.4.1 development environment (28dac151ff397d6d8537d78634552798). It
> was crashing after about 20 min, it has been running todat for well over an
> hour.
> Thanks for your help.
> Todd
>
|
|
|
|
|
|
Sean Boudreau(deleted)
|
Re: AW: Re: RE: Problem with slinger crashing.
|
|
Sean Boudreau(deleted)
03/27/2013 7:29 PM
post100213
|
Re: AW: Re: RE: Problem with slinger crashing.
Attach to it with gdb and let it run. gdb may print if it's being hit with a signal.
Sent from my BlackBerry 10 smartphone.
From: Todd Peterson
Sent: Wednesday, March 27, 2013 7:24 PM
To: ostech-core_os
Reply To: ostech-core_os@community.qnx.com
Subject: Re: AW: Re: RE: Problem with slinger crashing.
The newer version exits as well. No dump produced or diagnostics in syslog. Nothing gets printed to console window. Are
there 'normal' exit conditions? If so, could these be instrumented to print a message of some sort? For now, I wrote a
script that checks if slinger is running every 5 secs and restarts if it isn't. Ugly way to do it.
> Thomas,
> I think I had a version 6.4.0 of slinger running. md5sum did not match slinger
> in my 6.4.1 development environment (28dac151ff397d6d8537d78634552798). It
> was crashing after about 20 min, it has been running todat for well over an
> hour.
> Thanks for your help.
> Todd
>
_______________________________________________
OSTech
http://community.qnx.com/sf/go/post100212
To cancel your subscription to this discussion, please e-mail ostech-core_os-unsubscribe@community.qnx.com
|
|
|
|
|
|
Thomas Haupt
|
AW: Re: AW: Re: RE: Problem with slinger crashing.
|
|
Thomas Haupt
03/28/2013 1:31 AM
post100215
|
AW: Re: AW: Re: RE: Problem with slinger crashing.
You can also set a breakpoint on exit(), to see if it terminates deliberately. If it does, take a bacltrace...
- Thomas
----- Originalnachricht -----
Von: Todd Peterson [mailto:community-noreply@qnx.com]
Gesendet: Wednesday, March 27, 2013 07:24 PM
An: ostech-core_os <ostech-core_os@community.qnx.com>
Betreff: Re: AW: Re: RE: Problem with slinger crashing.
The newer version exits as well. No dump produced or diagnostics in syslog. Nothing gets printed to console window. Are
there 'normal' exit conditions? If so, could these be instrumented to print a message of some sort? For now, I wrote a
script that checks if slinger is running every 5 secs and restarts if it isn't. Ugly way to do it.
> Thomas,
> I think I had a version 6.4.0 of slinger running. md5sum did not match slinger
> in my 6.4.1 development environment (28dac151ff397d6d8537d78634552798). It
> was crashing after about 20 min, it has been running todat for well over an
> hour.
> Thanks for your help.
> Todd
>
_______________________________________________
OSTech
http://community.qnx.com/sf/go/post100212
To cancel your subscription to this discussion, please e-mail ostech-core_os-unsubscribe@community.qnx.com
|
|
|
|
|
|
Gervais Mulongoy
|
Re: Problem with slinger crashing.
|
|
Gervais Mulongoy
03/27/2013 7:45 AM
post100197
|
Re: Problem with slinger crashing.
Hello Todd,
You can always start dumper to point to another directory (option -d i
believe), such as /tmp or /dev/shmem and collect the core file there.
Gervais
|
|
|
|
|
|
Thomas Haupt
|
Re: RE: Problem with slinger crashing.
|
|
Thomas Haupt
02/06/2009 9:33 PM
post21699
|
Re: RE: Problem with slinger crashing.
Oh, by the way...
milliseconds since 0:0:0 1/01/1970 isn't exactly what I'd call 'random'... more like semi-random, if there is such a
thing... ;-)
- Thomas, at 1233974008666
|
|
|
|
|
|
