Project Home
Project Home		 Source Code
Source Code		 Documents
Documents		 Wiki
Wiki		 Discussion Forums
Discussions		 Project Information
Project Info
QNX Community pkgsrc Project/Discussion/General/OS lockup do to Apache 22/List of Posts
Forum Topic - OS lockup do to Apache 22: (9 Items)
   
Update
Greg Morehead
07/31/2009 10:39 AM
post35041
OS lockup do to Apache 22   
We have been attempting to identify a system lockup for a few months.

Long story short, we found with slinger running everything was stable.

We have suspected apache but had no real proof this was the root cause.  Finally I captured what I believe to be 
definitive evidence. 

After the system locked up, I power cycled and reviewed all the logs.  The last log on the system was apache attempting 
to serve up the includes from a page request. 

In ssl_request_log, I found that only 11 of 21 include files had been served up.  This corresponds with the fact that I 
was waiting forever for my page to load.


This basically kills apache as on option on our system.  Does anyone have a suggestion for a ssl compliant replacement??
?
Sean Boudreau(deleted)
07/31/2009 11:18 AM
post35054
Re: OS lockup do to Apache 22   
On Fri, Jul 31, 2009 at 10:39:51AM -0400, Greg Morehead wrote:
> 
> We have been attempting to identify a system lockup for a few months.
> 
> Long story short, we found with slinger running everything was stable.
> 
> We have suspected apache but had no real proof this was the root cause.  Finally I captured what I believe to be 
definitive evidence. 
> 
> After the system locked up, I power cycled and reviewed all the logs.  The last log on the system was apache 
attempting to serve up the includes from a page request. 
> 
> In ssl_request_log, I found that only 11 of 21 include files had been served up.  This corresponds with the fact that 
I was waiting forever for my page to load.
> 
> 
> This basically kills apache as on option on our system.  Does anyone have a suggestion for a ssl compliant replacement
???
> 

I'd like to try to debug the lockup.  Anything on
the (text mode) console?  Can you try running a
high prio shell and look around while in this state?
What version / platform?  SMP?

Regards,

-seanb
Greg Morehead
07/31/2009 6:00 PM
post35114
Re: OS lockup do to Apache 22   
> I'd like to try to debug the lockup.  Anything on
> the (text mode) console?  Can you try running a
> high prio shell and look around while in this state?
> What version / platform?  SMP?

- We are running the 6.4.1 release
- On x86, intel processor
- Not running SMP

I have seen this on two different boxes.

After reboot everything was working fine again.  Chkfsys had no issues.

This is very intermittent.  Not sure how to replicate this in a timely manner.

The text mode console was locked up as well.  Nothing was in the syslog (we have the syslog writing to a file.)
Mitchell Schoenbrun
07/31/2009 9:31 PM
post35117
Re: OS lockup do to Apache 22   
I think a starting defensive maneuver would be to run this with the adaptive partition scheduler so that you can always 
run a pidin.
Greg Morehead
08/01/2009 8:36 AM
post35121
Re: OS lockup do to Apache 22   
> I think a starting defensive maneuver would be to run this with the adaptive 
> partition scheduler so that you can always run a pidin.

Early on in our attempts to find the root cause of the lockup we used pidin extensively and logged a bunch of info to 
files in order to catch any offending process with unexpected performance.  

All processes, including apache, were very stable up to the point of lockup.  Every indication pointed to an 
instantaneous anomaly that completely locks the system.
Mitchell Schoenbrun
07/31/2009 1:01 PM
post35079
Re: OS lockup do to Apache 22   
I have a minor interest in this as I run Apache 22.   In the past I
setup SSL and have tested it, but don't run anything against it
normally.   I don't own a registered certificate.

Reading the info I've been trying to think a few things through.   In
general, open source like Apache and SSL have to be very high quality.
 They get run all over the place all the time.   On top of that, at
least where I am, they get compiled with GCC another open source.   So
the code being produced should be quite predicable in some sense and
should work quite predictably.

So where may the problem dwell?
1) It could be a bug in either Apache or SSL.  Very unlikely but possible.
2) It could be some subtle difference in the way some QNX system call works.
3) It could be, as Sean alludes to, related to thread interaction on
QNX running multi-processor.
4) It could be a site specific anomaly.

An example of 4) would be a file to be served that can't be read
properly on disk.

I've put these in the order that I think is increasing likely hood.
If it will be of any help, I would be willing to put the SSL pages on
my system where we could see if the problem reproduces here.   If the
problem shows up, then we've at least eliminated 4) and I could look a
little closer at 1-3.   There's nothing crucial running on my system
so I can tolerate it crashing while we look.

Mitchell
Mario Charest
07/31/2009 1:02 PM
post35080
RE: OS lockup do to Apache 22   

> -----Original Message-----
> From: Mitchell Schoenbrun [mailto:community-noreply@qnx.com]
> Sent: Friday, July 31, 2009 1:01 PM
> To: general-pkgsrc
> Subject: Re: OS lockup do to Apache 22
> 
> So where may the problem dwell?
> 1) It could be a bug in either Apache or SSL.  Very unlikely but
> possible.
> 2) It could be some subtle difference in the way some QNX system call
> works.
> 3) It could be, as Sean alludes to, related to thread interaction on
> QNX running multi-processor.

To rule this problem out, run Apache with the affinity set to only one core.

> 4) It could be a site specific anomaly.
> 
> An example of 4) would be a file to be served that can't be read
> properly on disk.
> 
> I've put these in the order that I think is increasing likely hood.
> If it will be of any help, I would be willing to put the SSL pages on
> my system where we could see if the problem reproduces here.   If the
> problem shows up, then we've at least eliminated 4) and I could look a
> little closer at 1-3.   There's nothing crucial running on my system
> so I can tolerate it crashing while we look.
> 
> Mitchell
> 
> 
> 
> 
> _______________________________________________
> 
> General
> http://community.qnx.com/sf/go/post35079
>
Mitchell Schoenbrun
07/31/2009 1:07 PM
post35081
Re: RE: OS lockup do to Apache 22   
> 
> To rule this problem out, run Apache with the affinity set to only one core.
> 

It would rule out some problems but not all.   I had a problem with 6.2.1 running multi-core with an Adaptec SCSI driver
.   I agree that this is not an Apache or SSL problem, but we don't know what the actual problem is yet.   Since it 
occurred at the time the disk was being read.

On the other hand, running the non-multi-core OS and seeing the problem would rule it out.
Greg Morehead
08/25/2009 2:41 PM
post36663
Re: RE: OS lockup do to Apache 22   
I have finally found the time to get back to this issue.  Has anybody attempted to replicate this?

Any suggestions on how I should begin addressing this issue?

Fortunately we have been able to keep the product locked behind a protected firewall, but this will change soon and I 
need a SSL enabled web server very soon.

Return


  A subsidiary of BlackBerry             All content ©2004-2013, QNX Software Systems