Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
BroadcastCommunity.qnx.com will be offline from May 31 6:00pm until June 2 12:00AM for upcoming system upgrades. For more information please go to https://community.qnx.com/sf/discussion/do/listPosts/projects.bazaar/discussion.bazaar.topc28418
Forum Topic - Excessive interrupt scheduling delay: (7 Items)
   
Excessive interrupt scheduling delay  
I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX.  
I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits 
InterruptWait.

First, some particulars on the operating environment:

Board: Advantech PCM-3343 PC-104 Module
Vortex86DX CPU
1 GHz clock
256 MB RAM
2 Ethernet ports
PC Video/KB/Mouse
4 USB ports
4 Serial ports
Compact Flash card as the hard disk

The OS is QNX 6.5.0.  It has the minimum image needed to get the hardware operational.  I'm debugging with the Momentics
 IDE using QCONN as the communications medium.  However, I get similar results when I copy the non-debug version of the 
program and run it from the command line.

I have distilled down the program to the simple test program attached to this post.  I am using the board's general 
purpose I/O to run the interrupt and signal the code start and stop points.  I made all measurements using a digital 
storage scope.

In the test program, the port stIoLoMap is used to signal when the code reaches a certain point.  Port stIoHiMap is used
 to generate the interrupt.  I have this output hard-wired to the IRQ5 pin on the PC-104 connector.  For each port, I'm 
using all 8 points to signal so I don't have to search for the exact pin to monitor.

I have two threads running in the test program that simulate the hardware environment the board is installed in.  One 
thread is setting the interrupt output every 1 mS, simulating an external hardware device.  The other thread sets up the
 interrupt handler and then waits for the interrupt to occur.

When the generating task sets the IRQ line, the interrupt routine is executed.  It masks the interrupt, sets the signal 
line high to mark the beginning of the interrupt, and then exits.  When the handler thread exits InterruptWait, it 
clears the interrupt line, resets the signal line and then unkasks the interrupt.

I see a pretty consistant 8uS interrupt latency with this setup.  And most of the time I see a varying scheduling 
latency of up to 35uS.  But sometimes the handler thread doesn't respond before the next 1mS interrupt.  Buy slowing the
 interrupts to 10mS, I've seen latencies as long as 3mS.  I've tried round robin and FIFO scheduling, and various fixed 
and generated priorities, but these do not seem to have an effect.

Am I doing something wrong here that would cause this?
Attachment: Text naked_int_test.c 3.12 KB
RE: Excessive interrupt scheduling delay  
Use the system profiler to understand what is going on. 

First usleep() as the documention says may take longer, read up on it and make sure you read the tick tock section.

You may also be influence by the dreaded SMI. Which is an interrupt the OS doesn't see and is used by the BIOS. Make 
sure you disable thing like USB emulation.

> -----Message d'origine-----
> De : Robert Murrell [mailto:community-noreply@qnx.com]
> Envoyé : 20 octobre 2010 17:01
> À : ostech-core_os
> Objet : Excessive interrupt scheduling delay
> 
> I'm working on a program that is being ported from an obsolete single-
> process multi-threaded operating system to QNX.  I'm seeing occasional long
> delays (up to 3 milliseconds) from when an interrupt fires to when the
> handling thread exits InterruptWait.
> 
> First, some particulars on the operating environment:
> 
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
> 
> The OS is QNX 6.5.0.  It has the minimum image needed to get the hardware
> operational.  I'm debugging with the Momentics IDE using QCONN as the
> communications medium.  However, I get similar results when I copy the
> non-debug version of the program and run it from the command line.
> 
> I have distilled down the program to the simple test program attached to this
> post.  I am using the board's general purpose I/O to run the interrupt and
> signal the code start and stop points.  I made all measurements using a digital
> storage scope.
> 
> In the test program, the port stIoLoMap is used to signal when the code
> reaches a certain point.  Port stIoHiMap is used to generate the interrupt.  I
> have this output hard-wired to the IRQ5 pin on the PC-104 connector.  For
> each port, I'm using all 8 points to signal so I don't have to search for the
> exact pin to monitor.
> 
> I have two threads running in the test program that simulate the hardware
> environment the board is installed in.  One thread is setting the interrupt
> output every 1 mS, simulating an external hardware device.  The other
> thread sets up the interrupt handler and then waits for the interrupt to
> occur.
> 
> When the generating task sets the IRQ line, the interrupt routine is
> executed.  It masks the interrupt, sets the signal line high to mark the
> beginning of the interrupt, and then exits.  When the handler thread exits
> InterruptWait, it clears the interrupt line, resets the signal line and then
> unkasks the interrupt.
> 
> I see a pretty consistant 8uS interrupt latency with this setup.  And most of
> the time I see a varying scheduling latency of up to 35uS.  But sometimes the
> handler thread doesn't respond before the next 1mS interrupt.  Buy slowing
> the interrupts to 10mS, I've seen latencies as long as 3mS.  I've tried round
> robin and FIFO scheduling, and various fixed and generated priorities, but
> these do not seem to have an effect.
> 
> Am I doing something wrong here that would cause this?
> 
> 
> 
> 
> _______________________________________________
> 
> OSTech
> http://community.qnx.com/sf/go/post71586
RE: Excessive interrupt scheduling delay  
In one project we saw similar (and longer) delays that were caused by
the CPU's thermal management functionality.

If the issue is caused by SMI or clock gating due to overheat then there
will be gaps in the system profiler output: you will not have even timer
interrupts during the period.
--
  Mate

-----Original Message-----
From: Mario Charest [mailto:community-noreply@qnx.com] 
Sent: October 21, 2010 6:23 AM
To: ostech-core_os
Subject: RE: Excessive interrupt scheduling delay

Use the system profiler to understand what is going on. 

First usleep() as the documention says may take longer, read up on it
and make sure you read the tick tock section.

You may also be influence by the dreaded SMI. Which is an interrupt the
OS doesn't see and is used by the BIOS. Make sure you disable thing like
USB emulation.
Re: Excessive interrupt scheduling delay  
Hi Robert,

what version of the PCI-bios are you using?  Are you using an image with 
the APIC support ?
Is Advantech offering a BSP for this board ?

Please have in mind that the PC-3343 supports only the ISA bus and the 
_ISA interrupts_.

ISA interrupts are not level triggered ... the rising edge of the 
interrupt signal creates the interrupt!
However ... you have not to mask the interrupt and to reset the 
interrupt line at the PIC(?) if you are using InterruptWait.
You have just to reset the interrupt signal at hardware level.

Regards

--Armin

http://www.steinhoff-automation.com





Robert Murrell wrote:
> I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX. 
 I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits
 InterruptWait.
>
> First, some particulars on the operating environment:
>
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
>
> The OS is QNX 6.5.0.  It has the minimum image needed to get the hardware operational.  I'm debugging with the 
Momentics IDE using QCONN as the communications medium.  However, I get similar results when I copy the non-debug 
version of the program and run it from the command line.
>
> I have distilled down the program to the simple test program attached to this post.  I am using the board's general 
purpose I/O to run the interrupt and signal the code start and stop points.  I made all measurements using a digital 
storage scope.
>
> In the test program, the port stIoLoMap is used to signal when the code reaches a certain point.  Port stIoHiMap is 
used to generate the interrupt.  I have this output hard-wired to the IRQ5 pin on the PC-104 connector.  For each port, 
I'm using all 8 points to signal so I don't have to search for the exact pin to monitor.
>
> I have two threads running in the test program that simulate the hardware environment the board is installed in.  One 
thread is setting the interrupt output every 1 mS, simulating an external hardware device.  The other thread sets up the
 interrupt handler and then waits for the interrupt to occur.
>
> When the generating task sets the IRQ line, the interrupt routine is executed.  It masks the interrupt, sets the 
signal line high to mark the beginning of the interrupt, and then exits.  When the handler thread exits InterruptWait, 
it clears the interrupt line, resets the signal line and then unkasks the interrupt.
>
> I see a pretty consistant 8uS interrupt latency with this setup.  And most of the time I see a varying scheduling 
latency of up to 35uS.  But sometimes the handler thread doesn't respond before the next 1mS interrupt.  Buy slowing the
 interrupts to 10mS, I've seen latencies as long as 3mS.  I've tried round robin and FIFO scheduling, and various fixed 
and generated priorities, but these do not seem to have an effect.
>
> Am I doing something wrong here that would cause this?
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post71586
Re: Excessive interrupt scheduling delay  
Well, I had been staring at the problem all afternoon but it didn't click until I got home.  I had looked at this output
 of pidin and it didn't look right but I didn't connect it with what was going on:

  172047   1 ./naked_int_test   255r NANOSLEEP                               
  172047   2 ./naked_int_test    10r INTR                                    
  172047   3 ./naked_int_test   254r NANOSLEEP                               

The main thread, which is spending its time sleeping is at priority 255.  The high priority interrupt handler is at 10! 
 A quick check of the code shows that the return status of pthread_create is being used as the thread handle for 
SetThreadPriority.  I don't know how thread three got the priority it has.  Thanks for those that replied.
Re: Excessive interrupt scheduling delay  
  pthread_create doesn't return the newly created thread id, it returns 
EOK or an errno.

you should use

pthread_create(&newtid, ....);
return newtid;

BTW - why not set the thread priority in _beginthread (you can set it in 
the pthread_attr)

On 10-10-20 5:00 PM, Robert Murrell wrote:
> I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX. 
 I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits
 InterruptWait.
>
> First, some particulars on the operating environment:
>
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
>
> The OS is QNX 6.5.0.  It has the minimum image needed to get the hardware operational.  I'm debugging with the 
Momentics IDE using QCONN as the communications medium.  However, I get similar results when I copy the non-debug 
version of the program and run it from the command line.
>
> I have distilled down the program to the simple test program attached to this post.  I am using the board's general 
purpose I/O to run the interrupt and signal the code start and stop points.  I made all measurements using a digital 
storage scope.
>
> In the test program, the port stIoLoMap is used to signal when the code reaches a certain point.  Port stIoHiMap is 
used to generate the interrupt.  I have this output hard-wired to the IRQ5 pin on the PC-104 connector.  For each port, 
I'm using all 8 points to signal so I don't have to search for the exact pin to monitor.
>
> I have two threads running in the test program that simulate the hardware environment the board is installed in.  One 
thread is setting the interrupt output every 1 mS, simulating an external hardware device.  The other thread sets up the
 interrupt handler and then waits for the interrupt to occur.
>
> When the generating task sets the IRQ line, the interrupt routine is executed.  It masks the interrupt, sets the 
signal line high to mark the beginning of the interrupt, and then exits.  When the handler thread exits InterruptWait, 
it clears the interrupt line, resets the signal line and then unkasks the interrupt.
>
> I see a pretty consistant 8uS interrupt latency with this setup.  And most of the time I see a varying scheduling 
latency of up to 35uS.  But sometimes the handler thread doesn't respond before the next 1mS interrupt.  Buy slowing the
 interrupts to 10mS, I've seen latencies as long as 3mS.  I've tried round robin and FIFO scheduling, and various fixed 
and generated priorities, but these do not seem to have an effect.
>
> Am I doing something wrong here that would cause this?
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post71586

-- 
cburgess@qnx.com
Re: Excessive interrupt scheduling delay  
> BTW - why not set the thread priority in _beginthread (you can set it in 
> the pthread_attr)
> 

This code must also compile and run on a custom DSP board with its own development environment.  In this example, 
_beginthread and SetThreadPriority are QNX emulations of the native DSP library functions.