Robert Murrell
|
Excessive interrupt scheduling delay
|
Robert Murrell
10/20/2010 5:00 PM
post71586
|
Excessive interrupt scheduling delay
I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX.
I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits
InterruptWait.
First, some particulars on the operating environment:
Board: Advantech PCM-3343 PC-104 Module
Vortex86DX CPU
1 GHz clock
256 MB RAM
2 Ethernet ports
PC Video/KB/Mouse
4 USB ports
4 Serial ports
Compact Flash card as the hard disk
The OS is QNX 6.5.0. It has the minimum image needed to get the hardware operational. I'm debugging with the Momentics
IDE using QCONN as the communications medium. However, I get similar results when I copy the non-debug version of the
program and run it from the command line.
I have distilled down the program to the simple test program attached to this post. I am using the board's general
purpose I/O to run the interrupt and signal the code start and stop points. I made all measurements using a digital
storage scope.
In the test program, the port stIoLoMap is used to signal when the code reaches a certain point. Port stIoHiMap is used
to generate the interrupt. I have this output hard-wired to the IRQ5 pin on the PC-104 connector. For each port, I'm
using all 8 points to signal so I don't have to search for the exact pin to monitor.
I have two threads running in the test program that simulate the hardware environment the board is installed in. One
thread is setting the interrupt output every 1 mS, simulating an external hardware device. The other thread sets up the
interrupt handler and then waits for the interrupt to occur.
When the generating task sets the IRQ line, the interrupt routine is executed. It masks the interrupt, sets the signal
line high to mark the beginning of the interrupt, and then exits. When the handler thread exits InterruptWait, it
clears the interrupt line, resets the signal line and then unkasks the interrupt.
I see a pretty consistant 8uS interrupt latency with this setup. And most of the time I see a varying scheduling
latency of up to 35uS. But sometimes the handler thread doesn't respond before the next 1mS interrupt. Buy slowing the
interrupts to 10mS, I've seen latencies as long as 3mS. I've tried round robin and FIFO scheduling, and various fixed
and generated priorities, but these do not seem to have an effect.
Am I doing something wrong here that would cause this?
|
|
|
Mario Charest
|
RE: Excessive interrupt scheduling delay
|
Mario Charest
10/20/2010 5:22 PM
post71593
|
RE: Excessive interrupt scheduling delay
Use the system profiler to understand what is going on.
First usleep() as the documention says may take longer, read up on it and make sure you read the tick tock section.
You may also be influence by the dreaded SMI. Which is an interrupt the OS doesn't see and is used by the BIOS. Make
sure you disable thing like USB emulation.
> -----Message d'origine-----
> De : Robert Murrell [mailto:community-noreply@qnx.com]
> Envoyé : 20 octobre 2010 17:01
> À : ostech-core_os
> Objet : Excessive interrupt scheduling delay
>
> I'm working on a program that is being ported from an obsolete single-
> process multi-threaded operating system to QNX. I'm seeing occasional long
> delays (up to 3 milliseconds) from when an interrupt fires to when the
> handling thread exits InterruptWait.
>
> First, some particulars on the operating environment:
>
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
>
> The OS is QNX 6.5.0. It has the minimum image needed to get the hardware
> operational. I'm debugging with the Momentics IDE using QCONN as the
> communications medium. However, I get similar results when I copy the
> non-debug version of the program and run it from the command line.
>
> I have distilled down the program to the simple test program attached to this
> post. I am using the board's general purpose I/O to run the interrupt and
> signal the code start and stop points. I made all measurements using a digital
> storage scope.
>
> In the test program, the port stIoLoMap is used to signal when the code
> reaches a certain point. Port stIoHiMap is used to generate the interrupt. I
> have this output hard-wired to the IRQ5 pin on the PC-104 connector. For
> each port, I'm using all 8 points to signal so I don't have to search for the
> exact pin to monitor.
>
> I have two threads running in the test program that simulate the hardware
> environment the board is installed in. One thread is setting the interrupt
> output every 1 mS, simulating an external hardware device. The other
> thread sets up the interrupt handler and then waits for the interrupt to
> occur.
>
> When the generating task sets the IRQ line, the interrupt routine is
> executed. It masks the interrupt, sets the signal line high to mark the
> beginning of the interrupt, and then exits. When the handler thread exits
> InterruptWait, it clears the interrupt line, resets the signal line and then
> unkasks the interrupt.
>
> I see a pretty consistant 8uS interrupt latency with this setup. And most of
> the time I see a varying scheduling latency of up to 35uS. But sometimes the
> handler thread doesn't respond before the next 1mS interrupt. Buy slowing
> the interrupts to 10mS, I've seen latencies as long as 3mS. I've tried round
> robin and FIFO scheduling, and various fixed and generated priorities, but
> these do not seem to have an effect.
>
> Am I doing something wrong here that would cause this?
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post71586
|
|
|
Mate Szarvas
|
RE: Excessive interrupt scheduling delay
|
Mate Szarvas
10/20/2010 9:08 PM
post71635
|
RE: Excessive interrupt scheduling delay
In one project we saw similar (and longer) delays that were caused by
the CPU's thermal management functionality.
If the issue is caused by SMI or clock gating due to overheat then there
will be gaps in the system profiler output: you will not have even timer
interrupts during the period.
--
Mate
-----Original Message-----
From: Mario Charest [mailto:community-noreply@qnx.com]
Sent: October 21, 2010 6:23 AM
To: ostech-core_os
Subject: RE: Excessive interrupt scheduling delay
Use the system profiler to understand what is going on.
First usleep() as the documention says may take longer, read up on it
and make sure you read the tick tock section.
You may also be influence by the dreaded SMI. Which is an interrupt the
OS doesn't see and is used by the BIOS. Make sure you disable thing like
USB emulation.
|
|
|
Armin Steinhoff
|
Re: Excessive interrupt scheduling delay
|
Armin Steinhoff
10/21/2010 3:45 AM
post71669
|
Re: Excessive interrupt scheduling delay
Hi Robert,
what version of the PCI-bios are you using? Are you using an image with
the APIC support ?
Is Advantech offering a BSP for this board ?
Please have in mind that the PC-3343 supports only the ISA bus and the
_ISA interrupts_.
ISA interrupts are not level triggered ... the rising edge of the
interrupt signal creates the interrupt!
However ... you have not to mask the interrupt and to reset the
interrupt line at the PIC(?) if you are using InterruptWait.
You have just to reset the interrupt signal at hardware level.
Regards
--Armin
http://www.steinhoff-automation.com
Robert Murrell wrote:
> I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX.
I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits
InterruptWait.
>
> First, some particulars on the operating environment:
>
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
>
> The OS is QNX 6.5.0. It has the minimum image needed to get the hardware operational. I'm debugging with the
Momentics IDE using QCONN as the communications medium. However, I get similar results when I copy the non-debug
version of the program and run it from the command line.
>
> I have distilled down the program to the simple test program attached to this post. I am using the board's general
purpose I/O to run the interrupt and signal the code start and stop points. I made all measurements using a digital
storage scope.
>
> In the test program, the port stIoLoMap is used to signal when the code reaches a certain point. Port stIoHiMap is
used to generate the interrupt. I have this output hard-wired to the IRQ5 pin on the PC-104 connector. For each port,
I'm using all 8 points to signal so I don't have to search for the exact pin to monitor.
>
> I have two threads running in the test program that simulate the hardware environment the board is installed in. One
thread is setting the interrupt output every 1 mS, simulating an external hardware device. The other thread sets up the
interrupt handler and then waits for the interrupt to occur.
>
> When the generating task sets the IRQ line, the interrupt routine is executed. It masks the interrupt, sets the
signal line high to mark the beginning of the interrupt, and then exits. When the handler thread exits InterruptWait,
it clears the interrupt line, resets the signal line and then unkasks the interrupt.
>
> I see a pretty consistant 8uS interrupt latency with this setup. And most of the time I see a varying scheduling
latency of up to 35uS. But sometimes the handler thread doesn't respond before the next 1mS interrupt. Buy slowing the
interrupts to 10mS, I've seen latencies as long as 3mS. I've tried round robin and FIFO scheduling, and various fixed
and generated priorities, but these do not seem to have an effect.
>
> Am I doing something wrong here that would cause this?
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post71586
|
|
|
Robert Murrell
|
Re: Excessive interrupt scheduling delay
|
Robert Murrell
10/21/2010 8:29 AM
post71688
|
Re: Excessive interrupt scheduling delay
Well, I had been staring at the problem all afternoon but it didn't click until I got home. I had looked at this output
of pidin and it didn't look right but I didn't connect it with what was going on:
172047 1 ./naked_int_test 255r NANOSLEEP
172047 2 ./naked_int_test 10r INTR
172047 3 ./naked_int_test 254r NANOSLEEP
The main thread, which is spending its time sleeping is at priority 255. The high priority interrupt handler is at 10!
A quick check of the code shows that the return status of pthread_create is being used as the thread handle for
SetThreadPriority. I don't know how thread three got the priority it has. Thanks for those that replied.
|
|
|
Colin Burgess(deleted)
|
Re: Excessive interrupt scheduling delay
|
Colin Burgess(deleted)
10/21/2010 9:12 AM
post71703
|
Re: Excessive interrupt scheduling delay
pthread_create doesn't return the newly created thread id, it returns
EOK or an errno.
you should use
pthread_create(&newtid, ....);
return newtid;
BTW - why not set the thread priority in _beginthread (you can set it in
the pthread_attr)
On 10-10-20 5:00 PM, Robert Murrell wrote:
> I'm working on a program that is being ported from an obsolete single-process multi-threaded operating system to QNX.
I'm seeing occasional long delays (up to 3 milliseconds) from when an interrupt fires to when the handling thread exits
InterruptWait.
>
> First, some particulars on the operating environment:
>
> Board: Advantech PCM-3343 PC-104 Module
> Vortex86DX CPU
> 1 GHz clock
> 256 MB RAM
> 2 Ethernet ports
> PC Video/KB/Mouse
> 4 USB ports
> 4 Serial ports
> Compact Flash card as the hard disk
>
> The OS is QNX 6.5.0. It has the minimum image needed to get the hardware operational. I'm debugging with the
Momentics IDE using QCONN as the communications medium. However, I get similar results when I copy the non-debug
version of the program and run it from the command line.
>
> I have distilled down the program to the simple test program attached to this post. I am using the board's general
purpose I/O to run the interrupt and signal the code start and stop points. I made all measurements using a digital
storage scope.
>
> In the test program, the port stIoLoMap is used to signal when the code reaches a certain point. Port stIoHiMap is
used to generate the interrupt. I have this output hard-wired to the IRQ5 pin on the PC-104 connector. For each port,
I'm using all 8 points to signal so I don't have to search for the exact pin to monitor.
>
> I have two threads running in the test program that simulate the hardware environment the board is installed in. One
thread is setting the interrupt output every 1 mS, simulating an external hardware device. The other thread sets up the
interrupt handler and then waits for the interrupt to occur.
>
> When the generating task sets the IRQ line, the interrupt routine is executed. It masks the interrupt, sets the
signal line high to mark the beginning of the interrupt, and then exits. When the handler thread exits InterruptWait,
it clears the interrupt line, resets the signal line and then unkasks the interrupt.
>
> I see a pretty consistant 8uS interrupt latency with this setup. And most of the time I see a varying scheduling
latency of up to 35uS. But sometimes the handler thread doesn't respond before the next 1mS interrupt. Buy slowing the
interrupts to 10mS, I've seen latencies as long as 3mS. I've tried round robin and FIFO scheduling, and various fixed
and generated priorities, but these do not seem to have an effect.
>
> Am I doing something wrong here that would cause this?
>
>
>
>
> _______________________________________________
>
> OSTech
> http://community.qnx.com/sf/go/post71586
--
cburgess@qnx.com
|
|
|
Robert Murrell
|
Re: Excessive interrupt scheduling delay
|
Robert Murrell
10/21/2010 11:03 AM
post71773
|
Re: Excessive interrupt scheduling delay
> BTW - why not set the thread priority in _beginthread (you can set it in
> the pthread_attr)
>
This code must also compile and run on a custom DSP board with its own development environment. In this example,
_beginthread and SetThreadPriority are QNX emulations of the native DSP library functions.
|
|
|
|