Project Home
Project Home
Documents
Documents
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Interrupt latency: (9 Items)
   
Interrupt latency  
Dear QNX community,

We have developed a realtime QNX application that uses a FPGA PCI device to perform some time-critical tasks. The device
 raises an interrupt on line 0x9 every 10 usec (100KHz), and the application has to respond inside this period.

This works, most of the time. However, once every minute or so, we miss *exactly* 270 usec worth of interrupts. Kernel 
tracing says nothing happens in this period: The interrupt handler is entered exactly on time, and finishes 270 usec 
later instead of the usual <1 usec. During this period nothing seems to happen, no interrupts, no process running. 
Afterwards, normal service resumes.

I can send the .kev file if needed, but it is rather large.

We have disabled almost every hardware device (except for the network card and serial ports which we require), and every
 timing- or interrupt-related BIOS option, and are running out of ideas.

Any hint, or ideas where to look next, would be greatly appreciated!

Kind regards,
-  Theo van Klaveren
RE: Interrupt latency  

> -----Original Message-----
> From: Theo van Klaveren [mailto:community-noreply@qnx.com]
> Sent: March-02-09 10:20 AM
> To: ostech-core_os
> Subject: Interrupt latency
> 
> Dear QNX community,
> 
> We have developed a realtime QNX application that uses a FPGA PCI
> device to perform some time-critical tasks. The device raises an
> interrupt on line 0x9 every 10 usec (100KHz), and the application has
> to respond inside this period.
> 
> This works, most of the time. However, once every minute or so, we miss
> *exactly* 270 usec worth of interrupts. Kernel tracing says nothing
> happens in this period: The interrupt handler is entered exactly on
> time, and finishes 270 usec later instead of the usual <1 usec. During
> this period nothing seems to happen, no interrupts, no process running.
> Afterwards, normal service resumes.
> 

If this is x86, it well could be the infamous SMI.

http://en.wikipedia.org/wiki/System_Management_Mode

> I can send the .kev file if needed, but it is rather large.
> 
> We have disabled almost every hardware device (except for the network
> card and serial ports which we require), and every timing- or
> interrupt-related BIOS option, and are running out of ideas.
> 
> Any hint, or ideas where to look next, would be greatly appreciated!
> 
> Kind regards,
> -  Theo van Klaveren
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post23201
> 
Re: RE: Interrupt latency  
> 
> 
> > -----Original Message-----
> > From: Theo van Klaveren [mailto:community-noreply@qnx.com]
> > Sent: March-02-09 10:20 AM
> > To: ostech-core_os
> > Subject: Interrupt latency
> > 
> > Dear QNX community,
> > 
> > We have developed a realtime QNX application that uses a FPGA PCI
> > device to perform some time-critical tasks. The device raises an
> > interrupt on line 0x9 every 10 usec (100KHz), and the application has
> > to respond inside this period.
> > 
> > This works, most of the time. However, once every minute or so, we miss
> > *exactly* 270 usec worth of interrupts. Kernel tracing says nothing
> > happens in this period: The interrupt handler is entered exactly on
> > time, and finishes 270 usec later instead of the usual <1 usec. During
> > this period nothing 
> seems to happen, no interrupts, no process running.
> > Afterwards, normal service resumes.
> > 
> 
> If this is x86, it well could be the infamous SMI.

Yes, this is x86, and yes, this could very well be it! Any ideas on how to work around?

I forgot to mention in the original post, this behaviour stops after a certain period of time.

 - Theo
RE: RE: Interrupt latency  
> > If this is x86, it well could be the infamous SMI.
> 
> Yes, this is x86, and yes, this could very well be it! Any ideas on how
> to work around?

Only work around is to contact the maker and ask if the motherboard/bios supports disabling SMI.  Sometimes disabling 
certain feature like USB emulation, will help reduce the duration or number of invocation of the SMI.

> 
> I forgot to mention in the original post, this behaviour stops after a
> certain period of time.
> 
>  - Theo
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post23210
> 
Re: RE: RE: Interrupt latency  
> > > If this is x86, it well could be the infamous SMI.
> > 
> > Yes, this is x86, and yes, this could very well be it! Any ideas on how
> > to work around?
> 
> Only work around is to contact the maker and ask if the motherboard/bios 
> supports disabling SMI.  Sometimes disabling certain feature like USB 
> emulation, will help reduce the duration or number of invocation of the SMI.

Mario,

Thanks very, very much for your help. The problem is solved. I disabled SMI using a modified version of code posted on 
the Foundry27 forum, here:

http://community.qnx.com/sf/discussion/do/listPosts/projects.bsp/discussion.bsp.topc5720

The (modified) code works on Intel ICH4, and I'll post it here for posterity:

#include <stdlib.h>
#include <stdio.h>
#include <hw/pci.h>

#define BIT(n) (1 << (n))

// REGISTER LOCATIONS
#define SMI_LOCK 	BIT(4)	
#define SMI_EN    	BIT(3)

int
main(int argc, char *argv[])
{
	int pci;
	int rv;
	int bus;
	int func;
	unsigned reg;
	unsigned readback;

	// Attach pci device
	pci = pci_attach(0);
	if (pci < 0) {
		perror("pci_attach");
		return -1;
	}
	rv = pci_find_device(0x3580, 0x8086, 0, &bus, &func);
	if (rv == PCI_DEVICE_NOT_FOUND || rv < 0) {
		printf("device not found\n");
		return -1;
	}
	//_smi_device = p.GetDeviceObject(0x8086 , 0x2918);  
	// vid=0x8086 , did=0x2918
	pci_read_config32(bus, func, 0x60, 1, ®);
	printf("%8X\n", reg);
	if (reg & SMI_LOCK)
	{
		printf("SMI LOCK is set! Nothing we can do here :-(\n");
		return -1;
	}
	printf("SMI_LOCK not set.\n");
	if (!(reg & SMI_EN)) {
		printf("SMI_EN already unset!\n");
		return 0;
	}
	reg &= ~(SMI_EN);
	pci_write_config32(bus, func, 0x60, 1, ®);
	pci_read_config32(bus, func, 0x60, 1, &readback);
	if (!(readback & SMI_EN)) {
		printf("Success!\n");
		return 0;
	} else {
		printf("Fail!\n");
		return 0;
	}
	return 0;
}

Thanks again,
 - Theo
RE: RE: RE: Interrupt latency  

> -----Original Message-----
> From: Theo van Klaveren [mailto:community-noreply@qnx.com]
> Sent: March-03-09 9:03 AM
> To: ostech-core_os
> Subject: Re: RE: RE: Interrupt latency
> 
> > > > If this is x86, it well could be the infamous SMI.
> > >
> > > Yes, this is x86, and yes, this could very well be it! Any ideas on
> how
> > > to work around?
> >
> > Only work around is to contact the maker and ask if the
> motherboard/bios
> > supports disabling SMI.  Sometimes disabling certain feature like USB
> > emulation, will help reduce the duration or number of invocation of
> the SMI.
> 
> Mario,
> 
> Thanks very, very much for your help. The problem is solved. 

Glad I could help.

> I disabled SMI using a modified version of code posted on the Foundry27 forum,
> here:

You have to be careful when disabling this feature. I've seen BIOSes proving watchdog feature, which can't work (endless
 reboot) with SMI disable. Same thing for USB keyboard emulation.

> 
> http://community.qnx.com/sf/discussion/do/listPosts/projects.bsp/discus
> sion.bsp.topc5720
> 
> The (modified) code works on Intel ICH4, and I'll post it here for
> posterity:
> 
> #include <stdlib.h>
> #include <stdio.h>
> #include <hw/pci.h>
> 
> #define BIT(n) (1 << (n))
> 
> // REGISTER LOCATIONS
> #define SMI_LOCK 	BIT(4)
> #define SMI_EN    	BIT(3)
> 
> int
> main(int argc, char *argv[])
> {
> 	int pci;
> 	int rv;
> 	int bus;
> 	int func;
> 	unsigned reg;
> 	unsigned readback;
> 
> 	// Attach pci device
> 	pci = pci_attach(0);
> 	if (pci < 0) {
> 		perror("pci_attach");
> 		return -1;
> 	}
> 	rv = pci_find_device(0x3580, 0x8086, 0, &bus, &func);
> 	if (rv == PCI_DEVICE_NOT_FOUND || rv < 0) {
> 		printf("device not found\n");
> 		return -1;
> 	}
> 	//_smi_device = p.GetDeviceObject(0x8086 , 0x2918);
> 	// vid=0x8086 , did=0x2918
> 	pci_read_config32(bus, func, 0x60, 1, ®);
> 	printf("%8X\n", reg);
> 	if (reg & SMI_LOCK)
> 	{
> 		printf("SMI LOCK is set! Nothing we can do here :-(\n");
> 		return -1;
> 	}
> 	printf("SMI_LOCK not set.\n");
> 	if (!(reg & SMI_EN)) {
> 		printf("SMI_EN already unset!\n");
> 		return 0;
> 	}
> 	reg &= ~(SMI_EN);
> 	pci_write_config32(bus, func, 0x60, 1, ®);
> 	pci_read_config32(bus, func, 0x60, 1, &readback);
> 	if (!(readback & SMI_EN)) {
> 		printf("Success!\n");
> 		return 0;
> 	} else {
> 		printf("Fail!\n");
> 		return 0;
> 	}
> 	return 0;
> }
> 
> Thanks again,
>  - Theo
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post23306
> 
Re: RE: RE: RE: Interrupt latency  
> 
> 
> > -----Original Message-----
> > From: Theo van Klaveren [mailto:community-noreply@qnx.com]
> > Sent: March-03-09 9:03 AM
> > To: ostech-core_os
> > Subject: Re: RE: RE: Interrupt latency
> > 
> > > > > If this is x86, it well could be the infamous SMI.
> > > >
> > > > Yes, this is x86, and yes, this could very well be it! Any ideas on
> > how
> > > > to work around?
> > >
> > > Only work around is to contact the maker and ask if the
> > motherboard/bios
> > > supports disabling SMI.  Sometimes disabling certain feature like USB
> > > emulation, will help reduce the duration or number of invocation of
> > the SMI.
> > 
> > Mario,
> > 
> > Thanks very, very much for your help. The problem is solved. 
> 
> Glad I could help.
> 
> > I disabled SMI using a modified version of code posted on the Foundry27 
> forum,
> > here:
> 
> You have to be careful when disabling this feature. I've seen BIOSes proving 
> watchdog feature, which can't work (endless reboot) with SMI disable. Same 
> thing for USB keyboard emulation.

Right. It works on the (older) testing board I have here; in the actual machines (slightly newer boards) this causes a 
hard crash when the interrupts start coming.

Any ideas on how to circumvent? There seem to be no BIOS options for watchdogs or keyboard emulation.
Re: RE: RE: RE: Interrupt latency  
> > 
> > 
> > > -----Original Message-----
> > > From: Theo van Klaveren [mailto:community-noreply@qnx.com]
> > > Sent: March-03-09 9:03 AM
> > > To: ostech-core_os
> > > Subject: Re: RE: RE: Interrupt latency
> > > 
> > > > > > If this is x86, it well could be the infamous SMI.
> > > > >
> > > > > Yes, this is x86, and yes, this could very well be it! Any ideas on
> > > how
> > > > > to work around?
> > > >
> > > > Only work around is to contact the maker and ask if the
> > > motherboard/bios
> > > > supports disabling SMI.  Sometimes disabling certain feature like USB
> > > > emulation, will help reduce the duration or number of invocation of
> > > the SMI.
> > > 
> > > Mario,
> > > 
> > > Thanks very, very much for your help. The problem is solved. 
> > 
> > Glad I could help.
> > 
> > > I disabled SMI using a modified version of code posted on the Foundry27 
> > forum,
> > > here:
> > 
> > You have to be careful when disabling this feature. I've seen BIOSes proving
>  
> > watchdog feature, which can't work (endless reboot) with SMI disable. Same 
> > thing for USB keyboard emulation.
> 
> Right. It works on the (older) testing board I have here; in the actual 
> machines (slightly newer boards) this causes a hard crash when the interrupts 
> start coming.
> 
> Any ideas on how to circumvent? There seem to be no BIOS options for watchdogs
>  or keyboard emulation.


It turns out, the code was wrong. Here is the correct source, as now in use, which actually works on both platforms:

#include <stdlib.h>
#include <stdio.h>
#include <hw/inout.h>
#include <hw/pci.h>
#include <sys/mman.h>
#include <sys/neutrino.h>

/* PCI device information */
#define PLATFORM	"Intel ICH4"
#define PCI_VENDOR	0x8086	/* Vendor: Intel Corporation */
//#define PCI_DEVICE	0x3580	/* Device: Intel 825GM Host/PCI bridge */
#define PCI_DEVICE	0x24C0	/* Device: Intel 82801DB/DBL (ICH4/ICH4-L) LPC Interface Bridge */

/* Register locations */
#define PM_BASE		0x40
#define GEN_PMCON_1	0xA0
#define   SMI_LOCK	(1<<4)

#define ACPI_SMI_CNTL	0x30
#define   GBL_SMI_EN    (1<<0)

int
main(int argc, char *argv[])
{
	int pci;
	int rv;
	int bus;
	int func;
	unsigned int reg;
	unsigned int pmbase_addr;
	uintptr_t pmbase;

	printf("SMI disable hack for " PLATFORM "\n");

	/* Attach pci. */
	pci = pci_attach(0);
	if (pci < 0) {
		perror("pci_attach");
		return -1;
	}

	/* Find the PCI host bridge device. */
	rv = pci_find_device(PCI_DEVICE, PCI_VENDOR, 0, &bus, &func);
	if (rv == PCI_DEVICE_NOT_FOUND || rv < 0) {
		printf("Device not found.\n");
		return -1;
	}

	pci_read_config32(bus, func, PM_BASE, 1, ®);
	printf("PMBASE: %08X\n", reg);
	pmbase_addr = reg & 0xFF80;

	pci_read_config32(bus, func, GEN_PMCON_1, 1, ®);
	printf("GEN_PMCON_1: %08X\n", reg);
	if (reg & SMI_LOCK) {
		printf("SMI_LOCK is set, nothing we can do!\n");
		return -1;
	}

	if (ThreadCtl(_NTO_TCTL_IO, 0)) {
		perror("_NTO_TCTL_IO");
		return -1;
	}
	pmbase = mmap_device_io(128, (uint64_t)pmbase_addr);
	if (pmbase == MAP_DEVICE_FAILED) {
		printf("Failed to map I/O space.\n");
		return -1;
	}
	reg = in32(pmbase + ACPI_SMI_CNTL);
	printf("ACPI_SMI_CNTL before: %08X\n", reg);
	if (!(reg & GBL_SMI_EN)) {
		printf("GBL_SMI_EN already cleared, success.\n");
		return 0;
	}
	reg &= ~(GBL_SMI_EN);
	out32(pmbase + ACPI_SMI_CNTL, reg);
	reg = in32(pmbase + ACPI_SMI_CNTL);
	printf("ACPI_SMI_CNTL after: %08X\n", reg);
	if (reg & GBL_SMI_EN) {
		printf("GBL_SMI_EN not cleared, failed!\n");
		return...
RE: RE: Interrupt latency  
> > If this is x86, it well could be the infamous SMI.
> 
> Yes, this is x86, and yes, this could very well be it! Any ideas on how
> to work around?

You can probably access the SMI pin on the  CPU and check if it's activated.

> 
> I forgot to mention in the original post, this behaviour stops after a
> certain period of time.
> 
>  - Theo
> 
> 
> _______________________________________________
> OSTech
> http://community.qnx.com/sf/go/post23210
>