|
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/12/2009 11:49 AM
post29279
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
MAC addresses differ because they tried swapping to backup server (identical configuration) to find out if it is not
caused by defective network card.
> Here it is again for i82544
> ===========
> # nicinfo /dev/io-net/en1
> INTEL 82544 Gigabit (Copper) Ethernet Controller
>
> Physical Node ID ........................... 003048 B037CB
> Current Physical Node ID ................... 003048 B037CB
> ===========
> And here it is once again for e1000
> ===========
> # nicinfo /dev/io-net/en1
> INTEL PRO/1000 Gigabit (Copper) Ethernet Controller
>
> Physical Node ID ........................... 003048 B0CC4F
> Current Physical Node ID ................... 003048 B0CC4F
> ===========
|
|
|
|
|
|
|
RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/12/2009 2:32 PM
post29295
|
RE: RE: RE: RE: How to find out if that's a HW fault?
Looking at the PCI interrupts, interrupts 10 and 11 are shared between
the Network and USB controllers. Is the customer running any USB
devices? If not, can they slay io-usb and see if that helps anything.
The disk controller is also on interrupt 11. Is disk performance
affected at all?
Thanks, Hugh.
-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com]
Sent: Tuesday, May 12, 2009 8:10 AM
To: general-networking
Subject: Re: RE: RE: RE: How to find out if that's a HW fault?
Hi Hugh,
At last, they've sent me the 'pci -v' output. Here it is
=================================
PCI version = 3.00
Class = Bridge (Host/PCI)
Vendor ID = 8086h, Intel Corporation
Device ID = 2970h, Memory Controller Hub
PCI index = 0h
Class Codes = 060000h
Revision ID = 2h
Bus number = 0
Device number = 0
Function num = 0
Status Reg = 2090h
Command Reg = 106h
Header type = 0h Single-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 0h
Cache Line Size= 0h
Subsystem Vendor ID = 15d9h
Subsystem ID = b580h
Max Lat = 0ns
Min Gnt = 0ns
PCI Int Pin = NC
Interrupt line = 0
CPU Interrupt = 0h
Capabilities Pointer = e0h
Capability ID = 9h - Vendor Specific
Capabilities = 9109h - cb000368h
Class = Display (VGA)
Vendor ID = 8086h, Intel Corporation
Device ID = 2972h, Integrated Graphics Controller
PCI index = 0h
Class Codes = 030000h
Revision ID = 2h
Bus number = 0
Device number = 2
Function num = 0
Status Reg = 90h
Command Reg = 7h
Header type = 0h Single-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 0h
Cache Line Size= 0h
PCI Mem Address = d0000000h 32bit length 1048576 enabled
PCI Mem Address = c0000000h prefetchable 64bit length 268435456 enabled
PCI IO Address = 30c0h length 8 enabled
Subsystem Vendor ID = 15d9h
Subsystem ID = b580h
Max Lat = 0ns
Min Gnt = 0ns
PCI Int Pin = INT A
Interrupt line = 10
CPU Interrupt = ah
Capabilities Pointer = 90h
Capability ID = 5h - Message Signaled Interrupts
Capabilities = 0h - 0h
Capability ID = 1h - Power Management
Capabilities = 22h - 0h
Class = Bridge (PCI/PCI)
Vendor ID = 8086h, Intel Corporation
Device ID = 27d0h, 82801G (ICH7 Family) PCI Express Port 1
PCI index = 0h
Class Codes = 060400h
Revision ID = 1h
Bus number = 0
Device number = 28
Function num = 0
Status Reg = 10h
Command Reg = 4h
Header type = 1h Multi-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 0h
Cache Line Size= 8h un-cacheable
Primary Bus Number = 0h
Secondary Bus Number = 2h
Subordinate Bus Number = 2h
Secondary Latency Timer = 0h
I/O Base = f0h
I/O Limit = 0h
Secondary Status = 2000h
Memory Base = fff0h
Memory Limit = 0h
Prefetchable Memory Base = fff1h
Prefetchable Memory Limit= 1h
Prefetchable Base Upper 32 Bits = 0h
Prefetchable Limit Upper 32 Bits = 0h
I/O Base Upper 16 Bits = 0h
I/O Limit Upper 16 Bits = 0h
Bridge Control = 0h
PCI Int Pin = INT A
Interrupt line = 10
CPU Interrupt = ah
Capabilities Pointer = 40h
Capability ID = 10h - PCI Express
Capabilities = 141h - fc0h
Capability ID = 5h - Message Signaled Interrupts
Capabilities = 0h - 0h
Capability ID = dh - PCI Bridge Subsystem Vendor ID
Capabilities = 0h - b58015d9h
Capability ID = 1h - Power Management
Capabilities = c802h - 0h
Class = Bridge (PCI/PCI)
Vendor ID = 8086h, Intel Corporation
Device ID = 27e0h, 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5
PCI index = 0h
Class Codes ...
View Full Message
|
|
|
|
|
|
|
Re: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/12/2009 4:48 PM
post29315
|
Re: RE: RE: RE: RE: How to find out if that's a HW fault?
> Looking at the PCI interrupts, interrupts 10 and 11 are shared between
> the Network and USB controllers. Is the customer running any USB
> devices? If not, can they slay io-usb and see if that helps anything.
I've already told them to try disabling USBs in BIOS if that is possible. I'll also pass your suggestion regarding io-
usb to them.
> The disk controller is also on interrupt 11. Is disk performance
> affected at all?
I don't know. How can we find that out? We hardly ever use any disk I/O - just for logging (very rarely) and for
configuration storage.
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 8:41 AM
post29342
|
RE: RE: RE: RE: RE: How to find out if that's a HW fault?
Please can you get me the sloginfo output after the customer discovers
that the link is down. I have been testing here all night without any
problems. I have also been removing cables and re-inserting them while
communication is in progress, without any problems.
Thanks, Hugh.
-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com]
Sent: Tuesday, May 12, 2009 4:49 PM
To: general-networking
Subject: Re: RE: RE: RE: RE: How to find out if that's a HW fault?
> Looking at the PCI interrupts, interrupts 10 and 11 are shared between
> the Network and USB controllers. Is the customer running any USB
> devices? If not, can they slay io-usb and see if that helps anything.
I've already told them to try disabling USBs in BIOS if that is
possible. I'll also pass your suggestion regarding io-usb to them.
> The disk controller is also on interrupt 11. Is disk performance
> affected at all?
I don't know. How can we find that out? We hardly ever use any disk I/O
- just for logging (very rarely) and for configuration storage.
_______________________________________________
General
http://community.qnx.com/sf/go/post29315
|
|
|
|
|
|
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 8:48 AM
post29343
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
> Please can you get me the sloginfo output after the customer discovers that the link is down.
Do I need any specific parameters for system to get a good output from sloginfo? For example, we do not run instrumented
kernel - just a simple one.
> I have been testing here all night without any problems. I have also been removing cables and re-inserting them while
communication is in progress, without any problems.
EN0 never dies. It's only EN1 on motherboard with two interfaces.
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 8:57 AM
post29344
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
No, you don't need any specific parameters. Qnet logs messages to
sloginfo, so I want to see what it displays. I have been removing and
re-inserting cables in both en0 and en1 without any problems. I have one
link connected with Qnet and the other with TCP/IP and am running tests
on both links.
Hugh.
-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com]
Sent: Wednesday, May 13, 2009 8:48 AM
To: general-networking
Subject: Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
> Please can you get me the sloginfo output after the customer discovers
that the link is down.
Do I need any specific parameters for system to get a good output from
sloginfo? For example, we do not run instrumented kernel - just a simple
one.
> I have been testing here all night without any problems. I have also
been removing cables and re-inserting them while communication is in
progress, without any problems.
EN0 never dies. It's only EN1 on motherboard with two interfaces.
_______________________________________________
General
http://community.qnx.com/sf/go/post29343
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 9:00 AM
post29345
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-13-09 8:58 AM
> To: general-networking
> Subject: RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
>
> No, you don't need any specific parameters. Qnet logs messages to
> sloginfo, so I want to see what it displays.
That`s the default but could have been disable. I know I do because for something like a cable being disconnect after a
while sloginfo is filled with qnet data and other important stuff is getting flushed away. Maybe also the content of /
proc/qnetstat?
>I have been removing and
> re-inserting cables in both en0 and en1 without any problems. I have
> one
> link connected with Qnet and the other with TCP/IP and am running tests
> on both links.
>
> Hugh.
>
> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: Wednesday, May 13, 2009 8:48 AM
> To: general-networking
> Subject: Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
>
> > Please can you get me the sloginfo output after the customer
> discovers
> that the link is down.
>
> Do I need any specific parameters for system to get a good output from
> sloginfo? For example, we do not run instrumented kernel - just a
> simple
> one.
>
> > I have been testing here all night without any problems. I have also
> been removing cables and re-inserting them while communication is in
> progress, without any problems.
>
> EN0 never dies. It's only EN1 on motherboard with two interfaces.
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post29343
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post29344
>
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 9:04 AM
post29346
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
> after a while sloginfo is filled with qnet data
start qnet "no_slog=1" to turn off event logging
to sloginfo. It will still log events to it's
(smaller) internal ringbuffer, which you dump with:
# cat /proc/qnetstats
--
aboyd
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 9:23 AM
post29350
|
RE: RE: RE: RE: RE: How to find out if that's a HW fault?
I might take a moment to repeat the suggestion, of
reconfiguring the PCI bus to get rid of shared interrupts.
--
aboyd
|
|
|
|
|
|
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/13/2009 9:37 AM
post29352
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
>
> I might take a moment to repeat the suggestion, of
> reconfiguring the PCI bus to get rid of shared interrupts.
>
I do not have access to that hardware. I live in different country, on the other end of continent. And can't even check
if BIOS supports that reconfiguration. With integrated hardware it is not that you just pull a card out and insert it
into another slot to change interrupt. :(
Let's see if there will be any effect after USBs are disabled and/or io-usb killed.
|
|
|
|
|
|
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/21/2009 9:34 AM
post29811
|
Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
Hi, Hugh
Sorry, I don't have anything yet because our guy seems to have made all the changes I sent him at once instead of trying
three different tests. :) However it looks like the faults do not hapen any more. So, I suppose, disabling USB helped.
> Please can you get me the sloginfo output after the customer discovers
> that the link is down. I have been testing here all night without any
> problems. I have also been removing cables and re-inserting them while
> communication is in progress, without any problems.
>
> Thanks, Hugh.
>
> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: Tuesday, May 12, 2009 4:49 PM
> To: general-networking
> Subject: Re: RE: RE: RE: RE: How to find out if that's a HW fault?
>
> > Looking at the PCI interrupts, interrupts 10 and 11 are shared between
> > the Network and USB controllers. Is the customer running any USB
> > devices? If not, can they slay io-usb and see if that helps anything.
>
> I've already told them to try disabling USBs in BIOS if that is
> possible. I'll also pass your suggestion regarding io-usb to them.
>
> > The disk controller is also on interrupt 11. Is disk performance
> > affected at all?
>
> I don't know. How can we find that out? We hardly ever use any disk I/O
> - just for logging (very rarely) and for configuration storage.
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post29315
|
|
|
|
|
|
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
|
|
06/15/2009 8:03 AM
post31709
|
RE: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
Hi Oleh,
Sorry, I have been away on vacation. So I guess that this problem is
resolved. Somehow that USB must have been interfering with the Ethernet,
but without having the hardware, I couldn't tell you why.
Hugh.
-----Original Message-----
From: Oleh Derevenko [mailto:community-noreply@qnx.com]
Sent: Thursday, May 21, 2009 9:35 AM
To: general-networking
Subject: Re: RE: RE: RE: RE: RE: How to find out if that's a HW fault?
Hi, Hugh
Sorry, I don't have anything yet because our guy seems to have made all
the changes I sent him at once instead of trying three different tests.
:) However it looks like the faults do not hapen any more. So, I
suppose, disabling USB helped.
> Please can you get me the sloginfo output after the customer discovers
> that the link is down. I have been testing here all night without any
> problems. I have also been removing cables and re-inserting them while
> communication is in progress, without any problems.
>
> Thanks, Hugh.
>
> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: Tuesday, May 12, 2009 4:49 PM
> To: general-networking
> Subject: Re: RE: RE: RE: RE: How to find out if that's a HW fault?
>
> > Looking at the PCI interrupts, interrupts 10 and 11 are shared
between
> > the Network and USB controllers. Is the customer running any USB
> > devices? If not, can they slay io-usb and see if that helps
anything.
>
> I've already told them to try disabling USBs in BIOS if that is
> possible. I'll also pass your suggestion regarding io-usb to them.
>
> > The disk controller is also on interrupt 11. Is disk performance
> > affected at all?
>
> I don't know. How can we find that out? We hardly ever use any disk
I/O
> - just for logging (very rarely) and for configuration storage.
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post29315
_______________________________________________
General
http://community.qnx.com/sf/go/post29811
|
|
|
|
|
|
|
RE: RE: RE: How to find out if that's a HW fault?
|
|
05/05/2009 10:42 AM
post28701
|
RE: RE: RE: How to find out if that's a HW fault?
> I was wondering if it was a specific process using
> all available CPU time
run the following, before the slowdown happens,
in a spare window or console:
# top -p 255
> # nicinfo /dev/io-net/en1
> Packets Dropped on receive ................. 7481
This may be a red herring, but dropping packets on
receive is not good - it really hurts performance.
It might be related to your slowdown problem - perhaps
the driver rx thread is not getting a chance to run -
but if the packet loss occurs BEFORE the slowdown, it
is an indication that you need to increase the size of
your receive descriptor ring with the "receive=X" option
to the i82544 driver, where X is a number like 512 or
1024.
--
aboyd
|
|
|
|
|
|
|
Re: RE: RE: RE: How to find out if that's a HW fault?
|
|
05/06/2009 5:27 AM
post28767
|
Re: RE: RE: RE: How to find out if that's a HW fault?
>
> > I was wondering if it was a specific process using
> > all available CPU time
>
> run the following, before the slowdown happens,
> in a spare window or console:
>
> # top -p 255
Sorry, I don't have anything like this in my 6.3.0. :)
>
> > # nicinfo /dev/io-net/en1
> > Packets Dropped on receive ................. 7481
>
> This may be a red herring, but dropping packets on
> receive is not good - it really hurts performance.
>
> It might be related to your slowdown problem - perhaps
> the driver rx thread is not getting a chance to run -
> but if the packet loss occurs BEFORE the slowdown, it
> is an indication that you need to increase the size of
> your receive descriptor ring with the "receive=X" option
> to the i82544 driver, where X is a number like 512 or
> 1024.
>
Well we used this machine configuration before and normally there are all zeroes in errors. There might be few buffer
underruns on transmission from time to time.
|
|
|
|
|
|
|
RE: RE: How to find out if that's a HW fault?
|
|
05/07/2009 9:34 AM
post28897
|
RE: RE: How to find out if that's a HW fault?
Would you post the one for 6.4.1? On a machine with DID 1096, it does detect the hardware but there is no networking
and are lots of error reported by nicinfo. It works with 6.3.2, I'm told it works with 6.4.0 but I will be confirming
this latter.
> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:32 AM
> To: general-networking
> Subject: RE: RE: How to find out if that's a HW fault?
>
> The devn-e1000.so driver supersedes the i82544 driver. I have attached
> an experimental copy.
>
>
> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: Monday, May 04, 2009 9:19 AM
> To: general-networking
> Subject: Re: RE: How to find out if that's a HW fault?
>
> >
> > After considerable effort on Hugh Brown's part, intel discovered that
> > some i82544 variants have a hardware bug where they can lose
> > interrupts during certain race conditions. Oops.
> >
> > Later versions of the i82544 drivers (both io-net and io-pkt
> variants)
> > have fixes to work around this bug.
>
> Hm... Were there any updates to network drivers in 6.3.0 after SP3? I
> thought I had the latest ones available. I'll check downloads.
>
> > What is your DID?
>
> Class = Network (Ethernet)
> Vendor ID = 8086h, Intel Corporation
> Device ID = 109ah, 82573L Gigabit Ethernet Controller
>
>
>
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28603
>
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28612
|
|
|
|
|
|
|
RE: RE: How to find out if that's a HW fault?
|
|
05/07/2009 9:44 AM
post28902
|
RE: RE: How to find out if that's a HW fault?
Here is the latest experimental driver.
-----Original Message-----
From: Mario Charest [mailto:community-noreply@qnx.com]
Sent: Thursday, May 07, 2009 9:35 AM
To: general-networking
Subject: RE: RE: How to find out if that's a HW fault?
Would you post the one for 6.4.1? On a machine with DID 1096, it does
detect the hardware but there is no networking and are lots of error
reported by nicinfo. It works with 6.3.2, I'm told it works with 6.4.0
but I will be confirming this latter.
> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:32 AM
> To: general-networking
> Subject: RE: RE: How to find out if that's a HW fault?
>
> The devn-e1000.so driver supersedes the i82544 driver. I have attached
> an experimental copy.
>
>
> -----Original Message-----
> From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> Sent: Monday, May 04, 2009 9:19 AM
> To: general-networking
> Subject: Re: RE: How to find out if that's a HW fault?
>
> >
> > After considerable effort on Hugh Brown's part, intel discovered
that
> > some i82544 variants have a hardware bug where they can lose
> > interrupts during certain race conditions. Oops.
> >
> > Later versions of the i82544 drivers (both io-net and io-pkt
> variants)
> > have fixes to work around this bug.
>
> Hm... Were there any updates to network drivers in 6.3.0 after SP3? I
> thought I had the latest ones available. I'll check downloads.
>
> > What is your DID?
>
> Class = Network (Ethernet)
> Vendor ID = 8086h, Intel Corporation
> Device ID = 109ah, 82573L Gigabit Ethernet Controller
>
>
>
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28603
>
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28612
_______________________________________________
General
http://community.qnx.com/sf/go/post28897
|
|
|
|
|
|
|
RE: RE: How to find out if that's a HW fault?
|
|
05/07/2009 11:55 AM
post28950
|
RE: RE: How to find out if that's a HW fault?
Thanks you, looking good so far!
> -----Original Message-----
> From: Hugh Brown [mailto:community-noreply@qnx.com]
> Sent: May-07-09 9:45 AM
> To: general-networking
> Subject: RE: RE: How to find out if that's a HW fault?
>
> Here is the latest experimental driver.
>
>
> -----Original Message-----
> From: Mario Charest [mailto:community-noreply@qnx.com]
> Sent: Thursday, May 07, 2009 9:35 AM
> To: general-networking
> Subject: RE: RE: How to find out if that's a HW fault?
>
> Would you post the one for 6.4.1? On a machine with DID 1096, it does
> detect the hardware but there is no networking and are lots of error
> reported by nicinfo. It works with 6.3.2, I'm told it works with 6.4.0
> but I will be confirming this latter.
>
> > -----Original Message-----
> > From: Hugh Brown [mailto:community-noreply@qnx.com]
> > Sent: May-04-09 9:32 AM
> > To: general-networking
> > Subject: RE: RE: How to find out if that's a HW fault?
> >
> > The devn-e1000.so driver supersedes the i82544 driver. I have
> attached
> > an experimental copy.
> >
> >
> > -----Original Message-----
> > From: Oleh Derevenko [mailto:community-noreply@qnx.com]
> > Sent: Monday, May 04, 2009 9:19 AM
> > To: general-networking
> > Subject: Re: RE: How to find out if that's a HW fault?
> >
> > >
> > > After considerable effort on Hugh Brown's part, intel discovered
> that
> > > some i82544 variants have a hardware bug where they can lose
> > > interrupts during certain race conditions. Oops.
> > >
> > > Later versions of the i82544 drivers (both io-net and io-pkt
> > variants)
> > > have fixes to work around this bug.
> >
> > Hm... Were there any updates to network drivers in 6.3.0 after SP3? I
> > thought I had the latest ones available. I'll check downloads.
> >
> > > What is your DID?
> >
> > Class = Network (Ethernet)
> > Vendor ID = 8086h, Intel Corporation
> > Device ID = 109ah, 82573L Gigabit Ethernet Controller
> >
> >
> >
> >
> >
> > _______________________________________________
> > General
> > http://community.qnx.com/sf/go/post28603
> >
> >
> >
> > _______________________________________________
> > General
> > http://community.qnx.com/sf/go/post28612
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28897
>
>
>
> _______________________________________________
> General
> http://community.qnx.com/sf/go/post28902
|
|
|
|
|
|
|
RE: How to find out if that's a HW fault?
|
|
05/04/2009 9:34 AM
post28616
|
RE: How to find out if that's a HW fault?
> -----Original Message-----
> From: Andrew Boyd [mailto:community-noreply@qnx.com]
> Sent: May-04-09 9:07 AM
> To: general-networking
> Subject: RE: How to find out if that's a HW fault?
>
>
> After considerable effort on Hugh Brown's part,
Hugh, thank you for your considerable effort ;-)
|
|
|
|
|
|