Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - network buffer issues: Page 1 of 3 (22 Items)
   
network buffer issues  
Hi all,

We have recently started a project based on io-pkt infrastructure,   how ever we've observed following symptoms 
regarding network stack, I'd appreciate if anyone could shed some light on where this problem might come from? (network 
stack setup/configuration or build)


============================================
#1 . ping test with bigger buffer (ping out / ping in  has same symptom ): 
 

[/]# ping -c 100 -s 4432 172.16.1.48

PING 172.16.1.48 (172.16.1.48): 4432 data bytes

4440 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=9 ms

4440 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=9 ms

4440 bytes from 172.16.1.48: icmp_seq=2 ttl=128 time=8 ms

4440 bytes from 172.16.1.48: icmp_seq=3 ttl=128 time=8 ms

4440 bytes from 172.16.1.48: icmp_seq=4 ttl=128 time=725 ms

 

----172.16.1.48 PING Statistics----

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max = 8/151/725 ms   variance = 102977 ms^2

[/]# ping -c 100 -s 4433 172.16.1.48

PING 172.16.1.48 (172.16.1.48): 4433 data bytes

 

----172.16.1.48 PING Statistics----

100 packets transmitted, 0 packets received, 100% packet loss

[/]#

 

ping test with bigger buffer is OK within LAN , not cross LAN, and same behavior happens to both wired and wireless 
interfaces (wireless interface using wpa_supplicant provided on the board to associate with AP in an open mode) , so I 
suspect that there is an buffer issue at layer that is higher than layer2 (tcp/ip possible); 
[/etc]# ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192

        inet 127.0.0.1 netmask 0xff000000

ath0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

        ssid AcmeWifi

        powersave off

        bssid 00:03:52:e0:46:f0 chan 7

        address: 00:06:80:00:a1:ed

        media: IEEE802.11 autoselect (OFDM12 mode 11g)

        status: active

        inet 192.168.10.87 netmask 0xffffff00 broadcast 192.168.10.255

en0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

        address: 00:55:44:33:22:11

        inet 172.16.3.54 netmask 0xffff0000 broadcast 172.16.255.255

 

 

============= Ethernet interface en0 ==============

 

[/etc]# ping -s 5000 172.16.3.1

PING 172.16.3.1 (172.16.3.1): 5000 data bytes

5008 bytes from 172.16.3.1: icmp_seq=0 ttl=64 time=11 ms

5008 bytes from 172.16.3.1: icmp_seq=1 ttl=64 time=12 ms

 

----172.16.3.1 PING Statistics----

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 11/11/12 ms   variance = 23 ms^2

 

[/etc]# ping -s 5000 172.16.1.48

PING 172.16.1.48 (172.16.1.48): 5000 data bytes

 

----172.16.1.48 PING Statistics----

221 packets transmitted, 0 packets received, 100% packet loss

 

 

[/etc]# ping  172.16.1.48

PING 172.16.1.48 (172.16.1.48): 56 data bytes

64 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=1 ms

64 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=1 ms

 

----172.16.1.48 PING Statistics----

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 1/1/1 ms   variance = 0 ms^2

 

============== wireless interface  ath0 =========

 

/etc]# ifconfig en0 down

[/etc]# ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192

        inet 127.0.0.1 netmask 0xff000000

ath0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

        ssid Devicescape-Guest

        powersave off

        bssid 00:14:c2:a5:4c:41 chan 10

        address: 00:06:80:00:a1:ed

        media: IEEE802.11 autoselect (OFDM54 mode 11g)

        status: active

        inet 192.168.30.10 netmask 0xffffff00 broadcast 192.168.30.255

en0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500

        address: 00:55:44:33:22:11

        inet 172.16.3.54 netmask 0xffff0000 broadcast 172.16.255.255

[/etc]# ping...
View Full Message
Re: network buffer issues  
On Thu, Jan 03, 2008 at 12:16:07PM -0500, Shannon Gu wrote:
> Hi all,
> 
> We have recently started a project based on io-pkt infrastructure,   how ever we've observed following symptoms 
regarding network stack, I'd appreciate if anyone could shed some light on where this problem might come from? (network 
stack setup/configuration or build)
> 
> 
> ============================================
> #1 . ping test with bigger buffer (ping out / ping in  has same symptom ): 
>  
> 
> [/]# ping -c 100 -s 4432 172.16.1.48
> 
> PING 172.16.1.48 (172.16.1.48): 4432 data bytes
> 
> 4440 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=9 ms
> 
> 4440 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=9 ms
> 
> 4440 bytes from 172.16.1.48: icmp_seq=2 ttl=128 time=8 ms
> 
> 4440 bytes from 172.16.1.48: icmp_seq=3 ttl=128 time=8 ms
> 
> 4440 bytes from 172.16.1.48: icmp_seq=4 ttl=128 time=725 ms
> 
>  
> 
> ----172.16.1.48 PING Statistics----
> 
> 5 packets transmitted, 5 packets received, 0% packet loss
> 
> round-trip min/avg/max = 8/151/725 ms   variance = 102977 ms^2
> 
> [/]# ping -c 100 -s 4433 172.16.1.48
> 
> PING 172.16.1.48 (172.16.1.48): 4433 data bytes
> 
>  
> 
> ----172.16.1.48 PING Statistics----
> 
> 100 packets transmitted, 0 packets received, 100% packet loss
> 
> [/]#
> 

Something must have changed between the two runs.  It's hard
to say with the providfed info but it may be a routing issue.
Make sure 172.16.1.48 has a route back to you.

>  
> 
> ping test with bigger buffer is OK within LAN , not cross LAN, and same behavior happens to both wired and wireless 
interfaces (wireless interface using wpa_supplicant provided on the board to associate with AP in an open mode) , so I 
suspect that there is an buffer issue at layer that is higher than layer2 (tcp/ip possible); 
> [/etc]# ifconfig
> 
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
> 
>         inet 127.0.0.1 netmask 0xff000000
> 
> ath0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> 
>         ssid AcmeWifi
> 
>         powersave off
> 
>         bssid 00:03:52:e0:46:f0 chan 7
> 
>         address: 00:06:80:00:a1:ed
> 
>         media: IEEE802.11 autoselect (OFDM12 mode 11g)
> 
>         status: active
> 
>         inet 192.168.10.87 netmask 0xffffff00 broadcast 192.168.10.255
> 
> en0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> 
>         address: 00:55:44:33:22:11
> 
>         inet 172.16.3.54 netmask 0xffff0000 broadcast 172.16.255.255
> 
>  
> 
>  
> 
> ============= Ethernet interface en0 ==============
> 
>  
> 
> [/etc]# ping -s 5000 172.16.3.1
> 
> PING 172.16.3.1 (172.16.3.1): 5000 data bytes
> 
> 5008 bytes from 172.16.3.1: icmp_seq=0 ttl=64 time=11 ms
> 
> 5008 bytes from 172.16.3.1: icmp_seq=1 ttl=64 time=12 ms
> 
>  
> 
> ----172.16.3.1 PING Statistics----
> 
> 2 packets transmitted, 2 packets received, 0% packet loss
> 
> round-trip min/avg/max = 11/11/12 ms   variance = 23 ms^2
> 
>  
> 
> [/etc]# ping -s 5000 172.16.1.48
> 
> PING 172.16.1.48 (172.16.1.48): 5000 data bytes
> 
>  
> 
> ----172.16.1.48 PING Statistics----
> 
> 221 packets transmitted, 0 packets received, 100% packet loss
> 
>  
> 
>  
> 
> [/etc]# ping  172.16.1.48
> 
> PING 172.16.1.48 (172.16.1.48): 56 data bytes
> 
> 64 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=1 ms
> 
> 64 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=1 ms
> 
>  
> 
> ----172.16.1.48 PING Statistics----
> 
> 2 packets transmitted, 2 packets...
View Full Message
Re: network buffer issues  

============================================
> > #1 . ping test with bigger buffer (ping out / ping in  has same symptom ): 
> >  
> > 
> > [/]# ping -c 100 -s 4432 172.16.1.48
> > 
> > PING 172.16.1.48 (172.16.1.48): 4432 data bytes
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=9 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=9 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=2 ttl=128 time=8 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=3 ttl=128 time=8 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=4 ttl=128 time=725 ms
> > 
> >  
> > 
> > ----172.16.1.48 PING Statistics----
> > 
> > 5 packets transmitted, 5 packets received, 0% packet loss
> > 
> > round-trip min/avg/max = 8/151/725 ms   variance = 102977 ms^2
> > 
> > [/]# ping -c 100 -s 4433 172.16.1.48
> > 
> > PING 172.16.1.48 (172.16.1.48): 4433 data bytes
> > 
> >  
> > 
> > ----172.16.1.48 PING Statistics----
> > 
> > 100 packets transmitted, 0 packets received, 100% packet loss
> > 
> > [/]#
> > 
> 
> Something must have changed between the two runs.  It's hard
> to say with the providfed info but it may be a routing issue.
> Make sure 172.16.1.48 has a route back to you.
>


Yes 172.16.1.48 has a route back, but expeirence the same size issues to the target. 
Another observation is that the packet size is destination dependent, for every host I pinged, each one fixes at certain
 buffer size like shreshhold, some at couple of kilo bytes, some at several, I have not seen any one exeeded 10 Kb ...

Thanks a lot for your prompt response!
Regards,
Shannon
 
RE: network buffer issues  
Is the router between the subnet not letting larger ICMP packets through?
Is the truncation tied to a subnet or to particular hosts on the subnet?  

Is it possible to put a sniffer on one of the other hosts to see if the
packets are being truncated?  Are the other hosts all running the same OS?
Does the reverse path (pinging from the host to QNX work)?

Problem here is that we can't really tell where things are falling apart. It
would seem (:->) that if everything on the same subnet is working with large
packets then the QNX side of things is working properly.  What you really
need to do is follow the larger packets through the network and try and see
where they're getting lost or discarded.


	Robert.

-----Original Message-----
From: Shannon Gu [mailto:shannon@devicescape.com] 
Sent: Monday, January 07, 2008 1:59 PM
To: ionetmig-networking
Subject: Re: network buffer issues



============================================
> > #1 . ping test with bigger buffer (ping out / ping in  has same symptom
): 
> >  
> > 
> > [/]# ping -c 100 -s 4432 172.16.1.48
> > 
> > PING 172.16.1.48 (172.16.1.48): 4432 data bytes
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=0 ttl=128 time=9 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=1 ttl=128 time=9 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=2 ttl=128 time=8 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=3 ttl=128 time=8 ms
> > 
> > 4440 bytes from 172.16.1.48: icmp_seq=4 ttl=128 time=725 ms
> > 
> >  
> > 
> > ----172.16.1.48 PING Statistics----
> > 
> > 5 packets transmitted, 5 packets received, 0% packet loss
> > 
> > round-trip min/avg/max = 8/151/725 ms   variance = 102977 ms^2
> > 
> > [/]# ping -c 100 -s 4433 172.16.1.48
> > 
> > PING 172.16.1.48 (172.16.1.48): 4433 data bytes
> > 
> >  
> > 
> > ----172.16.1.48 PING Statistics----
> > 
> > 100 packets transmitted, 0 packets received, 100% packet loss
> > 
> > [/]#
> > 
> 
> Something must have changed between the two runs.  It's hard
> to say with the providfed info but it may be a routing issue.
> Make sure 172.16.1.48 has a route back to you.
>


Yes 172.16.1.48 has a route back, but expeirence the same size issues to the
target. 
Another observation is that the packet size is destination dependent, for
every host I pinged, each one fixes at certain buffer size like shreshhold,
some at couple of kilo bytes, some at several, I have not seen any one
exeeded 10 Kb ...

Thanks a lot for your prompt response!
Regards,
Shannon
 

_______________________________________________
io-net migration
http://community.qnx.com/sf/go/post4099
Re: network buffer issues  
============================================
> > #2. wpa_supplicnt in PSK mode stopped and network on board is basically 
> crashed: 
> >  
> > a. wpa_supplicant stopped during 4-way handshake after receiving Key; 
> sniffer shows that the client(the board) never response to key exchange; 
> > 
> > b. AP shows that WPA pairwise key handshake timed out ( no reply was 
> received for the first massage) 
> > 
> > c. On board , the shell still working, except ifconfig command hangs; but it
>  is able to quit after inputting Ctrl-C; 
> > 
> > d. supplicant involves pcap lib when using security mode to transfer packets
>  while in open mode does not; 
> >     So if I put a stop at calling pcap_inject before let it continue, then 
> network system on board wont crash, the program will go on and on, the 
> buffered packets will be eventually sent out; 
> >     but this is only at a slow speed, if I hit the continue right away then 
> the system will hang right away, just like what a~c described; the result is 
> that the board will never get authenticated with any AP as it always times out
> ??? 
> 
> Some more detailed info on how to reproduce this would be
> appreciated: config files etc.
> 


1. wpa_supplicant command is :
    [/]#wpa_supplicant -Dbsd -iath0 -c/etc/shxy.conf -ddd 
2. [/]# cat etc/shxy.conf
ctrl_interface=/var/run/wpa_supplicant
network={
        disabled=0
        ssid="tomato-DLS"
        proto=WPA2
        key_mgmt=WPA-PSK
        psk="secureDLS"
}

3. I attached some reference files here:
 -- An Ethereal file that captured all the related packets;
 -- A wpa_supplicant log file; (next post)


Thanks a lot,
Shannon
Attachment: Text WPA-PSK-iopkt-buffer-issues 3.21 KB
Re: network buffer issues  
============================================
> > #2. wpa_supplicnt in PSK mode stopped and network on board is basically 
> crashed: 


This is to attach the log file

Regards,
Shannon
Attachment: Text supplicant-log-WPA-PSK-io-pkt-buffer-issues 4.81 KB
RE: network buffer issues  
Hi Shannon:

	I've got someone looking into this to see if we can reproduce it.
Sorry for not getting back to you sooner.  Most people have been away for
the last couple of weeks on Christmas break.

	Robert.

-----Original Message-----
From: Shannon Gu [mailto:shannon@devicescape.com] 
Sent: Monday, January 07, 2008 4:11 PM
To: ionetmig-networking
Subject: Re: network buffer issues

============================================
> > #2. wpa_supplicnt in PSK mode stopped and network on board is basically 
> crashed: 


This is to attach the log file

Regards,
Shannon

_______________________________________________
io-net migration
http://community.qnx.com/sf/go/post4108
Re: RE: network buffer issues  
Hi, I have tried to reproduce the problem (WPA2 4-way handshake) but I can't reproduce it here. I have tried to 
associate to an access point with the short preamble ability (as the one you are using has this ability) and also 
another access point without the short preamble ability. Also besides the io-pkt-v4-hc (that is what you are running), I
 have also tried another flavor like io-pkt-v6. They all work fine. 

So I think it would be better to solve/understand the following two suspectable points. I also listed my wpa conf file 
at the end and attached my log of accessing an WPA2+psk access point. Hope it be helpful.

1) the ifconfig got blocked (although ^C can make it free). I can't see it on my side either. Can you check the pidin to
 see what is the status of the io-pkt? I noticed in you very first email, there is a script, (?) bt_detect? where you 
have one io-net and two io-pkt instance running? If you intended to run more than one instances of io-net and/or io-pkt,
 you have to use the prefix option, otherwise, neither of them work as expected. Anyway, please use pidin to see if any 
suspectable point. I think we'd better to solve it first (ifconfig blocks) and go further onto such as wpa2.

==== the section in your script ?? ===============
/usr/sbin/random

#/proc/boot/io-net -ptcpip -ppppmgr &

/proc/boot/waitfor /dev/random 4

#/sbin/io-pkt-v4-hc random -ptcpip -pppmgr &

/platform/bin/wd &

/platform/bin/pm &

/platform/bin/doc &

/platform/bin/adi &

/proc/boot/waitfor /dev/pm/wifi/controls/enable 4

/sbin/pci-lge5200b

/proc/boot/waitfor /dev/pci 4

/sbin/io-pkt-v4-hc random -dath -ptcpip -pppmgr &

====================


2) I noticed there is more than 1 second between the broadcast and the first message sending to your access point. It 
might indicate a problem of difficulty of accessing your access point. You may run "ifconfig ath0 scan" to see if you 
can get the list of wireless devices around you and please check if the signal of your access point is too weak. Also 
please check your access configuration settings and make sure it is correct. (I blindly suspect the signal is too weak 
on your side and if it is the case, move closer to your access point and try again ...)

I list below the wpa.conf that I am using and the attached is the log file that the io-pkt+ath successfully associated 
to my access point (Linksys Wireless-G WRT54GS).

=== my wpa2.conf ===
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
ap_scan=1

network={
       ssid="linksys"
#	proto=RSN 
	proto=WPA2
       key_mgmt=WPA-PSK
#       pairwise=CCMP TKIP
#       group=CCMP TKIP
       psk="LINKSYSWPAKEY"
}
===================
Attachment: Text mylog 10.41 KB
Re: RE: network buffer issues  
Hi Robert,

I think I need to give more background of this project. The nature of it is to port our wpa_supplicant to QNX platform 
on top of io-pkt stack.
 
The hardware and the kernel come with it are from our customer;  these network issues has been found on this particular 
target, therefore I dont think they can be reproduced on a different type of target.

The network subnet envrionment is pretty standard, and works fine with all other devices that has been connected with --
 including different type board/cpu/wireless driver, for example, we have put a PPC board running QNX 6 with io-net 
network stack and a Broadcom chip on the same subnet, it can shoot big buffer packets no problem at all (up to the limit
 65K); so I wouldn't suspect the subnet too much, (let me know if this makes sense). 

I have requested our customer to reproduce the problem on same configuration target, but I have not gotten results back;
 

I suspect the problem is related to network configuration while building the kernel or some memory issues specific to 
this board -- any advice regarding this perspective? Or have you encountered similar problems before?


Thanks a lot, 
Regards,
Shannon
Re: RE: network buffer issues  
Dear team

I'm LGE engineer. I performed same test by using h/w which are same with the one Shannon used for his test.
And found that the board stop running  during wpa_supplicant execution. 

Following is my configuration and I'm attaching execution log. 

[/etc]# cat shxy.conf
ctrl_interface=/usr/sbin/wpa_supplicant
network={
        disabled=0
        ssid="FCP_dooeui"
        proto=WPA
        key_mgmt=WPA-PSK
        psk="secureDLS"
}
 
[/etc]# 
 
 
[/]# 
[/]# ifconfig ath0 ssid FCP_dooeui
[/]# ifconfig ath0 192.168.1.50 netmask 255.255.255.0
[/]# route add default 192.168.1.245
add net default: gateway 192.168.1.245
[/]# 
[/]# wpa_supplicant -Dbsd -iath0 -c/etc/shxy.conf -ddd 
Attachment: Text LGE_testResult.txt 5.36 KB