Project Home
Project Home
Wiki
Wiki
Discussion Forums
Discussions
Project Information
Project Info
Forum Topic - Problems with Multicast Receive: (9 Items)
   
Problems with Multicast Receive  
Hi,

I am having some problem with the multicast packets that I receive. I use a raw socket to send multicast packet and set 
the TTL field to 1 before I send the IP packet. On the other point to point interface when I receive the multicast 
packet, through tcpdump program I see that I receive the packet with the TTL field correctly set. However, when I 
receive the packet in my application and check the TTL field, it is set to 255 (0xff)!!!! Im fully expecting it to be 0 
as the TTL field ought to just decrement by 1.

Another problem is this:

I have multiple interfaces and all interfaces join a specific multicast group. With the above described setup, I seem to
 receive multicast packets coming in from a different interface arrive on a wrong interface i.e, multicast packets say 
arriving on en6 comes in on en3. Is that due to the TTL field now having set to 255?

Help!!
RE: Problems with Multicast Receive  
Hi Santosh:

	Sorry about that.  This one seems to have slipped through the
cracks.

First:  On the receive side, is the TTL set to 0 or 1 as seen by
tcpdump? 

It does indeed seem strange that the TTL wraps to 255 on receive at the
application layer.  I'll have to take a look at the source...  Out of
curiosity, why would the value of the TTL matter to your application?
Do you have some sample source code that we can have a look at for the
send and receive apps?

For the multicast packet being received on an unexpected interface,
would you know if the problem is with the reported interface or is the
multicast packet actually on the wire in a wrong subnet?  What does your
network look like (in terms of subnets / routers and connectivity)?
What drivers are you using?  When you say point-to-point, do you mean
that these are PPP links over serial?

I'll also ask the obvious question... what happens if you set the TTL to
2?

	Robert.

-----Original Message-----
From: Santosh Kumar [mailto:community-noreply@qnx.com] 
Sent: Saturday, November 22, 2008 12:14 AM
To: technology-networking
Subject: Problems with Multicast Receive

Hi,

I am having some problem with the multicast packets that I receive. I
use a raw socket to send multicast packet and set the TTL field to 1
before I send the IP packet. On the other point to point interface when
I receive the multicast packet, through tcpdump program I see that I
receive the packet with the TTL field correctly set. However, when I
receive the packet in my application and check the TTL field, it is set
to 255 (0xff)!!!! Im fully expecting it to be 0 as the TTL field ought
to just decrement by 1.

Another problem is this:

I have multiple interfaces and all interfaces join a specific multicast
group. With the above described setup, I seem to receive multicast
packets coming in from a different interface arrive on a wrong interface
i.e, multicast packets say arriving on en6 comes in on en3. Is that due
to the TTL field now having set to 255?

Help!!

_______________________________________________
Technology
http://community.qnx.com/sf/go/post17128
RE: Problems with Multicast Receive  
 
Hi Santosh:

	I've got some of this info from the internal support people.
I'll work through them for the time being and post the resolution (if
relevant) here to tie up the thread for others who may be interested.

	Thanks!
		Robert.


-----Original Message-----
From: Robert Craig 
Sent: Monday, December 01, 2008 3:41 PM
To: 'post17128@community.qnx.com'
Cc: Ted Zhang
Subject: RE: Problems with Multicast Receive

Hi Santosh:

	Sorry about that.  This one seems to have slipped through the
cracks.

First:  On the receive side, is the TTL set to 0 or 1 as seen by
tcpdump? 

It does indeed seem strange that the TTL wraps to 255 on receive at the
application layer.  I'll have to take a look at the source...  Out of
curiosity, why would the value of the TTL matter to your application?
Do you have some sample source code that we can have a look at for the
send and receive apps?

For the multicast packet being received on an unexpected interface,
would you know if the problem is with the reported interface or is the
multicast packet actually on the wire in a wrong subnet?  What does your
network look like (in terms of subnets / routers and connectivity)?
What drivers are you using?  When you say point-to-point, do you mean
that these are PPP links over serial?

I'll also ask the obvious question... what happens if you set the TTL to
2?

	Robert.

-----Original Message-----
From: Santosh Kumar [mailto:community-noreply@qnx.com]
Sent: Saturday, November 22, 2008 12:14 AM
To: technology-networking
Subject: Problems with Multicast Receive

Hi,

I am having some problem with the multicast packets that I receive. I
use a raw socket to send multicast packet and set the TTL field to 1
before I send the IP packet. On the other point to point interface when
I receive the multicast packet, through tcpdump program I see that I
receive the packet with the TTL field correctly set. However, when I
receive the packet in my application and check the TTL field, it is set
to 255 (0xff)!!!! Im fully expecting it to be 0 as the TTL field ought
to just decrement by 1.

Another problem is this:

I have multiple interfaces and all interfaces join a specific multicast
group. With the above described setup, I seem to receive multicast
packets coming in from a different interface arrive on a wrong interface
i.e, multicast packets say arriving on en6 comes in on en3. Is that due
to the TTL field now having set to 255?

Help!!

_______________________________________________
Technology
http://community.qnx.com/sf/go/post17128
Re: RE: Problems with Multicast Receive  
Hi Robert,

Please see my response embedded below:

Robert Craig wrote:
> Hi Santosh:
>
> 	Sorry about that.  This one seems to have slipped through the
> cracks.
>
> First:  On the receive side, is the TTL set to 0 or 1 as seen by
> tcpdump? 
[SK]: TTL is set to 1at the receive end when observed through tcpdump
>
> It does indeed seem strange that the TTL wraps to 255 on receive at the
> application layer.  I'll have to take a look at the source...  Out of
> curiosity, why would the value of the TTL matter to your application?
> Do you have some sample source code that we can have a look at for the
> send and receive apps?
[SK] We have maximum upto 8 interfaces that join the same multicast group on each node in the network. When a multicast 
message is received by the node from its neighbor, we want to ensure that the packet dies after one hop and do not get 
propagated to potentially 7 other interfaces. In other words we do not want to see the multicast packets get forwarded. 
to the other interfaces. As such, I dont think it is an issue by itself since we are not setting up the node to be a 
multicast router, but I  have a doubt that my observation of seeing a packet seemingly coming from a different interface
 than expected is because the packet does not die and somehow gets forwarded/multicasted to other interfaces on the same
 node which also have joined to the same multicast group.

>
> For the multicast packet being received on an unexpected interface,
> would you know if the problem is with the reported interface or is the
> multicast packet actually on the wire in a wrong subnet?  What does your
> network look like (in terms of subnets / routers and connectivity)?
> What drivers are you using?  When you say point-to-point, do you mean
> that these are PPP links over serial?
>
[SK] our network setup is like this:

CPU running io-pkt<-->BCM5691 Chip<-->Gige Link<->BCM5691 Chip<->CPU running io-pkt.

The CPU is not a switching interface of the Chip and as per observation of tcpdump the TTL isnt decremented.
> I'll also ask the obvious question... what happens if you set the TTL to
> 2?
>
[SK] I have checked that and the result is the same. TTL still is seen as 255 at application layer.

-Santosh
RE: RE: Problems with Multicast Receive  
Hmmm...  The fact that the TTL is the same whether you start with 1 or 2 is now making me wonder if it's just an 
artifact of lib socket or something else above the IP layer.  From my quick dig through the code, the only place that 
the TTL is decremented is during a forwarding or bridging event (which is as it should be). If we were dealing with a 
simple decrement problem, then we'd see different results.

With a case like:
CPU running io-pkt<-->BCM5691 Chip<-->Gige Link<->BCM5691 Chip<->CPU running io-pkt.

io-pkt shouldn't have any opportunity to decrement the TTL because it isn't doing any forwarding / bridging.  I'm 
assuming that the BCM5691 chip is acting like a layer2 switch so it shouldn't be decrementing the TTL either (and you 
see the TTL=1 using tcpdump anyway, so that's OK).  

So there could be something between the IP layer and the app that resets the TTL for some (unknown) reason.
Do you have a short code snippet showing how you set up the receive side of the raw socket?  

I'm thinking that the TTL being reported isn't your real problem, though.  The REAL problem is that you're seeing a 
multicast packet being delivered and it's either coming from an interface that you think that it shouldn't be coming 
from OR the packet is reported as coming in on the wrong interface.

To check and see if the packet is coming in on a particular interface, you should be able to use the BPF interface.  It 
grabs packets directly from the driver receive function so it bypasses any of the stack processing.  You should be able 
to get it to catch a multicast Ethernet packet which will tell you a) whether or not the packet is coming in on that 
interface and b) what the TTL is "on the wire" before it goes through any protocol processing.  Are you using your own 
custom drivers written as for io-net using the shim or is it some other driver?

Is it possible that the BCM5691 chip is unexpectedly bridging packets out other interfaces?

     Robert.



-----Original Message-----
From: Santosh Kumar [mailto:community-noreply@qnx.com]
Sent: Mon 12/1/2008 7:00 PM
To: technology-networking
Subject: Re: RE: Problems with Multicast Receive
 
Hi Robert,

Please see my response embedded below:

Robert Craig wrote:
> Hi Santosh:
>
> 	Sorry about that.  This one seems to have slipped through the
> cracks.
>
> First:  On the receive side, is the TTL set to 0 or 1 as seen by
> tcpdump? 
[SK]: TTL is set to 1at the receive end when observed through tcpdump
>
> It does indeed seem strange that the TTL wraps to 255 on receive at the
> application layer.  I'll have to take a look at the source...  Out of
> curiosity, why would the value of the TTL matter to your application?
> Do you have some sample source code that we can have a look at for the
> send and receive apps?
[SK] We have maximum upto 8 interfaces that join the same multicast group on each node in the network. When a multicast 
message is received by the node from its neighbor, we want to ensure that the packet dies after one hop and do not get 
propagated to potentially 7 other interfaces. In other words we do not want to see the multicast packets get forwarded. 
to the other interfaces. As such, I dont think it is an issue by itself since we are not setting up the node to be a 
multicast router, but I  have a doubt that my observation of seeing a packet seemingly coming from a different interface
 than expected is because the packet does not die and somehow gets forwarded/multicasted to other interfaces on the same
 node which also have joined to the same multicast group.

>
> For the multicast packet being received on an unexpected interface,
> would you know if the problem is with the reported interface or is the
> multicast packet actually on the wire in a wrong subnet?  What does your
> network look like (in terms of subnets / routers and connectivity)?
> What...
View Full Message
Attachment: Text winmail.dat 4.62 KB
Re: RE: RE: Problems with Multicast Receive  
Hi Robert,

I agree that my real problem is about seeing the packet coming in from a wrong interface.

Im not sure BPF is going to help in my case. Let me explain the setup, which is slightly more complicated.

The BCM5691 has got 12 switching ports and one CPU port (PCI based) which is not a part of switching interface but is 
used purely for management. We had a requirement which required us to be able to send special packets on a specific 
switching port and also know which switching port the packet came in from. Given the features of BCM5691, the way we 
achieved this is that for these special packets that had these requirement, we created an ethernet interface for each of
 the switching port. I modified the io-pkt such that we are able to send out packets on a given interface (based on 
ifindex). These interfaces all used the same PCI interface as physical medium.

We setup the BCM5691 L2 tables such that these special packets are passed on to the CPU port and thus would be received 
by the BCM5691 driver. The driver knows which interface the packet arrived on and passes it to the right ethernet port 
that io-pkt recognizes.

Do you think BPF would be useful in this case? I think it would only see what the driver passes. So, if the bug is in 
the driver BPF might not be able to match. In any case, we checked the driver logs and it did seem to pass the packet to
 the right interface.

We are using custom driver written for io-net using shim. The HAL layer of the driver is a modified version of QNX 
porting attempt which we got from QNX. Mainly however, Broadcom provides a driver library that controls the chip. Our 
driver merely makes calls to their API's.

Attached is the code snippet of receive code:

#define MAX_IP_PACKET_SIZE 16384

typedef struct _ipv4_packet_t_ {
    u_int8_t vhlen; //Version and header length.
    u_int8_t tos; //Type of service.
    u_int16_t totlen; //Total length in bytes.
    u_int16_t id; //Identification.
    u_int16_t odmffo; //0,DF,MF and framgment offset fields.
    u_int8_t ttl; //Time to live.
    u_int8_t protocol; //Protocol type in payload.
    u_int16_t chksum; //Checksum.
    struct in_addr source; //Source IPv4 address.
    struct in_addr destination; //Destination IPv4 address.
} ipv4_packet_t;

static u_int8_t __irp_rcvbuffer[MAX_IP_PACKET_SIZE]; /* Receiving buffer */

pkt_length = SSAPI_recvfrom( fd, __irp_rcvbuffer, sizeof( __irp_rcvbuffer ), 0,
                                              ( struct sockaddr * )&from, &fromlen, &if_index );
if ( pkt_length != -1 ) {
    /*Match protocol and process the data packet */
    rcv_pkt = ( ipv4_packet_t * ) __irp_rcvbuffer;
    printf("ttl=%hhu\n",rcv_pkt->ttl);
}

/*----------------------------------------------------------------------*/
/* Wrapper function to receive packet inside the internal optical network */
public int32_t
SSAPI_recvfrom(int sock_fd, object_t buf, size_t len, int flags, struct sockaddr *from, socklen_t *fromlen, uint32_t *
port_number)
{

    struct msghdr msg;
    struct iovec iov;
    int32_t pkt_length;
    u_int8_t cmsgdata[128];
    struct cmsghdr *cmsg;

    msg.msg_name=from;
    msg.msg_namelen=sizeof(struct sockaddr);
    iov.iov_len = len;
    iov.iov_base = buf;
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;
    msg.msg_control = cmsgdata;
    msg.msg_controllen = sizeof( cmsgdata );
    msg.msg_flags=0;

    /*recvmsg*/
    pkt_length = recvmsg( sock_fd, &msg, flags );
    if ( pkt_length >= 0 ) {
        *fromlen=msg.msg_namelen;
        if (port_number) {
            for ( cmsg = SSI_CMSG_FIRSTHDR( &msg ); cmsg != NULL;
                   cmsg = SSI_CMSG_NXTHDR( &msg, cmsg ) ) {
                if ( cmsg->cmsg_level == IPPROTO_IP && 
                     cmsg->cmsg_type ==IP_RECVIF ) {
                    struct sockaddr_dl *sock_dl;

                    sock_dl = ( struct sockaddr_dl * )CMSG_DATA( cmsg );
                 ...
View Full Message
RE: RE: RE: Problems with Multicast Receive  
Hi Santosh:

Well that certainly adds another layer of complexity to things...  So
the packets are sent out specific interfaces over individual drivers
which have APIs into the BCM5691 chip.  Packets of interest are all
received on the management interface, and passed into a separate layer
which then determines the appropriate io-pkt interface that the packet
should be forwarded on to....

Let's go back to the TTL for a minute.  Is io-pkt set up to do
forwarding at all?  Unless you specifically enable it with a sysctl, no
forwarding should be happening and the fact that you're seeing a TTL of
255 may be an artifact of the way that the packet is being transferred
out of the stack.

If you have a single ingress port for all of the packets, can you
instrument it so that it keeps track of the TTL of all multicast packets
received?  Any received with a TTL not equal to 1 would suggest that
something is incorrectly decrementing the TTL.  If all the packets are
received with the right TTL, then the TTL can't be the issue and you're
very likely dealing with something going wrong in the BCM driver layer.

We'll continue trying to duplicate your TTL issue here as well to figure
out why it shows up as 255 after leaving the stack.

   R.

-----Original Message-----
From: Santosh Kumar [mailto:community-noreply@qnx.com] 
Sent: Monday, December 01, 2008 11:10 PM
To: technology-networking
Subject: Re: RE: RE: Problems with Multicast Receive

Hi Robert,

I agree that my real problem is about seeing the packet coming in from a
wrong interface.

Im not sure BPF is going to help in my case. Let me explain the setup,
which is slightly more complicated.

The BCM5691 has got 12 switching ports and one CPU port (PCI based)
which is not a part of switching interface but is used purely for
management. We had a requirement which required us to be able to send
special packets on a specific switching port and also know which
switching port the packet came in from. Given the features of BCM5691,
the way we achieved this is that for these special packets that had
these requirement, we created an ethernet interface for each of the
switching port. I modified the io-pkt such that we are able to send out
packets on a given interface (based on ifindex). These interfaces all
used the same PCI interface as physical medium.

We setup the BCM5691 L2 tables such that these special packets are
passed on to the CPU port and thus would be received by the BCM5691
driver. The driver knows which interface the packet arrived on and
passes it to the right ethernet port that io-pkt recognizes.

Do you think BPF would be useful in this case? I think it would only see
what the driver passes. So, if the bug is in the driver BPF might not be
able to match. In any case, we checked the driver logs and it did seem
to pass the packet to the right interface.

We are using custom driver written for io-net using shim. The HAL layer
of the driver is a modified version of QNX porting attempt which we got
from QNX. Mainly however, Broadcom provides a driver library that
controls the chip. Our driver merely makes calls to their API's.

Attached is the code snippet of receive code:

#define MAX_IP_PACKET_SIZE 16384

typedef struct _ipv4_packet_t_ {
    u_int8_t vhlen; //Version and header length.
    u_int8_t tos; //Type of service.
    u_int16_t totlen; //Total length in bytes.
    u_int16_t id; //Identification.
    u_int16_t odmffo; //0,DF,MF and framgment offset fields.
    u_int8_t ttl; //Time to live.
    u_int8_t protocol; //Protocol type in payload.
    u_int16_t chksum; //Checksum.
    struct in_addr source; //Source IPv4 address.
    struct in_addr destination; //Destination IPv4 address.
} ipv4_packet_t;

static u_int8_t __irp_rcvbuffer[MAX_IP_PACKET_SIZE]; /* Receiving buffer
*/

pkt_length = SSAPI_recvfrom( fd, __irp_rcvbuffer, sizeof(
__irp_rcvbuffer ), 0,
                                 ...
View Full Message
Re: RE: RE: RE: Problems with Multicast Receive  
Hi Robert,

Any update on this from your end?

Out io-pkt is not setup to do any forwarding. When we check the TTL on tcpdump we see the TTL to be 1/2 depending on 
what we set it to be. Does seeing tcpdump output mean that the driver layer is passed or does tcpdump just shows that 
the data on the wire is good?

Thanks,
Santosh
> Hi Santosh:
> 
> Well that certainly adds another layer of complexity to things...  So
> the packets are sent out specific interfaces over individual drivers
> which have APIs into the BCM5691 chip.  Packets of interest are all
> received on the management interface, and passed into a separate layer
> which then determines the appropriate io-pkt interface that the packet
> should be forwarded on to....
> 
> Let's go back to the TTL for a minute.  Is io-pkt set up to do
> forwarding at all?  Unless you specifically enable it with a sysctl, no
> forwarding should be happening and the fact that you're seeing a TTL of
> 255 may be an artifact of the way that the packet is being transferred
> out of the stack.
> 
> If you have a single ingress port for all of the packets, can you
> instrument it so that it keeps track of the TTL of all multicast packets
> received?  Any received with a TTL not equal to 1 would suggest that
> something is incorrectly decrementing the TTL.  If all the packets are
> received with the right TTL, then the TTL can't be the issue and you're
> very likely dealing with something going wrong in the BCM driver layer.
> 
> We'll continue trying to duplicate your TTL issue here as well to figure
> out why it shows up as 255 after leaving the stack.
> 
>    R.
> 
> -----Original Message-----
> From: Santosh Kumar [mailto:community-noreply@qnx.com] 
> Sent: Monday, December 01, 2008 11:10 PM
> To: technology-networking
> Subject: Re: RE: RE: Problems with Multicast Receive
> 
> Hi Robert,
> 
> I agree that my real problem is about seeing the packet coming in from a
> wrong interface.
> 
> Im not sure BPF is going to help in my case. Let me explain the setup,
> which is slightly more complicated.
> 
> The BCM5691 has got 12 switching ports and one CPU port (PCI based)
> which is not a part of switching interface but is used purely for
> management. We had a requirement which required us to be able to send
> special packets on a specific switching port and also know which
> switching port the packet came in from. Given the features of BCM5691,
> the way we achieved this is that for these special packets that had
> these requirement, we created an ethernet interface for each of the
> switching port. I modified the io-pkt such that we are able to send out
> packets on a given interface (based on ifindex). These interfaces all
> used the same PCI interface as physical medium.
> 
> We setup the BCM5691 L2 tables such that these special packets are
> passed on to the CPU port and thus would be received by the BCM5691
> driver. The driver knows which interface the packet arrived on and
> passes it to the right ethernet port that io-pkt recognizes.
> 
> Do you think BPF would be useful in this case? I think it would only see
> what the driver passes. So, if the bug is in the driver BPF might not be
> able to match. In any case, we checked the driver logs and it did seem
> to pass the packet to the right interface.
> 
> We are using custom driver written for io-net using shim. The HAL layer
> of the driver is a modified version of QNX porting attempt which we got
> from QNX. Mainly however, Broadcom provides a driver library that
> controls the chip. Our driver merely makes calls to their API's.
> 
> Attached is the code snippet of receive code:
> 
> #define MAX_IP_PACKET_SIZE 16384
> 
> typedef struct...
View Full Message
Problems with Multicast Receive  
Hi Santosh:
	Tcpdump uses the BPF interface which is embedded in the driver.
If you look in the driver receive source, you'll see something like 

if (ifp->if_bpf)
	bpf_mtap(ifp->if_bpf, rm);

This is the place that the packet is tapped out and sent to BPF.

I was hoping to hear back from you on instrumenting the driver. There is
something else that you can try though.  All ethernet packets eventually
go through the ether_input routine in sys/net/if_ethersubr.c  There's a
pfil entry in there that you can use to examine all packets being
received by all interfaces.

Unfortunately, there isn't a generic pfil hook in ether_output (there is
one for each interface, but not for all interfaces)....  I can look at
adding one in if you think it would help to debug your problem.

Here's sample code for how you can get hold of the mbufs immediately
before they enter the stack (laong with the if information).  Install it
under sys/lsm and add in the ppc variant that you need.  You can then
alter the input_hook function to examine the mbuf (and pull out the
ethernet and IP info) and also see which interface the packet came from
(from the passed in ifp param).  If you log the output you can tell
whether or not you ever get a packet with a TTL=255 and probably
correlate the packet you receive with the TTL that you see at the top
level.

	Robert.





-----Original Message-----
From: Santosh Kumar [mailto:community-noreply@qnx.com] 
Sent: Tuesday, December 09, 2008 4:04 AM
To: technology-networking
Subject: Re: RE: RE: RE: Problems with Multicast Receive

Hi Robert,

Any update on this from your end?

Out io-pkt is not setup to do any forwarding. When we check the TTL on
tcpdump we see the TTL to be 1/2 depending on what we set it to be. Does
seeing tcpdump output mean that the driver layer is passed or does
tcpdump just shows that the data on the wire is good?

Thanks,
Santosh
> Hi Santosh:
> 
> Well that certainly adds another layer of complexity to things...  So 
> the packets are sent out specific interfaces over individual drivers 
> which have APIs into the BCM5691 chip.  Packets of interest are all 
> received on the management interface, and passed into a separate layer

> which then determines the appropriate io-pkt interface that the packet

> should be forwarded on to....
> 
> Let's go back to the TTL for a minute.  Is io-pkt set up to do 
> forwarding at all?  Unless you specifically enable it with a sysctl, 
> no forwarding should be happening and the fact that you're seeing a 
> TTL of
> 255 may be an artifact of the way that the packet is being transferred

> out of the stack.
> 
> If you have a single ingress port for all of the packets, can you 
> instrument it so that it keeps track of the TTL of all multicast 
> packets received?  Any received with a TTL not equal to 1 would 
> suggest that something is incorrectly decrementing the TTL.  If all 
> the packets are received with the right TTL, then the TTL can't be the

> issue and you're very likely dealing with something going wrong in the
BCM driver layer.
> 
> We'll continue trying to duplicate your TTL issue here as well to 
> figure out why it shows up as 255 after leaving the stack.
> 
>    R.
> 
> -----Original Message-----
> From: Santosh Kumar [mailto:community-noreply@qnx.com]
> Sent: Monday, December 01, 2008 11:10 PM
> To: technology-networking
> Subject: Re: RE: RE: Problems with Multicast Receive
> 
> Hi Robert,
> 
> I agree that my real problem is about seeing the packet coming in from

> a wrong interface.
> 
> Im not sure BPF is going to help in my case. Let me explain the setup,

> which is slightly more complicated.
> 
> The BCM5691 has got 12 switching ports and one CPU port (PCI based) 
> which is not a part of...
View Full Message
Attachment: Compressed file pf.tar 20 KB