# netstat -g IPv4 Multicast Group Memberships Group Link-layer Address Netif 224.0.0.1 01:00:5e:00:00:01 em0 224.0.0.1 01:00:5e:00:00:01 em3 224.0.0.1 01:00:5e:00:00:01 em7 224.0.0.1 <none> lo0 224.0.0.1 <none> lo1 # /usr/local/sbin/ospfd --vty_addr 127.0.0.1 2007/11/10 11:09:18 OSPF: OSPFd 0.99.9 starting: vty@2604 2007/11/10 11:09:18 OSPF: interface 81.94.118.8 [5] join AllSPFRouters Multicast group. 2007/11/10 11:09:18 OSPF: can't setsockopt IP_ADD_MEMBERSHIP (fd 5, addr 81.94.118.72, ifindex 9, AllSPFRouters): Invalid argument; perhaps a kernel limit on # of multicast group memberships has been exceeded? 2007/11/10 11:09:18 OSPF: *** sendmsg in ospf_write failed to 224.0.0.5, id 0, off 0, len 64, interface em7, mtu 1500: No buffer space available # netstat -g IPv4 Multicast Group Memberships Group Link-layer Address Netif 224.0.0.5 01:00:5e:00:00:05 em0 224.0.0.1 01:00:5e:00:00:01 em0 224.0.0.1 01:00:5e:00:00:01 em3 224.0.0.1 01:00:5e:00:00:01 em7 224.0.0.1 <none> lo0 224.0.0.1 <none> lo1 The interfaces for ospf are em3 and em7 (not em0!) # ifconfig bce0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4> ether 00:1a:a0:15:f3:46 media: Ethernet autoselect (none) status: no carrier em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b7:b0 inet 10.32.104.75 netmask 0xffff0000 broadcast 10.32.255.255 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b7:b1 media: Ethernet autoselect status: no carrier em2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b7:b2 media: Ethernet autoselect status: no carrier em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b7:b3 inet 81.94.118.8 netmask 0xffffffc0 broadcast 81.94.118.63 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active em4: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b3:8c media: Ethernet autoselect status: no carrier em5: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b3:8d media: Ethernet autoselect status: no carrier em6: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b3:8e media: Ethernet autoselect status: no carrier em7: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM> ether 00:0e:0c:c5:b3:8f inet 81.94.118.72 netmask 0xffffffc0 broadcast 81.94.118.127 media: Ethernet autoselect status: no carrier bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4> ether 00:1a:a0:15:f3:44 media: Ethernet autoselect (none) status: no carrier lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 inet 81.94.118.136 netmask 0xffffffff # cat /usr/local/etc ! ! Zebra configuration saved from vty ! 2007/11/09 23:04:36 ! hostname ospfd password zebra log stdout ! ! ! interface em3 ip ospf cost 1000 ip ospf priority 0 ! interface lo1 ip ospf cost 5000 ! router ospf ospf router-id 81.94.118.136 network 81.94.118.0/26 area 0.0.0.0 network 81.94.118.72/26 area 0.0.0.0 network 81.94.118.136/32 area 0.0.0.0 ! line vty # uname -a FreeBSD monitor.as34288.net 7.0-BETA2 FreeBSD 7.0-BETA2 #2: Fri Nov 9 23:35:59 UTC 2007 root@monitor.as34288.net:/usr/src/sys/amd64/compile/monitor amd64
(just cross-checked with openospfd, openospfd bind to the multicast groups correctly on the same system. So we can rule out a FreeBSD problem.)
Same problem on another FreeBSD 7 system
Same problem: FreeBSD mars.oxford1.vtdn.net 7.0-BETA3 FreeBSD 7.0-BETA3 #0: Thu Nov 22 10:39:23 EST 2007 mars.oxford1.vtdn.net:/usr/obj/usr/src/sys/VTDN i386 Using fxp0 and xl0, when attempting to do OSPF over fxp0 (ospfd itself confirms that it joined the mcast group for fxp0) it actually joins the ospf multicast group on xl0.. which is completely the wrong interface.
Can you please post the output of zebra 'show interface' and ospfd 'show ip ospf interface' on a system that is exhibiting this problem? Thanks, Andy
as requested: monitor.as34288.net> show interface Interface bce0 is down index 1 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface bce1 is down index 10 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em0 is up, line protocol detection is disabled index 2 metric 1 mtu 1500 flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> inet 10.32.104.75/16 broadcast 10.32.255.255 input packets 1189, bytes 443397, dropped 0, multicast packets 592 input errors 0 output packets 742, bytes 87104, multicast packets 2 output errors 0 collisions 0 Interface em1 is down index 3 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em2 is down index 4 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em3 is up, line protocol detection is disabled index 5 metric 1 mtu 1500 flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> inet 81.94.118.8/26 broadcast 81.94.118.63 input packets 51, bytes 3060, dropped 0, multicast packets 51 input errors 0 output packets 4, bytes 276, multicast packets 3 output errors 0 collisions 0 Interface em4 is down index 6 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em5 is down index 7 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em6 is down index 8 metric 1 mtu 1500 flags: <BROADCAST,SIMPLEX,MULTICAST> input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 Interface em7 is up, line protocol detection is disabled index 9 metric 1 mtu 1500 flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> inet 81.94.118.136/26 broadcast 81.94.118.191 input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 42, multicast packets 0 output errors 0 collisions 0 Interface lo0 is up, line protocol detection is disabled index 11 metric 1 mtu 16384 flags: <UP,LOOPBACK,RUNNING,MULTICAST> inet 127.0.0.1/8 broadcast 127.0.0.1 inet6 fe80::1/64 inet6 ::1/128 broadcast ::1 input packets 64, bytes 3572, dropped 0, multicast packets 0 input errors 0 output packets 64, bytes 3572, multicast packets 0 output errors 0 collisions 0 Interface lo1 is up, line protocol detection is disabled index 12 metric 1 mtu 16384 flags: <UP,LOOPBACK,RUNNING,MULTICAST> inet 81.94.118.193/32 broadcast 81.94.118.193 input packets 0, bytes 0, dropped 0, multicast packets 0 input errors 0 output packets 0, bytes 0, multicast packets 0 output errors 0 collisions 0 ospfd> sh ip ospf interface bce0 is down ifindex 1, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface bce1 is down ifindex 10, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em0 is up ifindex 2, MTU 1500 bytes, BW 0 Kbit <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> OSPF not enabled on this interface em1 is down ifindex 3, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em2 is down ifindex 4, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em3 is up ifindex 5, MTU 1500 bytes, BW 0 Kbit <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> Internet Address 81.94.118.8/26, Broadcast 81.94.118.63, Area 0.0.0.0 MTU mismatch detection:enabled Router ID 81.94.118.136, Network Type BROADCAST, Cost: 1000 Transmit Delay is 1 sec, State DROther, Priority 0 No designated router on this network No backup designated router on this network Multicast group memberships: OSPFAllRouters Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5 Hello due in 3.925s Neighbor Count is 0, Adjacent neighbor count is 0 em4 is down ifindex 6, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em5 is down ifindex 7, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em6 is down ifindex 8, MTU 1500 bytes, BW 0 Kbit <BROADCAST,SIMPLEX,MULTICAST> OSPF not enabled on this interface em7 is up ifindex 9, MTU 1500 bytes, BW 0 Kbit <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> OSPF not enabled on this interface lo0 is up ifindex 11, MTU 16384 bytes, BW 0 Kbit <UP,LOOPBACK,RUNNING,MULTICAST> OSPF not enabled on this interface lo1 is up ifindex 12, MTU 16384 bytes, BW 0 Kbit <UP,LOOPBACK,RUNNING,MULTICAST> OSPF not enabled on this interface monitor[root]:/usr/ports/net/quagga(117)# netstat -g Link-layer Multicast Group Memberships Group Link-layer Address Netif 01:00:5e:00:00:05 <none> em0 01:00:5e:00:00:01 <none> em0 01:00:5e:00:00:01 <none> em3 01:00:5e:00:00:01 <none> em7 IPv4 Multicast Group Memberships Group Link-layer Address Netif 224.0.0.5 01:00:5e:00:00:05 em0 224.0.0.1 01:00:5e:00:00:01 em0 224.0.0.1 01:00:5e:00:00:01 em3 224.0.0.1 01:00:5e:00:00:01 em7 224.0.0.1 <none> lo0 224.0.0.1 <none> lo1 IPv6 Multicast Group Memberships Group Link-layer Address Netif ff01:b::1 <none> lo0 ff02:b::2:217e:b36c <none> lo0 ff02:b::1 <none> lo0 ff02:b::1:ff00:1 <none> lo0
Changed summary to [FreeBSD 7] because this includes both AMD64 and i386, not specific to one architecture.
Sounds like FreeBSD 7 has changed a system interface. Do we know what the change was? Further was it intentional? (Changing a public kernel/userspace interface is usually a bug).
I've already finished debugging this completely. FreeBSD 7 made some fundamental changes to the multicast processing which Quagga relies on a 'hack' in the kernel to function properly. The specific issue seems to occur when quagga calls IP_MULTICAST_IF with an IP address of 0.0.0.0/8. FreeBSD will default to whatever interface the default route is (usually). Please see comments from other users at: http://www.nabble.com/forum/ViewPost.jtp?post=11172865&framed=y Here's the changelog from FreeBSD source tree: 20070612: The IPv4 multicast socket code has been considerably modified, and moved to the file sys/netinet/in_mcast.c. Initial support for the RFC 3678 Source-Specific Multicast Socket API has been added to the IPv4 network stack. Strict multicast and broadcast reception is now the default for UDP/IPv4 sockets; the net.inet.udp.strict_mcast_mship sysctl variable has now been removed. The RFC 1724 hack for interface selection has been removed; the use of the Linux-derived ip_mreqn structure with IP_MULTICAST_IF has been added to replace it. Consumers such as routed will soon be updated to reflect this. These changes affect users who are running routed(8) or rdisc(8) from the FreeBSD base system on point-to-point or unnumbered interfaces. My solution on my system was to edit lib/sockopt.c and remove the following code: Remove from: #ifdef HAVE_STRUCT_IP_MREQN_IMR_IFINDEX to #endif Remove: #ifdef HAVE_BSD_STRUCT_IP_MREQ_HACK if (ifindex) m.s_addr = htonl(ifindex); else #endif Thus making the complete function setsockopt_multicast_ipv4 looking like: int setsockopt_multicast_ipv4(int sock, int optname, struct in_addr if_addr /* required */, unsigned int mcast_addr, unsigned int ifindex /* optional: if non-zero, may be used instead of if_addr */) { struct in_addr m; struct ip_mreq mreq; int ret; m = if_addr; switch (optname) { case IP_MULTICAST_IF: return setsockopt (sock, IPPROTO_IP, optname, (void *)&m, sizeof(m)); break; case IP_ADD_MEMBERSHIP: case IP_DROP_MEMBERSHIP: memset (&mreq, 0, sizeof(mreq)); mreq.imr_multiaddr.s_addr = mcast_addr; mreq.imr_interface = m; ret = setsockopt (sock, IPPROTO_IP, optname, (void *)&mreq, sizeof(mreq)); if ((ret < 0) && (optname == IP_ADD_MEMBERSHIP) && (errno == EADDRINUSE)) { /* see above: handle possible problem when interface comes back up */ char buf[2][INET_ADDRSTRLEN]; zlog_info("setsockopt_multicast_ipv4 attempting to drop and " "re-add (fd %d, ifaddr %s, mcast %s, ifindex %u)", sock, inet_ntop(AF_INET, &if_addr, buf[0], sizeof(buf[0])), inet_ntop(AF_INET, &mreq.imr_multiaddr, buf[1], sizeof(buf[1])), ifindex); setsockopt (sock, IPPROTO_IP, IP_DROP_MEMBERSHIP, (void *)&mreq, sizeof(mreq)); ret = setsockopt (sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, (void *)&mreq, sizeof(mreq)); } return ret; break; default: /* Can out and give an understandable error */ errno = EINVAL; return -1; break; } }
Quagga already supports ip_mreqn, so the addition of it to FreeBSD will cause Quagga to automatically stop using the old hack and start passing ifindex explicitly with IP_MULTICAST_IF and IP_{ADD,DROP}_MEMBERSHIP. There's one fly in the ointment though, which is that the current mreqn sockopt code isn't working on FreeBSD, it doesn't result in ospfd joining to the correct interfaces. This /seems/ to be due to the fact that, at present in FreeBSD 7 beta, only IP_MULTICAST_IF has been extended to support mreqn. IP_ADD/DROP_MEMBERSHIP still assumes an old ip_mreq. Quagga however is passing an ip_mreqn, leaving the address blank and passing ifindex. FreeBSD get 0 address and thus picks the first capable interface instead, blithely unaware that an ifindex was passed in. I presume it is intended to complete the mreqn support in short order in FreeBSD (for this half-implementation breaks all multicast code that detects mreqn presence at runtime and assumes therefore that it works for ADD/DROP_MEMBERSHIP - which it does on Linux). If that presumption is correct, then this is a temporary bug in FreeBSD and there's nothing to be changed in Quagga.
I think that this is bug in quagga: using ip_mreqn with IP_ADD/DROP_MEMBERSHIP is linux specific hack: rfc 3678 allow only ip_mreq for IP_ADD/DROP_MEMBERSHIP. At the same time qуагга only checks existence of structure ip_mreqn and think that it can be used in IP_ADD/DROP_MEMBERSHIP. I think that the correct solve of this problem is extending quagga configure to check support of rfc 3678 (for example by checking existence MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP constants) and then using MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP instead of IP_ADD_MEMBERSHIP/IP_DROP_MEMBERSHIP I have patch, which change sockopt.c to use MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP : http://levsha.org.ua/sockopt.c.diff But this patch is dirty: i think that patch break working in linux. If it is interesting to community I shall try to realize it correctly: whit checking if MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP is exists on configure time
Hi Mykola! > I think that this is bug in quagga: using ip_mreqn with > IP_ADD/DROP_MEMBERSHIP is linux specific hack: rfc 3678 allow only > ip_mreq for IP_ADD/DROP_MEMBERSHIP. Looking at 4.1.1. of RFC3678 it seems that you are right. Although this hack was appealing to me [1], I prefer your solution, see below. > At the same time quagga only checks existence of structure ip_mreqn > and think that it can be used in IP_ADD/DROP_MEMBERSHIP. Yes, this seems to be a collateral damage of the above mentioned "Linuxism". > I have patch, which change sockopt.c to use > MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP : > http://levsha.org.ua/sockopt.c.diff Thanx. I currently test it and it works fine on yesterdays RELENG_7! > But this patch is dirty: i think that patch break working in linux. If it is > interesting to community I shall try to realize it correctly: whit checking if > MCAST_JOIN_GROUP/MCAST_LEAVE_GROUP is exists on configure time I think that would be the best solution. But it should also work on Linux, at least I've read that Linux supports RFC3678 for some time now. Although it's possible to adapt the FreeBSD 7 kernel to use ip_mreqn with IP_ADD/DROP_MEMBERSHIP [1,2], the FreeBSD developer who implemented RFC3678 in FreeBSD (bms@) wrote that he considers this a step backwards. Nevertheless the presence of ip_mreqn is currently still required on FreeBSD because "The use of ip_mreqn with IP_MULTICAST_IF papers over the glaring crack in the legacy BSD multicast API" (citing bms@). Since the older (RFC1724, 3.3) hack of using 0.0.0.0/8 to encode interface indices was not reintroduced to FreeBSD 8-current, the patch to quagga provided in [3] probably won't work there. To summarize: If you or a quagga developer could fully integrate your RFC3678-compatibility patch into quagga, that would (IMHO) be the favorable solution. If I can help with testing on FreeBSD 7, please let me know. Thanx! Johannes [1] <http://lists.freebsd.org/pipermail/freebsd-current/2007-December/081080.html> [2] <http://lists.freebsd.org/pipermail/freebsd-current/2007-July/074487.html> [3] <http://bugzilla.quagga.net/show_bug.cgi?id=420#c8>
Is there any work being done to make this patch suitable for inclusion with Quagga (e.g. patch configure, make it not break non-RFC3678 systems)? NB: Patches should probably be attached in bugzilla, rather than left on external webservers to avoid losing them.
Created attachment 276 [details] The sockopt patch from the URL in the comments
Created attachment 326 [details] Updated sockopt patch This patch adds a necessary test for HAVE_STRUCT_IP_MREQN_IMR_IFINDEX so that Linux compatibility is maintained.
correction: the test is for HAVE_STRUCT_SOCKADDR_IN_SIN_LEN
A recent commit to RE-testing by Dmitry should have this bug resolved. Gentlemen, could you verify if it resolves the issue on your side? Thank you.