Upon zebra shutdown hash_clean_and_free is called
where user free function is passed,
The free function should not call hash_release
which lead to double free of hash bucket.
Fix:
The fix is to avoid calling hash_release from
free function if its called from hash_clean_and_free
path.
10 0x00007f0422b7df1f in free () from /lib/x86_64-linux-gnu/libc.so.6
11 0x00007f0422edd779 in qfree (mt=0x7f0423047ca0 <MTYPE_HASH_BUCKET>,
ptr=0x55fc8bc81980) at ../lib/memory.c:130
12 0x00007f0422eb97e2 in hash_clean (hash=0x55fc8b979a60,
free_func=0x55fc8a529478 <svd_nh_del_terminate>) at
../lib/hash.c:290
13 0x00007f0422eb98a1 in hash_clean_and_free (hash=0x55fc8a675920
<svd_nh_table>, free_func=0x55fc8a529478 <svd_nh_del_terminate>) at
../lib/hash.c:305
14 0x000055fc8a5323a5 in zebra_vxlan_terminate () at
../zebra/zebra_vxlan.c:6099
15 0x000055fc8a4c9227 in zebra_router_terminate () at
../zebra/zebra_router.c:276
16 0x000055fc8a4413b3 in zebra_finalize (dummy=0x7fffb881c1d0) at
../zebra/main.c:269
17 0x00007f0422f44387 in event_call (thread=0x7fffb881c1d0) at
../lib/event.c:2011
18 0x00007f0422ecb6fa in frr_run (master=0x55fc8b733cb0) at
../lib/libfrr.c:1243
19 0x000055fc8a441987 in main (argc=14, argv=0x7fffb881c4a8) at
../zebra/main.c:584
Signed-off-by: Chirag Shah <chirag@nvidia.com>
For those packets that we are not sending 16k of data, but something
far less than 256 bytes. Reduce those stream sizes we allocate
to something much more reasonable.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
When a MAC gets deleted but associated neighbors remain, the MAC is
kept in the zebra MAC database as an internal ("auto") entry. When
this happens, reset the MAC's remote sequence number. This ensures that
when the host with the MAC later comes up behind a remote VTEP, the
local switch accepts the MAC and installs it into the bridge FDB and
we don't end up in a situation where remote MACs are not installed
into the bridge FDB.
This fix is a corollary of CM-22753 and is this time done for local
MACs upon delete.
Note: Commit is marked Cumulus-only because I need to evalute more
comprehensive changes before upstreaming it.
Ticket: CM-29581
Reviewed By: As above
Testing Done:
1. Multiple rounds of manual testing
2. Two rounds of evpn-smoke, 1 round of precommit
Signed-off-by: Vivek Venkatraman <vivek@cumulusnetworks.com>
Acked-by: Chirag Shah <chirag@cumulusnetworks.com>
Acked-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
Trigger: Zebra core seen when we convert l2vni to l3vni and back
BackTrace:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xe9) [0x7f4af96989d9]
/usr/lib/frr/zebra(zebra_vxlan_if_vni_up+0x250) [0x5561022ae030]
/usr/lib/frr/zebra(netlink_vlan_change+0x2f4) [0x5561021fd354]
/usr/lib/frr/zebra(netlink_parse_info+0xff) [0x55610220d37f]
/usr/lib/frr/zebra(+0xc264a) [0x55610220d64a]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f4af967e96d]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f4af9637588]
/usr/lib/frr/zebra(main+0x402) [0x5561021f4d32]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f4af932624a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f4af9326305]
/usr/lib/frr/zebra(_start+0x21) [0x5561021f72f1]
Root Cause:
In working case,
- We get a RTM_NEWLINK whose ctx is enqueued by zebra dplane and
dequeued by zebra main and processed i.e.
(102000 is deleted from vxlan99) before we handle RTM_NEWVLAN.
- So in handling of NEWVLAN (vxlan99) we bail out since find with
vlan id 703 does not exist.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/working/nocras.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 23:09:33.741105 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 23:09:33.744061 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 23:09:33.767240 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 23:09:33.767380 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (73), result QUEUED
2024/07/18 23:09:33.767389 ZEBRA: [NVFT0-HS1EX] INTF_INSTALL for vxlan99(73)
2024/07/18 23:09:33.767404 ZEBRA: [TQR2A-H2RFY] Vlan-Vni(1186:1186-6000002:6000002) update for VxLAN IF vxlan99(73)
2024/07/18 23:09:33.767422 ZEBRA: [TP4VP-XZ627] Del L2-VNI 102000 intf vxlan99(73)
2024/07/18 23:09:33.767858 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 23:09:33.767866 ZEBRA: [KKZGZ-8PCDW] Cannot find VNI for VID (703) IF vxlan99 for vlan state update >>>>BAIL OUT
In failure case,
- The NEWVLAN is received first even before processing RTM_NEWLINK.
- Since the vxlan id 102000 is not removed from the vxlan99,
the find with vlan id 703 returns the 102000 one and we invoke
zebra_vxlan_if_vni_up where the interfaces don't match and assert.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/noworking/crash.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 22:26:43.829370 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 22:26:43.829646 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7fe07c026d30, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 22:26:43.853930 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 22:26:43.853949 ZEBRA: [K61WJ-XQQ3X] Intf vxlan99(73) L2-VNI 102000 is UP >>> VLAN PROCESSED BEFORE INTF EVENT
2024/07/18 22:26:43.853951 ZEBRA: [SPV50-BX2RP] RAJA zevpn_vxlanif vxlan48 and ifp vxlan99
2024/07/18 22:26:43.854005 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 22:26:43.854241 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=516, seq=0, pid=0
2024/07/18 22:26:43.854251 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=544, seq=0, pid=0
ZEBRA: in thread kernel_read scheduled from zebra/kernel_netlink.c:505 kernel_read()
Fix:
Similar to #13396, where link change
handling was offloaded to dplane, same is being done for vlan events.
Note: Prior to this change, zebra main thread was interested in the
RTNLGRP_BRVLAN. So all the kernel events pertaining to vlan was
handled by zebra main.
With this change change as well the handling of vlan events is still
with Zebra main. However we offload it via Dplane thread.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
Relocating functions used by vlan in if_netlink into zebra vxlan
Note: Static variable to the functions will be added back in the next
commit.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
In L3 BGP-EVPN, if there are both IPv4 and IPv6 routes in the VPN, zebra
maintains two instances of `struct zebra_neigh` object: one with IPv4
address of the nexthop, and another with IPv6 address that is an IPv4
mapped to IPv6, but only one intance of `struct zebra_mac` object, that
contains a list of nexthop addresses that use this mac.
The code in `zebra_vxlan` module uses the fact that the list is empty as
the indication that the `zebra_mac` object is unused, and needs to be
dropped. However, preexisting code used nexthop address converted to
IPv4 notation for the element of this list. As a result, when two
`zebra_neigh` objects, one IPv4 and one IPv6-mapped-IPv4 were linked to
the `zebra_mac` object, only one element was added to the list.
Consequently, when one of the two `zebra_neigh` objects was dropped, the
only element in the list was removed, making it empty, and `zebra_mac`
object was dropped, and neigbrour cache elements uninstalled from the
kernel.
As a result, after the last route in _one_ family was removed from a
remote vtep, all remaining routes in the _other_ family became
unreachable, because RMAC of the vtep was removed.
This commit makes `zebra_mac` use uncoerced IP address of the `zebra_neigh`
object for the entries in the `nh_list`. This way, `zebra_mac` object no
longer loses track of `zebra_neigh` objects that need it.
Bug-URL: https://github.com/FRRouting/frr/issues/16340
Signed-off-by: Eugene Crosser <crosser@average.org>
In the context of SVD (Single VxLAN Device) for L3VNI,
the remote VTEP's nexthop is programmed neighbor entry against
SVD along with neighbor entry against SVI.
However, when L3VNI is removed or the VRF is disabled, all SVI
based remote nexthop neighbors are uninstalled and deleted.
The SVD based neigh entries remains in Zebra and the Kernel.
Subsequently, when reconfiguring L3VNI and relearning the same nexthop,
the neighbor entry is not programmed is because it is not removed
from Zebra SVD neighbor hash table, leading to the failure to
reprogram the entry.
With this fix, the SVD nexthop neigh entry is uninstalled
and deleted from Zebra and Kernel.
Ticket: #3729045
Testing:
borderleaf:# ip neigh show 2.2.2.2
2.2.2.2 dev vlan2560_l3 lladdr 00:01:00:00:1d:09 extern_learn NOARP proto zebra
2.2.2.2 dev vxlan99 lladdr 00:01:00:00:1d:09 extern_learn NOARP proto zebra
With the fix:
Zebra log shows both enties SVD (vxlan99) and SVI (vlan2560_l3)
neighbor entries are deleted.
2024/05/03 18:41:33.527125 ZEBRA: [NH6N7-54CD1] Tx RTM_DELNEIGH family
ipv4 IF vxlan99(16) Neigh 2.2.2.2 MAC null flags 0x10 state 0x0
ext_flags 0x0
2024/05/03 18:41:33.527128 ZEBRA: [NH6N7-54CD1] Tx RTM_DELNEIGH family
ipv4 IF vlan2560_l3(18) Neigh 2.2.2.2 MAC null flags 0x10 state 0x0
ext_flags 0x0
borderleaf:# ip neigh show 2.2.2.2
borderleaf:#
Signed-off-by: Chirag Shah <chirag@nvidia.com>
A macvlan interface can have its underlying link-interface in another
namespace (aka. netns). However, by default, zebra does not know the
interface from the other namespaces. It results in a crash the pointer
to the link interface is NULL.
> 6 0x0000559d77a329d3 in zebra_vxlan_macvlan_up (ifp=0x559d798b8e00) at /root/frr/zebra/zebra_vxlan.c:4676
> 4676 link_zif = link_ifp->info;
> (gdb) list
> 4671 struct interface *link_ifp, *link_if;
> 4672
> 4673 zif = ifp->info;
> 4674 assert(zif);
> 4675 link_ifp = zif->link;
> 4676 link_zif = link_ifp->info;
> 4677 assert(link_zif);
> 4678
> (gdb) p zif->link
> $2 = (struct interface *) 0x0
> (gdb) p zif->link_ifindex
> $3 = 15
Fix the crash by returning when the macvlan link-interface is in another
namespace. No need to go further because any vxlan under the macvlan
interface would not be accessible by zebra.
Link: https://github.com/FRRouting/frr/issues/15370
Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
- unnecessary command duplication
- usage of oper data during validation
- unnecessary checks for things that can't happen
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
Whenever a link up change was detected on a macvlan device where
the linked device wasn't visible in the namespace zebra was
running in, the linked zebra interface was NULL. This was already
handled in the event of a link down, but was ommitted from the
upside. Added the same null check to the up-side.
Signed-off-by: Tomi Salminen <tlsalmin@gmail.com>
This is addressing remaining places returning
empty dict, earlier PR-13214 addressed few places.
Code has been changed to return {} for all the evpn clis
when evpn is disabled or no entry available.
```
cumulus@r2:mgmt:~$ sudo vtysh -c "show evpn json"
cumulus@r2:mgmt:~$
```
After Fix:-
```
cumulus@r1:mgmt:~$ sudo vtysh -c "show evpn json"
{
}
cumulus@r1:mgmt:~$
```
Ticket:#3417955
Issue:3417955
Testing: UT done
Signed-off-by: Sindhu Parvathi Gopinathan's <sgopinathan@nvidia.com>
Signed-off-by: Chirag Shah <chirag@nvidia.com>
Currently, json output of evpn route command are no pretty format.
This is an extremely expensive operation at high VNI scale
EVPN json non-pretty command support added:
```
show evpn mac vni <vni-id> detail json
show evpn vni detail json
```
Ticket:#3513256
Issue:3513256
Testing: UT done
Signed-off-by: Sindhu Parvathi Gopinathan's <sgopinathan@nvidia.com>
When route_next return node, it has lock the node. if return or break loop, should unlock node.
Signed-off-by: guozhongfeng <guozhongfeng.gzf@alibaba-inc.com>
The default vrf is generally non-NULL, except when shutdown. So, most
of the time it is not necessary to check if it is NULL, we should
remove the useless checks for it.
Searched them with exact match:
```
grep -rI "zebra_vrf_lookup_by_id(VRF_DEFAULT)" | wc -l
31
```
Signed-off-by: anlan_cs <vic.lan@pica8.com>
EVPN RMAC (Router MAC) nexthop list compare
function needs to return all values so
the list element can be compared and added/deleted
properly.
Ticket:#3486989
Testing Done:
Originate EVPN Type-5 route with PIP IP and MAC as remote
nexthops.
Change the PIP IP address which triggers nexthop change.
Before fix:
When PIP IP changes RMAC is deleted from remote VTEPs.
TORS1# show evpn next-hops vni 4001 | include 00:02:00:00:00:2d
27.0.0.11 00:02:00:00:00:2d
TORS1# show evpn rmac vni 4001 | include 00:02:00:00:00:2d
00:02:00:00:00:2d 27.0.0.11
----- Remote VTEP change nexthop IP to 172.16.16.16 -----
TORS1# show evpn next-hops vni 4001 | include 00:02:00:00:00:2d
172.16.16.16 00:02:00:00:00:2d
TORS1# show evpn rmac vni 4001 | include 00:02:00:00:00:2d
TORS1#
After fix:
RMAC is retained as its nexthop list is not empty,
thus it is not deleted from remote VTEPs.
TORS1# show evpn rmac vni 4001 | include 00:02:00:00:00:2d
00:02:00:00:00:2d 172.16.16.16
Log:
2023/06/27 00:50:36.833474 ZEBRA: [XREH0-ZYMH6] L3VNI 4001 Remote VTEP
change(27.0.0.11 -> 172.16.16.16) for RMAC 00:02:00:00:00:2d
Signed-off-by: Chirag Shah <chirag@nvidia.com>
"show evpn json" returns nothing when evpn is disabled.
Code has been fixed to return {} when evpn is disabled or no entry
available.
Before Fix:-
```
cumulus@r2:mgmt:~$ sudo vtysh -c "show evpn json"
cumulus@r2:mgmt:~$
```
After Fix:-
```
cumulus@r1:mgmt:~$ sudo vtysh -c "show evpn json"
{
}
cumulus@r1:mgmt:~$
```
Ticket:#3417955
Issue:3417955
Testing: UT done
Signed-off-by: Chirag Shah <chirag@nvidia.com>
Signed-off-by: Sindhu Parvathi Gopinathan <sgopinathan@nvidia.com>
The `show evpn next-hop svd *` command doesn't provide much
for users right now. Make it hidden so we can still debug
the tables with it.
Also remove SVD output from `show evpn next-hop vni all`.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Add some show commands and expand some already existing
commands so we can get debug info from the SVD global
neigh table inside zebra.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Install neigh entries always on SVD if it exists in
zebra. If zebra is using a Single Vxlan Device, we must
duplicate the install of our neigh entries to it so that
vxlan communication can also work across it in the downstream VNI
case.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
This patch addresses following issues,
- When the VLAN-VNI mapping is configured via a map and not using
individual VXLAN interfaces, upon removal of a VNI ensure that the
remote FDB entries are uninstalled correctly.
- When VNI configuration is performed using VLAN-VNI mapping (i.e., without
individual VXLAN interfaces) and flooded traffic is handled via multicast,
the multicast group corresponding to the VNI needs to be explicitly read
from the bridge FDB. This is relevant in the case of netlink interface to
the kernel and for the scenario where a new VNI is provisioned or comes up.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This patch addresses following
- Remove unused VLAN Id parameter when trying to determine the VNI associated
with a non-VLAN aware bridge. Also, add a check to ensure that in this case,
we have a per-VNI VXLAN interface. Due to sequence of events, it is possible
that we may have VLAN-VNI mappings, in which case the code should return
gracefully.
- With support for a container VXLAN interface that has VLAN-VNI mappings,
the VXLAN interface itself may be up but a particular VNI might have
been removed. Ensure that VNI mapping exists before proceeding with
further processing.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
Today to find the vni for a given (vlan, bridge) we walk over all interfaces
and filter the vxlan device associated with the bridge. With multiple vlan aware
bridge changes, we can derive the vni directly by looking up the hash table i.e.
the vlan_table of the associated (vlan, bridge) which would give the vni.
During vrf_terminate() call zebra_l2_bridge_if_cleanup if the interface
that we are removing is of type bridge. In this case, we walk over all
the vlan<->access_bd association and clean them up.
zebra_evpn_t is modified to record (vlan, bridge) details and the
corresponding vty is modified to print the same.
zevpn_bridge_if_set and zl3vni_bridge_if_set is used to set/unset the
association.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This change brings in following functionality
- netlink_bridge_vxlan_vlan_vni_map_update for single vxlan devices
This function is responsible for reading the vlan-vni map information
received from netlink and populating a new hash_table with the vlan-vni
data. Once all the vlan-vni data is collected, zebra_vxlan_if_vni_table_add_update
is called to update vni_table in vxlan interface and process each of the
vlan-vni data.
- refactoring changes for zevpn_build_hash_table
- existing zevpn_build_hash_table was walking over all the vxlan interfaces
and then processing the vni for each of them. In case of single vxlan device,
we will have more than one vni entries. This function is abstracted so that
it iterates over all the vni entries for single vxlan device. For traditional
vxlan device the zebra_vxlan_if_vni_iterate would only process single vni
associated with that device.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This change refactors the zebra_vxlan_if related functionality
to a new zebra_vxlan_if.c file. zebra_vxlan_if_up/down,
zebra_vxlan_if_add/update/del is moved zebra_vxlan_if.c
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
dplane_mac_info and dplane_neigh_info is modified to be vni aware.
dplane_rem_mac_add/del dplane_mac_init is modified to be vni aware.
During dplane context update (mac and neigh), we use the vni information
and if set, corresponding netlink attribute NDA_SRC_VNI is set and passed to the
dplane.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This change set introduces data structure changes required for multiple vlan aware bridge
functionality. A new structure zebra_l2_bridge_if encapsulates the vlan to access_bd
association of the bridge. A vlan_table hash_table is used to record each instance
of the vlan to access_bd of the bridge via zebra_l2_bridge_vlan structure.
vxlan iftype derivation: netlink attribute IFLA_VXLAN_COLLECT_METADATA is used
to derive the iftype of the vxlan device. If the attribute is present, then the
vxlan interface is treated as single vxlan device, otherwise it would default to
traditional vxlan device.
zebra_vxlan_check_readd_vtep, zebra_vxlan_dp_network_mac_add/del is modified to
be vni aware.
mac_fdb_read_for_bridge - is modified to be (vlan, bridge) aware
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This changeset introduces the data structure changes needed for
single vxlan device functionality. A new struct zebra_vxlan_vni_info
encodes the iftype and vni information for vxlan device.
The change addresses related access changes of the new data structure
fields from different files
zebra_vty is modified to take care of the vni dump information according
to the new vni data structure for vxlan devices.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
