Trigger: Zebra core seen when we convert l2vni to l3vni and back
BackTrace:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xe9) [0x7f4af96989d9]
/usr/lib/frr/zebra(zebra_vxlan_if_vni_up+0x250) [0x5561022ae030]
/usr/lib/frr/zebra(netlink_vlan_change+0x2f4) [0x5561021fd354]
/usr/lib/frr/zebra(netlink_parse_info+0xff) [0x55610220d37f]
/usr/lib/frr/zebra(+0xc264a) [0x55610220d64a]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f4af967e96d]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f4af9637588]
/usr/lib/frr/zebra(main+0x402) [0x5561021f4d32]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f4af932624a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f4af9326305]
/usr/lib/frr/zebra(_start+0x21) [0x5561021f72f1]
Root Cause:
In working case,
- We get a RTM_NEWLINK whose ctx is enqueued by zebra dplane and
dequeued by zebra main and processed i.e.
(102000 is deleted from vxlan99) before we handle RTM_NEWVLAN.
- So in handling of NEWVLAN (vxlan99) we bail out since find with
vlan id 703 does not exist.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/working/nocras.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 23:09:33.741105 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 23:09:33.744061 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 23:09:33.767240 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 23:09:33.767380 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (73), result QUEUED
2024/07/18 23:09:33.767389 ZEBRA: [NVFT0-HS1EX] INTF_INSTALL for vxlan99(73)
2024/07/18 23:09:33.767404 ZEBRA: [TQR2A-H2RFY] Vlan-Vni(1186:1186-6000002:6000002) update for VxLAN IF vxlan99(73)
2024/07/18 23:09:33.767422 ZEBRA: [TP4VP-XZ627] Del L2-VNI 102000 intf vxlan99(73)
2024/07/18 23:09:33.767858 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 23:09:33.767866 ZEBRA: [KKZGZ-8PCDW] Cannot find VNI for VID (703) IF vxlan99 for vlan state update >>>>BAIL OUT
In failure case,
- The NEWVLAN is received first even before processing RTM_NEWLINK.
- Since the vxlan id 102000 is not removed from the vxlan99,
the find with vlan id 703 returns the 102000 one and we invoke
zebra_vxlan_if_vni_up where the interfaces don't match and assert.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/noworking/crash.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 22:26:43.829370 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 22:26:43.829646 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7fe07c026d30, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 22:26:43.853930 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 22:26:43.853949 ZEBRA: [K61WJ-XQQ3X] Intf vxlan99(73) L2-VNI 102000 is UP >>> VLAN PROCESSED BEFORE INTF EVENT
2024/07/18 22:26:43.853951 ZEBRA: [SPV50-BX2RP] RAJA zevpn_vxlanif vxlan48 and ifp vxlan99
2024/07/18 22:26:43.854005 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 22:26:43.854241 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=516, seq=0, pid=0
2024/07/18 22:26:43.854251 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=544, seq=0, pid=0
ZEBRA: in thread kernel_read scheduled from zebra/kernel_netlink.c:505 kernel_read()
Fix:
Similar to #13396, where link change
handling was offloaded to dplane, same is being done for vlan events.
Note: Prior to this change, zebra main thread was interested in the
RTNLGRP_BRVLAN. So all the kernel events pertaining to vlan was
handled by zebra main.
With this change change as well the handling of vlan events is still
with Zebra main. However we offload it via Dplane thread.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
Relocating functions used by vlan in if_netlink into zebra vxlan
Note: Static variable to the functions will be added back in the next
commit.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
- unnecessary command duplication
- usage of oper data during validation
- unnecessary checks for things that can't happen
Signed-off-by: Igor Ryzhov <iryzhov@nfware.com>
Ticket: 2674793
Testing Done: precommit, evpn-min and evpn-smoke
The problem in this case is whenever we are triggering ifdown
followed by ifup of bridge, we see that remote mac entries
are programmed with vlan-1 in the fdb from zebra and never cleaned up.
bridge has vlan_default_pvid 1 which means any port that gets added
will initially have vlan 1 which then gets deleted by ifupdown2 and
the proper vlan gets added.
The problem lies in zebra where we are not cleaning up the remote
macs during vlan change.
Fix is to uninstall the remote macs and then install them
during vlan change.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Add some show commands and expand some already existing
commands so we can get debug info from the SVD global
neigh table inside zebra.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Install neigh entries always on SVD if it exists in
zebra. If zebra is using a Single Vxlan Device, we must
duplicate the install of our neigh entries to it so that
vxlan communication can also work across it in the downstream VNI
case.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
This change modifies zebra_vxlan_if_up/down/add/update and del functionality
to be per vni based.
zebra_vxlan_if_add/update/del and zebra_vxlan_if_up/down now handles
the vni operations based on vxlan device type (single or traditional vxlan device).
zebra_vxlan_if_vni_table_add_update
- This function handles the vlan-vni map update received from the netlink
interface to single vxlan device vni_table hash table.
zebra_vxlan_if_vni_mcast_group_update
- This function handles the new multicast group update received from
the netlink interface to single vxlan device vni_table hash table.
For traditional vxlan interfaces, the vni and mcast group
handling follows the traditional approach.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This change set introduces data structure changes required for multiple vlan aware bridge
functionality. A new structure zebra_l2_bridge_if encapsulates the vlan to access_bd
association of the bridge. A vlan_table hash_table is used to record each instance
of the vlan to access_bd of the bridge via zebra_l2_bridge_vlan structure.
vxlan iftype derivation: netlink attribute IFLA_VXLAN_COLLECT_METADATA is used
to derive the iftype of the vxlan device. If the attribute is present, then the
vxlan interface is treated as single vxlan device, otherwise it would default to
traditional vxlan device.
zebra_vxlan_check_readd_vtep, zebra_vxlan_dp_network_mac_add/del is modified to
be vni aware.
mac_fdb_read_for_bridge - is modified to be (vlan, bridge) aware
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
This changeset introduces the data structure changes needed for
single vxlan device functionality. A new struct zebra_vxlan_vni_info
encodes the iftype and vni information for vxlan device.
The change addresses related access changes of the new data structure
fields from different files
zebra_vty is modified to take care of the vni dump information according
to the new vni data structure for vxlan devices.
Signed-off-by: Sharath Ramamurthy <sramamurthy@nvidia.com>
New show command "show evpn mac vni xx detail [json]"
to display details of all the mac entries for the
requested VNI.
Output of show evpn mac vni xx detail json:
{
"numMacs":2,
"macs":{
"ca:be:63:7c:81:05":{
"type":"local",
"intf":"veth100",
"ifindex":8,
"uptime":"00:06:55",
"localSequence":0,
"remoteSequence":0,
"detectionCount":0,
"isDuplicate":false,
"syncNeighCount":0,
"neighbors":{
"active":[
"fe80::c8be:63ff:fe7c:8105"
],
"inactive":[
]
}
}
}
}
Also added remoteEs field in the JSON output of
"show evpn mac vni xx json".
Output of show evpn mac vni xx json:
"00:02:00:00:00:0d":{
"type":"remote",
"remoteEs":"03:44:38:39:ff:ff:02:00:00:02",
"localSequence":0,
"remoteSequence":0,
"detectionCount":0,
"isDuplicate":false
}
Signed-off-by: Pooja Jagadeesh Doijode <pdoijode@nvidia.com>
Use "get" as the name for checking the status of the bgp
accept lower seq knob. This already has an equivalent "set"
so makes sense to keep it consistent.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Add a knob to accept lower seq number in evpn updates
from BGP in Zebra.
Note: Knob is enabled by default
Signed-off-by: Stephen Worley <sworley@nvidia.com>
The global vrf in zebra is always non-NULL. In general, it is bound to
default vrf by `zebra_vrf_init()`, at other times bound to some specific
vrf. Anyway, non-NULL.
So remove all redundant checkings for the returned value of
`zebra_vrf_get_evpn()`.
Additionally, remove the unnecessary check for `zvrf` in
`zebra_vxlan_cleanup_tables()`.
Signed-off-by: anlan_cs <vic.lan@pica8.com>
when running bgp evpn rt5 setup, the Rmac sent in BGP updates
stands for the MAC address of the bridge interface. After
having loaded frr configuration, the Rmac address is not refreshed.
This issue can be easily reproduced by executing some commands:
ip netns exec cust1 ip link set dev br1000 address 2e🆎45:aa:bb:cc
Actually, the BGP EVPN contexts are kept unchanged.
That commit proposes to fix this by intercepting the mac address
change, and refreshing the vxlan interfaces attached to te bridge
interface that changed its MAC address.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
Move remote VTEP updates from immediate, inline processing
in their ZAPI message handlers to the main workqueue.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
When a local ES flaps there are two modes in which the local
MACs are failed over -
1. Fast failover - A backup NHG (ES-peer group) is programmed in the
dataplane per-access port. When a local ES flaps the MAC entries
are left unaltered i.e. pointing to the down access port. And the
dataplane redirects traffic destined to the oper-down access port
via the backup NHG.
2. Slow failover - This mode needs to be turned on to allow dataplanes
not capable of re-directing traffic. In this mode local MAC entries
on a down local ES are re-programmed to point to the ES-peers'
NHG. And vice-versa i.e. when the ES comes up the MAC entries
are re-programmed with the access port as dest.
Fast failover is on by default. Slow failover can be enabled via the
following config -
evpn mh redirect-off
Signed-off-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
Zebra's clear duplicate detect command is rpc converted.
There is condition where cli fails with human readable message.
Using northboun's errmsg buffer to display error message to
user.
Testing:
bharat# clear evpn dup-addr vni 1002 ip 2011:11::11
Error type: generic error
Error description: Requested IP's associated MAC aa:aa:aa:aa:aa:aa is still in duplicate state
bharat# clear evpn dup-addr vni 1002 ip 11.11.11.11
Error type: generic error
Error description: Requested IP's associated MAC aa:aa:aa:aa:aa:aa is still in duplicate state
Signed-off-by: Chirag Shah <chirag@nvidia.com>
MAC-IP routes are used for syncing local entries across redundant
switches in an EVPN-MH setup. A path from a peer that has a local
ES as destination is tagged as a SYNC path. The SYNC path results in the
addition of local MAC and/or local neigh entry in zebra and in the
dataplane.
Implementation overview
=======================
1. Three new flags "local-inactive", "peer-active" and "peer-proxy"
are maintained per-local-MAC and per-local-Neigh entry.
2. The "peer-XXX" flags are set and cleared via SYNC path updates
from BGP. Proxy sync paths result in the setting of "peer-proxy" flag
(and non-proxies result in the "peer-active").
3. A neigh entry that has a "peer-XXX" flag set is programmed as
"static" in the dataplane.
4. A MAC entry that has a "peer-XXX" flag set or is referenced by
a sync-neigh entry (that has a "peer-XXX" flags set) is programmed
as "static" in the dataplane.
5. The sync-seq number is used to normalize the MM seq number across
all the redundant switches i.e. the max MM seq number across all
switches is used by each of the switches. This commit also includes
the changes needed for extended MM seq syncing.
6. A MAC/neigh entry has to be local-active or peer-active to sent to
BGP. An entry that is NOT local-active is sent with the proxy flag (so
BGP can "proxy" advertise it).
7. The "peer-active" flag is aged out by zebra by using a hold_timer
(this is instead of being abruptly dropped on SYNC path delete). This
age-out is needed to handle peer-switch restart (procedures are specified
in draft-rbickhart-evpn-ip-mac-proxy-adv). The holdtime needs to be
sufficiently long to allow an external neighmgr daemon or the dataplane
component to independently probe and establish local reachability of a
host. The MAC and neigh hold time values are configurable.
PS: In the future this probing may happen in FRR itself.
CLI changes to display sync info
================================
MAC
===
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
root@torm-11:mgmt:~# net show evpn mac vni 1000
Number of MACs (local and remote) known for this VNI: 6
Flags: N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
00:02:00:00:00:25 local vlan1000 1000 0/0
02:02:00:00:00:02 local PI hostbond1 1000 0/0
02:02:00:00:00:06 remote 03:00:00:00:00:02:11:00:00:01 0/0
02:02:00:00:00:01 local X hostbond1 1000 0/0
00:00:00:00:00:11 local PI hostbond1 1000 0/0
02:02:00:00:00:05 remote 03:00:00:00:00:02:11:00:00:01 0/0
root@torm-11:mgmt:~#
root@torm-11:mgmt:~# net show evpn mac vni 1000 mac 00:00:00:00:00:11
MAC: 00:00:00:00:00:11
ESI: 03:00:00:00:00:01:11:00:00:01
Intf: hostbond1(58) VLAN: 1000
Sync-info: neigh#: 0 local-inactive peer-active >>>>>>>>>>>>
Local Seq: 0 Remote Seq: 0
Neighbors:
No Neighbors
root@torm-11:mgmt:~#
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
neigh
=====
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
root@torm-11:mgmt:~# net show evpn arp vni 1003
Number of ARPs (local and remote) known for this VNI: 4
Flags: I=local-inactive, P=peer-active, X=peer-proxy
Neighbor Type Flags State MAC Remote ES/VTEP Seq #'s
2001:fee1:0:3::6 local active 00:02:00:00:00:25 0/0
45.0.3.66 local P active 00:02:00:00:00:66 0/0
45.0.3.6 local active 00:02:00:00:00:25 0/0
fe80::202:ff:fe00:25 local active 00:02:00:00:00:25 0/0
root@torm-11:mgmt:~#
root@torm-11:mgmt:~# net show evpn arp vni 1003 ip 45.0.3.66
IP: 45.0.3.66
Type: local
State: active
MAC: 00:02:00:00:00:66
Sync-info: peer-active >>>>>>>>>>>>>>>>
Local Seq: 0 Remote Seq: 0
root@torm-11:mgmt:~#
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Signed-off-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
1. Local ethernet segments are configured in zebra by attaching a
local-es-id and sys-mac to a access interface -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
!
interface hostbond1
evpn mh es-id 1
evpn mh es-sys-mac 00:00:00:00:01:11
!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
This info is then sent to BGP and used for the generation of EAD-per-ES
routes.
2. Access VLANs associated with an (ES) access port are translated into
ES-EVI objects and sent to BGP. This is used by BGP for the
generation of EAD-EVI routes.
3. Remote ESs are imported by BGP and sent to zebra. A list of VTEPs
is maintained per-remote ES in zebra. This list is used for the creation
of the L2-NHG that is used for forwarding traffic.
4. MAC entries with a non-zero ESI destination use the L2-NHG associated
with the ESI for forwarding traffic over the VxLAN overlay.
Please see zebra_evpn_mh.h for the datastruct organization details.
Signed-off-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
Use const with some args to ipaddr, zebra vxlan, mpls
lsp, and nexthop apis; add some extra checks to some
nexthop-related apis.
Signed-off-by: Mark Stapp <mjs@voltanet.io>
Readd the special MAC that represents the flood (head-end replication) entry
for EVPN-VxLAN upon getting a delete notification for it.
Signed-off-by: Vivek Venkatraman <vivek@cumulusnetworks.com>
Reviewed-by: Chirag Shah <chirag@cumulusnetworks.com>
Ticket: CM-25797
Ticket: CM-26238
Testing Done:
1. evpn-min, evpn-smoke - results summarized in CM-25798
macvlan interface up/down event triggers
bgp to send updates for evpn routes
with changed RMAC and nexthop IP values.
Ticket:CM-26190
Reviewed By:
Testing Done:
Signed-off-by: Chirag Shah <chirag@cumulusnetworks.com>
edge-2> show evpn vni detail json
{
"vni":79031,
"type":"L3",
...,
...
} <<<<<< no comma
{
"vni":79021,
"type":"L3",
...,
...
} <<<<<< no comma
{
} <<<<<< blank
edge-2>
The fix is to pack json info into json_array before printing it.
Signed-off-by: Lakshman Krishnamoorthy <lkrishnamoor@vmware.com>
zvni setup in zebra is controlled via bgpd i.e. advertise_all_vni
from bgpd triggers this setup. As a part of zvni creation we may need
to setup BUM mcast SG entries which are propagated to pimd for MDT setup.
Now pimd may not be present at the time of zvni creation or may restart
post zvni creation so we need a mechanism to replay (on pimd startup) and
to cleanup (on pimd stop). This is addressed via zebra_vxlan_sg_replay and
zebra_evpn_pim_cfg_clean_up.
Signed-off-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
When BGP daemon is down, Clean up its configuration state from zebra.
When the BGP daemon is up again, it will push its configuration to zebra
Delete the MAC and neighbor information received on the BGP session,
while retaining the local MAC and local ARP entries.
Signed-off-by: Kishore Aramalla karamalla@vmware.com
The multicast group ip address for BUM traffic is configurable per-l2-vni.
One way to configure that is to setup a vxlan device that per-l2-vni and
specify the address against that vxlan device -
root@TORS1:~# vtysh -c "show interface vx-1000" |grep -i vxlan
Interface Type Vxlan
VxLAN Id 1000 VTEP IP: 27.0.0.15 Access VLAN Id 1000 Mcast 239.1.1.100
root@TORS1:~# vtysh -c "show evpn vni 1000" |grep Mcast
Mcast group: 239.1.1.100
root@TORS1:~#
Signed-off-by: Anuradha Karuppiah <anuradhak@cumulusnetworks.com>
It had no logical reason to be in the default VRF. This moves it to the
zebra_router, which is better suited to store global references.
Signed-off-by: Tuetuopay <tuetuopay@me.com>
Sponsored-by: Scaleway
This replaces manual checks of the flag with a wrapper macro to convey
the meaning "is evpn enabled on this vrf?"
Signed-off-by: Tuetuopay <tuetuopay@me.com>
Sponsored-by: Scaleway
The EVPN VRF is defined by bgpd, and is the one vrf where
`advertise-all-vni` is present.
Signed-off-by: Tuetuopay <tuetuopay@me.com>
Sponsored-by: Scaleway
In Asymmetric and symetric routing scenario in EVPN
where each VTEP pair having different set of addresses
for the SVIs.
This knob allows reachability (ping connectivity) of
SVI IPs and resolve ARP resoultion VTEPs across racks.
This knob should not be used when same SVI IPs configured
on VTEPs across racks or when advertise default gateway
is configured.
Ticket:CM-23782
Testing Done:
Bring up EVPN symmetric routing topology with different
SVI IPs on different VTEPs. Enable advertise svi ip
at each VTEP, remote VTEPs installs arp entry for
SVI IPs via EVPN type-2 route exchange.
Signed-off-by: Chirag Shah <chirag@cumulusnetworks.com>
the l3vni structure is allocated only once, since that structure is only
used for default netns. For that, move the initialisation part is moved
to a proper place, where there is no risk of attempting to initialise it
more than once, even when vrf backend is netns.
Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>
Clear dup address vni needs to return non-zero value
in case of command is not successful.
Ticket:CM-23122
Testing Done:
run clear command and check upon failure return code is non-zero.
root@TORS1:~# vtysh -c "clear evpn dup-addr vni 1000 ip 45.0.1.26"
% Requested IP's associated MAC 00:01:02:03:04:05 is still in duplicate
% state
root@TORS1:~# echo $?
1
Signed-off-by: Chirag Shah <chirag@cumulusnetworks.com>
Change helps display detailed output for all possible VNI neighbors
without specifying VNI and ip. It helps in troubleshooting as a single
command can be fired to capture detailed info on all VNIs.
Ticket: CM-22832
Signed-off-by: Nitin Soni <nsoni@cumulusnetworks.com>
Reviewed-by: CCR-8034
Change helps display detailed output for all possible VNI MACs without
specifying VNI or mac. It helps in troubleshooting - a single
command can be fired to capture detailed info on all VNIs.
Also fixed and existing json related bug where json object is created by
a parent function and freed in child function.
Ticket: CM-22832
Signed-off-by: Nitin Soni <nsoni@cumulusnetworks.com>
Reviewed-by: CCR-8028
Change helps display detailed output for all possible VNIs without
specifying VNI. It helps in troubleshooting - a single command can
be fired to capture detailed info on all VNIs.
Ticket: CM-22831
Signed-off-by: Nitin Soni <nsoni@cumulusnetworks.com>
Reviewed-by: CCR-8013
