0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=130719886083648) at ./nptl/pthread_kill.c:44
1 __pthread_kill_internal (signo=6, threadid=130719886083648) at ./nptl/pthread_kill.c:78
2 __GI___pthread_kill (threadid=130719886083648, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
3 0x000076e399e42476 in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
4 0x000076e39a34f950 in core_handler (signo=6, siginfo=0x76e3985fca30, context=0x76e3985fc900) at lib/sigevent.c:258
5 <signal handler called>
6 __pthread_kill_implementation (no_tid=0, signo=6, threadid=130719886083648) at ./nptl/pthread_kill.c:44
7 __pthread_kill_internal (signo=6, threadid=130719886083648) at ./nptl/pthread_kill.c:78
8 __GI___pthread_kill (threadid=130719886083648, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
9 0x000076e399e42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
10 0x000076e399e287f3 in __GI_abort () at ./stdlib/abort.c:79
11 0x000076e39a39874b in _zlog_assert_failed (xref=0x76e39a46cca0 <_xref.27>, extra=0x0) at lib/zlog.c:789
12 0x000076e39a369dde in cancel_event_helper (m=0x5eda32df5e40, arg=0x5eda33afeed0, flags=1) at lib/event.c:1428
13 0x000076e39a369ef6 in event_cancel_event_ready (m=0x5eda32df5e40, arg=0x5eda33afeed0) at lib/event.c:1470
14 0x00005eda0a94a5b3 in bgp_stop (connection=0x5eda33afeed0) at bgpd/bgp_fsm.c:1355
15 0x00005eda0a94b4ae in bgp_stop_with_notify (connection=0x5eda33afeed0, code=8 '\b', sub_code=0 '\000') at bgpd/bgp_fsm.c:1610
16 0x00005eda0a979498 in bgp_packet_add (connection=0x5eda33afeed0, peer=0x5eda33b11800, s=0x76e3880daf90) at bgpd/bgp_packet.c:152
17 0x00005eda0a97a80f in bgp_keepalive_send (peer=0x5eda33b11800) at bgpd/bgp_packet.c:639
18 0x00005eda0a9511fd in peer_process (hb=0x5eda33c9ab80, arg=0x76e3985ffaf0) at bgpd/bgp_keepalives.c:111
19 0x000076e39a2cd8e6 in hash_iterate (hash=0x76e388000be0, func=0x5eda0a95105e <peer_process>, arg=0x76e3985ffaf0) at lib/hash.c:252
20 0x00005eda0a951679 in bgp_keepalives_start (arg=0x5eda3306af80) at bgpd/bgp_keepalives.c:214
21 0x000076e39a2c9932 in frr_pthread_inner (arg=0x5eda3306af80) at lib/frr_pthread.c:180
22 0x000076e399e94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
23 0x000076e399f26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) f 12
12 0x000076e39a369dde in cancel_event_helper (m=0x5eda32df5e40, arg=0x5eda33afeed0, flags=1) at lib/event.c:1428
1428 assert(m->owner == pthread_self());
In this decode the attempt to cancel the connection's events from
the wrong thread is causing the crash. Modify the code to create an
event on the bm->master to cancel the events for the connection.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
NEWLINK is only registered by the dplane thread, the main thread
doesn't care about it. So remove the real process of `netlink_link_change()`
for NEWLINK event in main thread.
And move NEWLINK/DELLINK event to the block where the dplane messages
are kept together.
Signed-off-by: anlan_cs <anlan_cs@126.com>
Trigger: Zebra core seen when we convert l2vni to l3vni and back
BackTrace:
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xe9) [0x7f4af96989d9]
/usr/lib/frr/zebra(zebra_vxlan_if_vni_up+0x250) [0x5561022ae030]
/usr/lib/frr/zebra(netlink_vlan_change+0x2f4) [0x5561021fd354]
/usr/lib/frr/zebra(netlink_parse_info+0xff) [0x55610220d37f]
/usr/lib/frr/zebra(+0xc264a) [0x55610220d64a]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f4af967e96d]
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f4af9637588]
/usr/lib/frr/zebra(main+0x402) [0x5561021f4d32]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f4af932624a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f4af9326305]
/usr/lib/frr/zebra(_start+0x21) [0x5561021f72f1]
Root Cause:
In working case,
- We get a RTM_NEWLINK whose ctx is enqueued by zebra dplane and
dequeued by zebra main and processed i.e.
(102000 is deleted from vxlan99) before we handle RTM_NEWVLAN.
- So in handling of NEWVLAN (vxlan99) we bail out since find with
vlan id 703 does not exist.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/working/nocras.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 23:09:33.741105 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 23:09:33.744061 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 23:09:33.767240 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 23:09:33.767380 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f2244000cf0, op INTF_INSTALL, ifindex (73), result QUEUED
2024/07/18 23:09:33.767389 ZEBRA: [NVFT0-HS1EX] INTF_INSTALL for vxlan99(73)
2024/07/18 23:09:33.767404 ZEBRA: [TQR2A-H2RFY] Vlan-Vni(1186:1186-6000002:6000002) update for VxLAN IF vxlan99(73)
2024/07/18 23:09:33.767422 ZEBRA: [TP4VP-XZ627] Del L2-VNI 102000 intf vxlan99(73)
2024/07/18 23:09:33.767858 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 23:09:33.767866 ZEBRA: [KKZGZ-8PCDW] Cannot find VNI for VID (703) IF vxlan99 for vlan state update >>>>BAIL OUT
In failure case,
- The NEWVLAN is received first even before processing RTM_NEWLINK.
- Since the vxlan id 102000 is not removed from the vxlan99,
the find with vlan id 703 returns the 102000 one and we invoke
zebra_vxlan_if_vni_up where the interfaces don't match and assert.
root@leaf2:mgmt:/var/log/frr# cat ~/raja_logs/noworking/crash.log | grep "RTM_NEWLINK\|QUEUED\|vxlan99\|in thread"
2024/07/18 22:26:43.829370 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=616, seq=0, pid=0
2024/07/18 22:26:43.829646 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7fe07c026d30, op INTF_INSTALL, ifindex (65), result QUEUED
2024/07/18 22:26:43.853930 ZEBRA: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan99 NS 0
2024/07/18 22:26:43.853949 ZEBRA: [K61WJ-XQQ3X] Intf vxlan99(73) L2-VNI 102000 is UP >>> VLAN PROCESSED BEFORE INTF EVENT
2024/07/18 22:26:43.853951 ZEBRA: [SPV50-BX2RP] RAJA zevpn_vxlanif vxlan48 and ifp vxlan99
2024/07/18 22:26:43.854005 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=508, seq=0, pid=0
2024/07/18 22:26:43.854241 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=516, seq=0, pid=0
2024/07/18 22:26:43.854251 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=544, seq=0, pid=0
ZEBRA: in thread kernel_read scheduled from zebra/kernel_netlink.c:505 kernel_read()
Fix:
Similar to #13396, where link change
handling was offloaded to dplane, same is being done for vlan events.
Note: Prior to this change, zebra main thread was interested in the
RTNLGRP_BRVLAN. So all the kernel events pertaining to vlan was
handled by zebra main.
With this change change as well the handling of vlan events is still
with Zebra main. However we offload it via Dplane thread.
Ticket :#3878175
Signed-off-by: Rajasekar Raja <rajasekarr@nvidia.com>
As part of the kernel netlink functionality, it is
possible that a bit of nested attributes can be
passed up. This attribute has a type value which
is stored in the lower 8 bits and in the upper 8
bits are a couple control flags that can be used.
FRR can parse this data and then just throw away
the value unless we mask off the upper 8 bits.
Let's ensure that it can be properly parsed.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
When zebra is started, someone may have configured an SRv6 encap source
address different from the default address ( :: ) in the kernel.
On startup, zebra should not assume that the actual SRv6 encap source
address is the default address ( :: ), but should retrieve the actual
source address from the kernel and put it in zebra configuration. In
other words, on startup we expect the actual SRv6 encap source
address and the configured one to be the same.
This commit makes the necessary changes to support the above.
Signed-off-by: Carmine Scarpitta <cscarpit@cisco.com>
Add a bunch of set functions and associated data structure in
zebra_dplane to allow the configuration of the source address for SRv6
encap in the data plane.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
Generic Netlink is an extension of Netlink meant for kernel-user space
communications. It supports the dynamic allocation of communication
channels. Kernel and user space applications register their services
with a Generic Netlink controller. The Generic Netlink controller is
responsible for assigning a unique channel number with each service.
Clients who want to use a service query the controller to see if
the service exists and to determine the correct channel number. The
channel number is used to access the requested service.
This commit adds the base functionality to get the channel number
assigned to a specific service. More precisely, this commit adds a
function `genl_resolve_family()` that takes the service name (called
family in the Generic Netlink terminology) as an input parameter and
queries the Generic Netlink controller to get the channel number
assigned with the requested service.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
zebra already supports several Netlink sockets which allow it to
communicate with the kernel. Each Netlink socket has a specific purpose:
we have a socket for incoming events from the kernel, a socket for
programming the dataplane, a socket for the kernel messages, a socket
used as the command channel. All the currently supported sockets are
based on the `NETLINK_ROUTE` protocol.
This commit adds a new Netlink socket that allows zebra to send
commands to the kernel using the `Generic Netlink` protocol.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
The `netlink_socket()` function is used in many places to create and
initialize Netlink sockets. Currently, it can only create
`NETLINK_ROUTE` Netlink sockets.
This commit generalizes the behavior of the `netlink_socket()` function,
enabling it to generate Netlink sockets of any type. Specifically, it
extends the `netlink_socket()` function with a new argument `nl_family`,
which allows developers to specify the Netlink family of the socket to
be created.
Signed-off-by: Carmine Scarpitta <carmine.scarpitta@uniroma2.it>
a) nl_batch_tx_buf was not being freed
b) the mlag_fifo was not being freed
c) the vrf->ns_ctxt was not being freed
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The headers associated with netlink code
really only belong in those that need it.
Move these headers out of lib/zebra.h and
into more appropriate places. bgp's usage
of the RT_TABLE_XXX defines are probably not
appropriate and will be cleaned up in future
commits.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Separate commit for clang-format cleanup of string
messages because I felt it would hide the actual
changes being made to the system.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
We may receive some xxxCHAIN netlink messages, but we ignore
them (currently). Add them to the basic handler callback so
that we don't log errors about them.
Signed-off-by: Mark Stapp <mjs@labn.net>
Code is looking up the nlsock to generate the batch messages
and then looking it up again to get the response. Let's
just look it up one time.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
a) Move the reads of link and address information
into the dplane
b) Move the startup read of data into the dplane
as well.
c) Break up startup reading of the linux kernel data
into multiple phases. As that we have implied ordering
of data that must be read first and if the dplane has
taken over some data reading then we must delay initial
read-in of other data.
Fixes: #13288
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
1) Add a bunch of get/set functions and associated data
structure in zebra_dplane to allow the setting and retrieval
of interface netlink data up into the master pthread.
2) Add a bit of code to breakup startup into stages. This is
because FRR currently has a mix of dplane and non dplane interactions
and the code needs to be paused before continuing on.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Effectively a massive search and replace of
`struct thread` to `struct event`. Using the
term `thread` gives people the thought that
this event system is a pthread when it is not
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
This is a first in a series of commits, whose goal is to rename
the thread system in FRR to an event system. There is a continual
problem where people are confusing `struct thread` with a true
pthread. In reality, our entire thread.c is an event system.
In this commit rename the thread.[ch] files to event.[ch].
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
ignore GETVLAN errors at startup like we are doing
for nexthop groups. Older platforms don't support the API.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Read in STP state changes for a Single Vxlan Device
via bridge vlan netlink messages. Map the vlanid to a
VNI in the SVD table and treat it similar to how
we handle proto down of the Vxlan device traditionally
in a non-SVD device scenario.
Forwarding == Interface UP
Blocking == Interface DOWN
Signed-off-by: Stephen Worley <sworley@nvidia.com>
Encode the vni label during route install on linux
systems via lwt encap 64bit LWTUNNEL_IP_ID. The kernel expects
this in network byte order, so we convert it.
Signed-off-by: Stephen Worley <sworley@nvidia.com>
This allows Zebra to manage QDISC, TCLASS, TFILTER in kernel and do cleaning
jobs when it starts up.
Signed-off-by: Siger Yang <siger.yang@outlook.com>
To resolve link dependencies of unordered interfaces, the commit
`520ebf72b27c2462ce8b0dc5a1d4cb83956df69c` has separated assignment of
`zif->link_ifindex` and `zif->link` from `netlink_interface()` during startup.
The fixup stage of `zebra_if_update_all_links()` goes into the last of
`interface_lookup_netlink()`, it can't be executed in the case of error in
above `netlink_parse_info()`s.
`RTM_GETTUNNEL` is not supported in linux kernel until 5.18, so
`netlink_parse_info()` will throw error with the previous versions.
If two conditions are met, (it is a common case)
1. Interfaces are created before frr restart/start
2. Linux kernel version < 5.18
the link dependencies will not be done, then evpn feature will be broken.
IMO we should just ignore this error.
Signed-off-by: anlan_cs <vic.lan@pica8.com>
This commit implements necessary netlink encoders for traffic control
including QDISC, TCLASS and TFILTER, and adds basic dplane operations.
Co-authored-by: Stephen Worley <sworley@nvidia.com>
Signed-off-by: Siger Yang <siger.yang@outlook.com>
There exists a possibility that an end operator has choosen
to compile FRR on an extremely old KERNEL that does not support
the SOL_NETLINK sockopt call. If so let's note it for them
instead of stuff silently not working.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
The usage of SOL_NETLINK for adding memberships of interest is
1 group per call. The netink_socket function implied that
the call could be a bitfield of values. This is not correct
at all. This will trip someone else up in the future when
a new value is needed. Let's get it right `now` before
it becomes a problem.
Let's also add a bit of extra code to give operator a better
understanding of what went wrong when a kernel does not
support the option.
Finally as a point of future reference should FRR just switch
over to a loop to add the required loops instead of having
this bastardized approach of some going in one way and some
going in another way?
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
'bridge vni add vni <id> dev <vxlan device>'
generates new RTM_NEWTUNNEL and RTM_DELTUNNEL
to add or remove vni to l3vxlan device.
Register new RTNLGRP_TUNNEL group to receive
new netlink notification.
Callback for the new RTM_xxxTUNNEL.
kernel patches:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
linux.git/commit/?h=v5.18-rc7&id=7b8135f4df98b155b23754b6065c157861e268f1
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
linux.git/commit/?h=v5.18-rc7&id=f9c4bb0b245cee35ef66f75bf409c9573d934cf9
Ticket:#3073812
Testing Done:
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
Signed-off-by: Chirag Shah <chirag@nvidia.com>
The kernel supports l3vxlan device to have (l3vni)
vni filter similar to vlan filtering on bridge device.
To receive netlink notification, FRR to register
for new netlink RTNLGRP_TUNNEL message.
This message required to register via additional
socket option as it's beyond bitmap size.
kernel patches:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
linux.git/commit/?h=v5.18-rc7&id=7b8135f4df98b155b23754b6065c157861e268f1
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/
linux.git/commit/?h=v5.18-rc7&id=f9c4bb0b245cee35ef66f75bf409c9573d934cf9
Ticket:#3073812
Testing Done:
Signed-off-by: Chirag Shah <chirag@nvidia.com>
There exists code paths in the linux kernel where a dump command
will be interrupted( I am not sure I understand what this really
means ) and the data sent back from the kernel is wrong or incomplete.
At this point in time I am not 100% certain what should be done, but
let's start noticing that this has happened so we can formulate a plan
or allow the end operator to know bad stuff is a foot at the circle K.
Signed-off-by: Donald Sharp <sharpd@nvidia.com>
