A deadlock occurs because there is a violation of the rule of lock ordering;
softnet_lock is held with hodling IFNET_LOCK, which violates the rule.
To avoid the deadlock, replace softnet_lock in in_control and in6_control
with KERNEL_LOCK.
We also need to add some KERNEL_LOCKs to protect the network stack surely.
This is required, for example, for PR kern/51356.
Fix PR kern/53043
are useless, because the IPv6 entry point (ip6_input) already performs
them.
The checks were first added in the protocol functions:
Wed Dec 22 04:03:02 1999 UTC (18 years, 1 month ago) by itojun
"drop IPv6 packets with v4 mapped address on src/dst. they are illegal
and may be used to fool IPv6 implementations (by using ::ffff:127.0.0.1 as
source you may be able to pretend the packet is from local node)"
Shortly afterwards they were also added in the IPv6 entry point, but
where not removed from the protocol functions:
Mon Jan 31 10:33:22 2000 UTC (18 years ago) by itojun
"be proactive about malicious packet on the wire. we fear that v4 mapped
address to be used as a tool to hose security filters (like bypassing
"local host only" filter by using ::ffff:127.0.0.1)."
OpenBSD did the same a few months ago. FreeBSD has never had these checks.
reference in RFCs about what a correct limit should be, but FreeBSD already
uses 15.
If an IPv6 packet has 50 options, there is clearly something wrong with it.
guaranteed to be in the same mbuf as ip6, so computing the difference
between the pointers may result in a wrong offset.
ip6 is now unused, so remove it.
just couldn't work.
By the way, I'm wondering what is the point of this block. Calling
ip6_hopopts_input() won't achieve anything useful, and it could actually
be a problem, because there are several paths in it that call icmp6_error,
which calls ip6_output, and then we're back in the same function. Besides
it is possible to reach icmp6_error with a packet we emitted (as opposed
to a packet we are forwarding), and in that case we are sending an ICMP
error back to ourselves.
m_pkthdr.len in ip6_insertfraghdr(), it gets recomputed after calling
this function.
If we cared there would be a bug, since we don't increase it in the
other branches.
mtod(m, char *) + len
is wrong, an option is allowed to be located in another mbuf of the chain.
If the offset of an option within the chain is bigger than the length of
the first mbuf in that chain, we are reading/writing one byte of packet-
controlled data beyond the end of the first mbuf.
The length of this first mbuf depends on the layout the network driver
chose. In the most difficult case, it will allocate a 2KB cluster, which
is bigger than the Ethernet MTU.
But there is at least one way of exploiting this case: by sending a
special combination of nested IPv6 fragments, the packet can control a
good bunch of 'len'. By luck, the memory pool containing clusters does not
embed the pool header in front of the items, so it is not straightforward
to predict what is located at 'mtod(m, char *) + len'.
However, by sending offending fragments in a loop, it is possible to
crash the kernel - at some point we will hit important data structures.
As far as I can tell, PF protects against this difficult case, because
it kicks nested fragments. NPF does not protect against this. IPF I don't
know.
Then there are the more easy cases, if the MTU is bigger than a cluster,
or if the network driver did not allocate a cluster, or perhaps if the
fragments are received via a tunnel; I haven't investigated these cases.
Change ip6_get_prevhdr so that it returns an offset in the chain, and
always use IP6_EXTHDR_GET to get a writable pointer. IP6_EXTHDR_GET
leaves M_PKTHDR untouched.
This place is still fragile.
By the way, it would be nice to revisit the use of 'ip6flow_lock' in
ip6flow_fastforward(): it is taken right away because of 'ip6flow_inuse',
but then we perform several checks that do not require it.
allocate, rather than relying on an arbitrary length passed in from
userland.
Allow copyout() of partial results if the user buffer is too small, to
be consistent with the way sysctl(3) is documented.
Garbage-collect now-unused third parrameter in the fill_[pd]rlist()
functions.
As discussed on IRC.
OK kamil@ and christos@
XXX Needs pull-up to netbsd-8 branch.
* l2tp_input(): use m_copydata, and ensure there is enough space in the
chain. Otherwise overflow.
* l2tp_tcpmss_clamp(): ensure there is enough space in the chain.
* in_l2tp_output(): don't check 'sc' against NULL, it can't be NULL.
* in_l2tp_input(): no need to call m_pullup since we use m_copydata.
Just check the space in the chain.
* in_l2tp_input(): if there is a cookie, make sure the chain has enough
space.
* in6_l2tp_input(): same changes as in_l2tp_input().
Ok knakahara@
* Move the structure definitions into frag6.c, they should not be used
elsewhere.
* Rename ip6af_mff -> ip6af_more, and switch it to bool, easier to
understand.
* Remove IP6_REASS_MBUF, no point in keeping this.
* Remove ip6q_arrive and ip6q_nxtp, unused.
* Style.
in6_ifattach_loopback can sleep so we cannot use pserialize for it. Fortunately
in6_ifattach is alwasy called with IFNET_LOCK so pserialize isn't needed there
actually.
pointer = opt - mtod(m, u_int8_t *)
because m may have gone through m_pulldown, and it is possible that
m->m_data is no longer the beginning of the packet.
struct nd_redirect (which is bigger). Also, make sure we can add a
struct nd_opt_rd_hdr.
Normally this doesn't change anything, since the mbuf has IPV6_MMTU
bytes, and it's always way bigger than what we need.
roundup(sizeof(*nd_opt) + ifp->if_addrlen, 8)
But we are not filling in the padding caused by the roundup, and therefore
several bytes are leaked, in the mbuf we're about to send to the network.
* Remove the (disabled) IPPROTO_ESP check. If the packet was decrypted it
will have M_DECRYPTED, and this is already checked.
* Memory leaks in icmp6_error2. They seem hardly triggerable.
* Fix miscomputation in _icmp6_input, the ICMP6 header is not guaranteed
to be located right after the IP6 header. ok mlelstv@
* Memory leak in _icmp6_input. This one seems to be impossible to trigger.
arp_dad_stoptimer and nd6_dad_stoptimer can be called with or without
softnet_lock held and unfortunately we have no easy way to statically know which.
So it is hard to use callout_halt there.
To address the situation, we use callout_stop to make the code safe. The new
approach copes with the issue by delegating the destruction of a callout to
callout itself, which allows us to not wait the callout to finish. This can be
done thanks to that DAD objects are separated from other data such as ifa.
The approach is suggested by riastradh@
Proposed on tech-kern@ and tech-net@
We must not call callout_halt of nd6_dad_timer with holding nd6_dad_lock because
the lock is taken in nd6_dad_timer. Once softnet_lock goes away, we can pass the
lock to callout_halt, but for now we cannot.
in6_ifinit checks the number of IPv6 addresses on a given interface and
if it's zero (i.e., an IPv6 address being assigned to the interface
is the first one), call if_addr_init. However, the actual assignment of
the address (ifa_insert) is out of in6_ifinit. The check and the
assignment must be done atomically.
Fix it by holding in6_ifaddr_lock during in6_ifinit and ifa_insert.
And also add missing pserialize to IFADDR_READER_FOREACH.
It reduces C&P codes such as "#ifndef NET_MPSAFE KERNEL_LOCK(1, NULL); ..."
scattered all over the source code and makes it easy to identify remaining
KERNEL_LOCK and/or softnet_lock that are held even if NET_MPSAFE.
No functional change
It happens because rtalloc1 is called from lltable with holding
IF_AFDATA_WLOCK.
If a route update is in action, rtalloc1 would wait for its completion with
holding IF_AFDATA_WLOCK. At the same moment, a softint (e.g., arpintr) may try
to take IF_AFDATA_WLOCK and get stuck on it. Unfortunately the stuck softint
prevents the route update from progressing because the route update calls
psref_target_destroy that needs the softint to complete.
A resource allocation graph of the senario looks like this:
route update =(psref_target_destroy)=> softint => IF_AFDATA_WLOCK
=(rt_update_wait)=> route update
Fix the deadlock by pulling rtalloc1 out of the lltable codes inside
IF_AFDATA_WLOCK.
Note that the deadlock happens only if NET_MPSAFE is enabled.
Currently softnet_lock is taken unnecessarily in some cases, e.g.,
icmp_input and encap4_input from ip_input, or not taken even if needed,
e.g., udp_input and tcp_input from ipsec4_common_input_cb. Fix them.
NFC if NET_MPSAFE is disabled (default).
The change introduces a global generation counter that is incremented when any
routes have been added or deleted. When a rtcache caches a rtentry into itself,
it also stores a snapshot of the generation counter. If the snapshot equals to
the global counter, the cache is still valid, otherwise invalidated.
One drawback of the change is that all rtcaches of all protocol families are
invalidated when any routes of any protocol families are added or deleted.
If that matters, we should have separate generation counters based on
protocol families.
This change removes LIST_ENTRY from struct route, which fixes a part of
PR kern/52515.
We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.
SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.
For more details please read the locking notes in key.c.
Proposed on tech-kern@ and tech-net@
It was created to copy FreeBSD, however actually the cache isn't
necessary. Remove it to simplify the code and reduce the cost to
maintain it (e.g., keep a consistency with a corresponding local
route).
kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()
all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.
Note that we should modularize netipsec and reduce reverse symbol references
(referencing symbols of netipsec from net, netinet and netinet6) though,
the task needs lots of code changes. Prior to doing so, rumpifying it and
having ATF tests should be useful.
For unknown reasons, IPv6 multicast addresses are linked to a first
IPv6 address assigned to an interface. Due to the design, when removing
a first address having multicast addresses, we need to save them to
somewhere and later restore them once a new IPv6 address is activated.
mkludge stuffs support the operations.
This change links multicast addresses to an interface directly and
throws the kludge away.
Note that as usual some obsolete member variables remain for kvm(3)
users. And also sysctl net.inet6.multicast_kludge remains to avoid
breaking old ifmcstat.
TODO: currently ifnet has a list of in6_multi but obviously the list
should be protocol independent. Provide a common structure (if_multi
or something) to handle in6_multi and in_multi together as well as
ifaddr does for in_ifaddr and in6_ifaddr.
We cannot remove a prefix there. Instead just invalidate it; the prefix
will be removed when purging an associated address. This is the same as
the original behavior.
The benefits of this change are:
- The flow is consistent with IPv4 (and FreeBSD and OpenBSD)
- old: ip6_output => nd6_output (do ND if needed) => L2_output (lookup a stored cache)
- new: ip6_output => L2_output (lookup a cache. Do ND if cache not found)
- We can remove some workarounds in nd6_output
- We can move L2 specific operations to their own place
- The performance slightly improves because one cache lookup is reduced
pr_input is currently called in softint. Some pr_input such as ICMP, ICMPv6
and CARP can add/delete/update IP addresses and routing table entries. For
example, icmp6_redirect_input updates an a routing table entry and
nd6_ra_input may delete an IP address.
Basically such operations shouldn't be done in softint. That aside, we have
a reason to avoid the situation; psz/psref waits cannot be used in softint,
however they are required to work in such pr_input in the MP-safe world.
The change implements the workqueue pr_input framework called wqinput which
provides a means to defer pr_input of a protocol to workqueue easily.
Currently icmp_input, icmp6_input, carp_proto_input and carp6_proto_input
are deferred to workqueue by the framework.
Proposed and discussed on tech-kern and tech-net
- Don't hold softnet_lock in some functions if NET_MPSAFE
- Add softnet_lock to sysctl_net_inet_icmp_redirtimeout
- Add softnet_lock to expire_upcalls of ip_mroute.c
- Restore softnet_lock for in{,6}_pcbpurgeif{,0} if NET_MPSAFE
- Mark some softnet_lock for future work
It's useless in this case, because without it we can know that
the lock is held or not on a next lock acquisition and even more
if LOCKDEBUG is enabled a failure on the acquisition will provide
useful information for debugging while an assertion failure will
provide just the fact that the assertion failed.
llentry timer (of nd6) holds both llentry's lock and softnet_lock.
A caller also holds them and calls callout_halt to wait for the
timer to quit. However we can pass only one lock to callout_halt,
so passing either of them can cause a deadlock. Fix it by avoid
calling callout_halt without holding llentry's lock.
BTW in the first place we cannot pass llentry's lock to callout_halt
because it's a rwlock...
in6_purgeaddr (in6_unlink_ifa) itself unrefernces a prefix entry and calls
nd6_prelist_remove if the counter becomes 0, so callers doesn't need to
handle the reference counting.
Performance-sensitive paths (sending/forwarding packets) call just one
reader lock. This is a trade-off between performance impact vs. the amount
of efforts; if we want to remove the reader lock, we need huge amount of
works including destroying objects with psz/psref in softint, for example.
The reference counter represents the numuber of references from IPv6
addresses to a prefix entry. If all IPv6 addresses assigned to an
interface are purged, all references to a prefix for the interface are
also released. For now nd6_purge is always called after purging all IPv6
addresses, so we can get rid of clearing pr->ndpr_refcnt from nd6_purge
and instead we can assert it's 0 there.
Note that nd6_ifdetach is only called via dom_ifdetach when processing
if_detach where dom_ifdetach is called after pr_purgeif that eventually
calls in6_ifdetach. So in the call path nd6_purge in nd6_ifdetach does
nothing. That said, we should explicitly make it sure to purge all
IPv6 addresses before nd6_purge for future changes (or the case I missed
something). So if_purgeaddrs is added to nd6_ifdetach.
There were two nd6_purge in in6_ifdetach for some reason, but at least now
We don't need extra nd6_purge. Remove it and instead add assertions that
check if surely purged.
The benefits of the change are:
- We can reduce codes
- We can provide the same behavior between drivers
- Where/When if_ipackets is counted up
- Note that some drivers still update packet statistics in their own
way (periodical update)
- Moved bpf_mtap run in softint
- This makes it easy to MP-ify bpf
Proposed on tech-kern and tech-net
See the following descriptions for details.
Proposed on tech-kern and tech-net
Overview
--------
We protect the routing table with a rwock and protect
rtcaches with another rwlock. Each rtentry is protected
from being freed or updated via reference counting and psref.
Global rwlocks
--------------
There are two rwlocks; one for the routing table (rt_lock) and
the other for rtcaches (rtcache_lock). rtcache_lock covers
all existing rtcaches; there may have room for optimizations
(future work).
The locking order is rtcache_lock first and rt_lock is next.
rtentry references
------------------
References to an rtentry is managed with reference counting
and psref. Either of the two mechanisms is used depending on
where a rtentry is obtained. Reference counting is used when
we obtain a rtentry from the routing table directly via
rtalloc1 and rtrequest{,1} while psref is used when we obtain
a rtentry from a rtcache via rtcache_* APIs. In both cases,
a caller can sleep/block with holding an obtained rtentry.
The reasons why we use two different mechanisms are (i) only
using reference counting hurts the performance due to atomic
instructions (rtcache case) (ii) ease of implementation;
applying psref to APIs such rtaloc1 and rtrequest{,1} requires
additional works (adding a local variable and an argument).
We will finally migrate to use only psref but we can do it
when we have a lockless routing table alternative.
Reference counting for rtentry
------------------------------
rt_refcnt now doesn't count permanent references such as for
rt_timers and rtcaches, instead it is used only for temporal
references when obtaining a rtentry via rtalloc1 and rtrequest{,1}.
We can do so because destroying a rtentry always involves
removing references of rt_timers and rtcaches to the rtentry
and we don't need to track such references. This also makes
it easy to wait for readers to release references on deleting
or updating a rtentry, i.e., we can simply wait until the
reference counter is 0 or 1. (If there are permanent references
the counter can be arbitrary.)
rt_ref increments a reference counter of a rtentry and rt_unref
decrements it. rt_ref is called inside APIs (rtalloc1 and
rtrequest{,1} so users don't need to care about it while
users must call rt_unref to an obtained rtentry after using it.
rtfree is removed and we use rt_unref and rt_free instead.
rt_unref now just decrements the counter of a given rtentry
and rt_free just tries to destroy a given rtentry.
See the next section for destructions of rtentries by rt_free.
Destructions of rtentries
-------------------------
We destroy a rtentry only when we call rtrequst{,1}(RTM_DELETE);
the original implementation can destroy in any rtfree where it's
the last reference. If we use reference counting or psref, it's
easy to understand if the place that a rtentry is destroyed is
fixed.
rt_free waits for references to a given rtentry to be released
before actually destroying the rtentry. rt_free uses a condition
variable (cv_wait) (and psref_target_destroy for psref) to wait.
Unfortunately rtrequst{,1}(RTM_DELETE) can be called in softint
that we cannot use cv_wait. In that case, we have to defer the
destruction to a workqueue.
rtentry#rt_cv, rtentry#rt_psref and global variables
(see rt_free_global) are added to conduct the procedure.
Updates of rtentries
--------------------
One difficulty to use refcnt/psref instead of rwlock for rtentry
is updates of rtentries. We need an additional mechanism to
prevent readers from seeing inconsistency of a rtentry being
updated.
We introduce RTF_UPDATING flag to rtentries that are updating.
While the flag is set to a rtentry, users cannot acquire the
rtentry. By doing so, we avoid users to see inconsistent
rtentries.
There are two options when a user tries to acquire a rtentry
with the RTF_UPDATING flag; if a user runs in softint context
the user fails to acquire a rtentry (NULL is returned).
Otherwise a user waits until the update completes by waiting
on cv.
The procedure of a updater is simpler to destruction of
a rtentry. Wait on cv (and psref) and after all readers left,
proceed with the update.
Global variables (see rt_update_global) are added to conduct
the procedure.
Currently we apply the mechanism to only RTM_CHANGE in
rtsock.c. We would have to apply other codes. See
"Known issues" section.
psref for rtentry
-----------------
When we obtain a rtentry from a rtcache via rtcache_* APIs,
psref is used to reference to the rtentry.
rtcache_ref acquires a reference to a rtentry with psref
and rtcache_unref releases the reference after using it.
rtcache_ref is called inside rtcache_* APIs and users don't
need to take care of it while users must call rtcache_unref
to release the reference.
struct psref and int bound that is needed for psref is
embedded into struct route. By doing so we don't need to
add local variables and additional argument to APIs.
However this adds another constraint to psref other than
reference counting one's; holding a reference of an rtentry
via a rtcache is allowed by just one caller at the same time.
So we must not acquire a rtentry via a rtcache twice and
avoid a recursive use of a rtcache. And also a rtcache must
be arranged to be used by a LWP/softint at the same time
somehow. For IP forwarding case, we have per-CPU rtcaches
used in softint so the constraint is guaranteed. For a h
rtcache of a PCB case, the constraint is guaranteed by the
solock of each PCB. Any other cases (pf, ipf, stf and ipsec)
are currently guaranteed by only the existence of the global
locks (softnet_lock and/or KERNEL_LOCK). If we've found the
cases that we cannot guarantee the constraint, we would need
to introduce other rtcache APIs that use simple reference
counting.
psref of rtcache is created with IPL_SOFTNET and so rtcache
shouldn't used at an IPL higher than IPL_SOFTNET.
Note that rtcache_free is used to invalidate a given rtcache.
We don't need another care by my change; just keep them as
they are.
Performance impact
------------------
When NET_MPSAFE is disabled the performance drop is 3% while
when it's enabled the drop is increased to 11%. The difference
comes from that currently we don't take any global locks and
don't use psref if NET_MPSAFE is disabled.
We can optimize the performance of the case of NET_MPSAFE
on by reducing lookups of rtcache that uses psref;
currently we do two lookups but we should be able to trim
one of two. This is a future work.
Known issues
------------
There are two known issues to be solved; one is that
a caller of rtrequest(RTM_ADD) may change rtentry (see rtinit).
We need to prevent new references during the update. Or
we may be able to remove the code (perhaps, need more
investigations).
The other is rtredirect that updates a rtentry. We need
to apply our update mechanism, however it's not easy because
rtredirect is called in softint and we cannot apply our
mechanism simply. One solution is to defer rtredirect to
a workqueue but it requires some code restructuring.
In the MP-safe world, a rtentry stemming from a rtcache can be freed at any
points. So we need to protect rtentries somehow say by reference couting or
passive references. Regardless of the method, we need to call some release
function of a rtentry after using it.
The change adds a new function rtcache_unref to release a rtentry. At this
point, this function does nothing because for now we don't add a reference
to a rtentry when we get one from a rtcache. We will add something useful
in a further commit.
This change is a part of changes for MP-safe routing table. It is separated
to avoid one big change that makes difficult to debug by bisecting.
nd6_purge depends on that IPv6 addresses are purged. If addresses remain,
pfxlist_onlink_check called from nd6_purge dereferences a dangling pointer
(ia->ia6_ndpr) that is freed before calling pfxlist_onlink_check. Fix it by
removing addresses before calling nd6_purge, which is the original behavior
that was changed by in6.c,v 1.203 and in6_ifattach.c,v 1.99.
Note that it seems the issue occurs because of a hack that forcibly destroys
prefix list entries of a given interface in nd6_purge. We should tackle the
hack in the future.
Fix PR kern/51467
This problem occurs only if NET_MPSAFE on.
ifconfig destroy side:
kernel entry point is ifioctl => if_clone_destroy.
pr_purgeif() acquires softnet_lock, and then ifa_remove() calls
pserialize_perform() holding softnet_lock.
ifconfig side:
kernel entry point is socreate.
pr_attach()(udp_attach_wrapper()) calls sosetlock(). In this call path,
sosetlock() try to acquire softnet_lock.
These can cause dead lock.
