mirror of https://github.com/torvalds/linux.git
ed9b70040d
By default, when a kmem_cache is created with SLAB_TYPESAFE_BY_RCU,
slub has to use extra storage for the freelist pointer after each
object, because slub assumes that any bit in the object
can be used by RCU readers.
Because proto_register() is also using SLAB_HWCACHE_ALIGN,
this forces slub to use one extra cache line per object.
We can instead put the slub freelist anywhere in the object,
granted the concurrent RCU readers are not supposed to
use the pointer value.
Add a new (struct sock)sk_freeptr field, in an union
with sk_rcu: No RCU readers would need to look at sk_rcu,
which is only used at free phase.
Tested:
grep . /sys/kernel/slab/TCP/{object_size,slab_size,objs_per_slab}
grep . /sys/kernel/slab/TCPv6/{object_size,slab_size,objs_per_slab}
Before:
/sys/kernel/slab/TCP/object_size:2368
/sys/kernel/slab/TCP/slab_size:2432
/sys/kernel/slab/TCP/objs_per_slab:13
/sys/kernel/slab/TCPv6/object_size:2496
/sys/kernel/slab/TCPv6/slab_size:2560
/sys/kernel/slab/TCPv6/objs_per_slab:12
After this patch, we can pack one more TCPv6 object per slab,
and object_size == slab_size.
/sys/kernel/slab/TCP/object_size:2368
/sys/kernel/slab/TCP/slab_size:2368
/sys/kernel/slab/TCP/objs_per_slab:13
/sys/kernel/slab/TCPv6/object_size:2496
/sys/kernel/slab/TCPv6/slab_size:2496
/sys/kernel/slab/TCPv6/objs_per_slab:13
Signed-off-by: Eric Dumazet <edumazet@google.com>
Link: https://patch.msgid.link/20260129153458.4163797-1-edumazet@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
|---|---|---|
| .. | ||
| Makefile | ||
| bpf_sk_storage.c | ||
| datagram.c | ||
| dev.c | ||
| dev.h | ||
| dev_addr_lists.c | ||
| dev_addr_lists_test.c | ||
| dev_api.c | ||
| dev_ioctl.c | ||
| devmem.c | ||
| devmem.h | ||
| drop_monitor.c | ||
| dst.c | ||
| dst_cache.c | ||
| failover.c | ||
| fib_notifier.c | ||
| fib_rules.c | ||
| filter.c | ||
| flow_dissector.c | ||
| flow_offload.c | ||
| gen_estimator.c | ||
| gen_stats.c | ||
| gro.c | ||
| gro_cells.c | ||
| gso.c | ||
| hotdata.c | ||
| hwbm.c | ||
| ieee8021q_helpers.c | ||
| link_watch.c | ||
| lock_debug.c | ||
| lwt_bpf.c | ||
| lwtunnel.c | ||
| mp_dmabuf_devmem.h | ||
| neighbour.c | ||
| net-procfs.c | ||
| net-sysfs.c | ||
| net-sysfs.h | ||
| net-traces.c | ||
| net_namespace.c | ||
| net_test.c | ||
| netclassid_cgroup.c | ||
| netdev-genl-gen.c | ||
| netdev-genl-gen.h | ||
| netdev-genl.c | ||
| netdev_config.c | ||
| netdev_queues.c | ||
| netdev_rx_queue.c | ||
| netevent.c | ||
| netmem_priv.h | ||
| netpoll.c | ||
| netprio_cgroup.c | ||
| of_net.c | ||
| page_pool.c | ||
| page_pool_priv.h | ||
| page_pool_user.c | ||
| pktgen.c | ||
| ptp_classifier.c | ||
| request_sock.c | ||
| rtnetlink.c | ||
| scm.c | ||
| secure_seq.c | ||
| selftests.c | ||
| skb_fault_injection.c | ||
| skbuff.c | ||
| skmsg.c | ||
| sock.c | ||
| sock_destructor.h | ||
| sock_diag.c | ||
| sock_map.c | ||
| sock_reuseport.c | ||
| stream.c | ||
| sysctl_net_core.c | ||
| timestamping.c | ||
| tso.c | ||
| utils.c | ||
| xdp.c | ||