dpdk demo test failed, by lkl wiki.

[amjy]

uname -a
Linux localhost.localdomain 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

build:
https://github.com/lkl/linux/wiki/Howto:-DPDK-with-LKL

dpdk setup:
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
mkdir /mnt/huge
mount -t hugetlbfs nodev /mnt/huge

echo 0 > /proc/sys/kernel/randomize_va_space

cd dpdk*/usertools/
modprobe uio
insmod ../build/kmod/igb_uio.ko
insmod ../build/kmod/rte_kni.ko

python dpdk-devbind.py --status
ifconfig enp0s9 down
python dpdk-devbind.py --bind=igb_uio enp0s9
python dpdk-devbind.py --status

ping:
LKL_HIJACK_NET_IFTYPE=dpdk LKL_HIJACK_NET_IFPARAMS=dpdk0 LKL_HIJACK_NET_IP=172.17.100.11 LKL_HIJACK_NET_NETMASK_LEN=16 ./bin/lkl-hijack.sh ping 172.17.0.1
connect: network unreachable

ip link:
[root@localhost lkl]# LKL_HIJACK_NET_IFTYPE=dpdk LKL_HIJACK_NET_IFPARAMS=dpdk0 LKL_HIJACK_NET_IP=172.17.100.11 LKL_HIJACK_NET_NETMASK_LEN=16 ./bin/lkl-hijack.sh ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: mtu 1480 qdisc noop state DOWN mode DEFAULT qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
[root@localhost lkl]#

[thehajime]

it looks LKL_HIJACK_NET_IFTYPE=dpdk is not correctly working.

could you show us the result of following commands ?

• python dpdk-devbind.py --status
• LKL_HIJACK_DEBUG=0x400 LKL_HIJACK_NET_IFTYPE=dpdk LKL_HIJACK_NET_IFPARAMS=dpdk0 LKL_HIJACK_NET_IP=172.17.100.11 LKL_HIJACK_NET_NETMASK_LEN=16 ./bin/lkl-hijack.sh ip l

[amjy]

[root@localhost lkl]# LKL_HIJACK_DEBUG=0x400 LKL_HIJACK_NET_IFTYPE=dpdk LKL_HIJACK_NET_IFPARAMS=dpdk0 LKL_HIJACK_NET_IP=172.17.100.11 LKL_HIJACK_NET_NETMASK_LEN=16 ./bin/lkl-hijack.sh ip l
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7ffff6000000 (size = 0x400000)
EAL: Ask a virtual area of 0x29000000 bytes
EAL: Virtual area found at 0x7fffcce00000 (size = 0x29000000)
EAL: Ask a virtual area of 0x1e00000 bytes
EAL: Virtual area found at 0x7fffcae00000 (size = 0x1e00000)
EAL: Ask a virtual area of 0x1e00000 bytes
EAL: Virtual area found at 0x7fffc8e00000 (size = 0x1e00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fffc8a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fffc8600000 (size = 0x200000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7fffc7800000 (size = 0xc00000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fffc7400000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fffc7000000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fffc6c00000 (size = 0x200000)
EAL: Requesting 371 pages of size 2MB from socket 0
EAL: TSC frequency is ~2400052 KHz
EAL: Master lcore 0 is ready (tid=f7fee8c0;cpuset=[0])
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
EAL: Not managed by a supported kernel driver, skipped
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL: probe driver: 8086:100e net_e1000_em
EAL: PCI memory mapped at 0x7ffff6400000
PMD: eth_em_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x100e
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
EAL: Driver (net_vmxnet3) doesn't match the device
EAL: Driver (net_ixgbe) doesn't match the device
EAL: Driver (net_ixgbe_vf) doesn't match the device
EAL: Driver (net_e1000_igb) doesn't match the device
EAL: Driver (net_e1000_igb_vf) doesn't match the device
EAL: Driver (net_e1000_em) doesn't match the device
EAL: Driver (net_virtio) doesn't match the device
dpdk: failed to allocate tx pool
[ 0.000000] Linux version 4.11.0+ (root@localhost.localdomain) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 Tue Aug 8 05:05:20 EDT 2017
[ 0.000000] bootmem address range: 0x7fffbc000000 - 0x7fffbffff000
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16159
[ 0.000000] Kernel command line:
[ 0.000000] PID hash table entries: 256 (order: -1, 2048 bytes)
[ 0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Memory available: 64492k/0k RAM
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS:4096
[ 0.000000] lkl: irqs initialized
[ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000002] lkl: time and timers initialized (irq1)
[ 0.000017] pid_max: default: 4096 minimum: 301
[ 0.000053] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.000055] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[ 0.006371] console [lkl_console0] enabled
[ 0.006403] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.006413] xor: automatically using best checksumming function 8regs
[ 0.006540] NET: Registered protocol family 16
[ 0.781662] raid6: int64x1 gen() 4824 MB/s
[ 1.505295] raid6: int64x1 xor() 3091 MB/s
[ 2.253836] raid6: int64x2 gen() 5545 MB/s
[ 3.060943] raid6: int64x2 xor() 4159 MB/s
[ 3.797952] raid6: int64x4 gen() 5109 MB/s
[ 4.485379] raid6: int64x4 xor() 3414 MB/s
[ 5.249543] raid6: int64x8 gen() 3973 MB/s
[ 6.049422] raid6: int64x8 xor() 2273 MB/s
[ 6.049459] raid6: using algorithm int64x2 gen() 5545 MB/s
[ 6.049468] raid6: .... xor() 4159 MB/s, rmw enabled
[ 6.049492] raid6: using intx1 recovery algorithm
[ 6.049930] clocksource: Switched to clocksource lkl
[ 6.050282] NET: Registered protocol family 2
[ 6.050709] TCP established hash table entries: 512 (order: 0, 4096 bytes)
[ 6.050723] TCP bind hash table entries: 512 (order: 0, 4096 bytes)
[ 6.050734] TCP: Hash tables configured (established 512 bind 512)
[ 6.051309] UDP hash table entries: 128 (order: 0, 4096 bytes)
[ 6.051332] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes)
[ 6.052726] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[ 6.056570] SGI XFS with ACLs, security attributes, no debug enabled
[ 6.065877] io scheduler noop registered
[ 6.065901] io scheduler deadline registered
[ 6.066005] io scheduler cfq registered (default)
[ 6.066011] io scheduler mq-deadline registered
[ 6.072841] NET: Registered protocol family 10
[ 6.073793] Segment Routing with IPv6
[ 6.073821] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 6.074568] Btrfs loaded, crc32c=crc32c-generic
[ 6.074721] Warning: unable to open an initial console.
[ 6.074756] This architecture does not have kernel memory protection.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: sit0@NONE: mtu 1480 qdisc noop state DOWN mode DEFAULT qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
[ 6.076022] reboot: Restarting system
[root@localhost lkl]#

[amjy]

[root@localhost usertools]# python dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:00:09.0 '82540EM Gigabit Ethernet Controller 100e' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:00:03.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s3 drv=e1000 unused=igb_uio Active
0000:00:08.0 '82540EM Gigabit Ethernet Controller 100e' if=enp0s8 drv=e1000 unused=igb_uio Active

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>
[root@localhost usertools]#

[amjy]

[root@localhost lkl]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:02.0 VGA compatible controller: InnoTek Systemberatung GmbH VirtualBox Graphics Adapter
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:04.0 System peripheral: InnoTek Systemberatung GmbH VirtualBox Guest Service
00:05.0 Multimedia audio controller: Intel Corporation 82801AA AC'97 Audio Controller (rev 01)
00:06.0 USB controller: Apple Inc. KeyLargo/Intrepid USB
00:07.0 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:08.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:09.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
00:0b.0 USB controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) USB2 EHCI Controller
00:0d.0 SATA controller: Intel Corporation 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode] (rev 02)
[root@localhost lkl]#

[amjy]

Centos 7 in Virtual Box;
Other dpdk applications without lkl work ok.

[thehajime]

dpdk: failed to allocate tx pool

this is the reason of the failure.

current implementation (especially parameter configuration of DPDK backend) is somehow brittle.
You may want to change MAX_PACKET_SZ or MBUF_NUM in virtio_net_dpdk.c (below) to avoid the above error (failed to allocate ...): those are dependent on the device and huge page configuration, which I haven't done with nicer way.

https://github.com/lkl/linux/blob/master/tools/lkl/lib/virtio_net_dpdk.c#L36

[amjy]

more memory required ?

[thehajime]

more memory required ?

or try to decrease the MAX_PACKET_SZ: the default value was increased to support TSO aggregated packet which has a larger size of packet from LKL stack.

[amjy]

The problem is not no enough memory, it is failed by rte_mempool_set_ops_byname(mp, "ring_mp_mc", NULL); return -22

call trace:
lkl_netdev_dpdk_create
-> rte_mempool_create
-> ret = rte_mempool_set_ops_byname(mp, "ring_mp_mc", NULL);

[amjy]

rte_mempool_ops_table.num_ops is 0 in rte_mempool_set_ops_byname

opt == NULL then return -EINVAL

[thehajime]

in such case (num_ops == 0), initialization process of DPDK went something wrong...
are you using dpdk v17.02 which can be built by tools/lkl/scripts/dpdk-sdk-build.sh ?

[amjy]

sorry. the dpdk latest version.
I tried tools/lkl/scripts/dpdk-sdk-build.sh

[root@localhost lkl]# LKL_HIJACK_NET_IFTYPE=dpdk LKL_HIJACK_NET_IFPARAMS=dpdk0 LKL_HIJACK_NET_IP=172.17.100.11 LKL_HIJACK_NET_NETMASK_LEN=16 ./bin/lkl-hijack.sh ping 172.17.0.1
PING 172.17.0.1 (172.17.0.1) 56(84) bytes of data.
Warning: time of day goes back (-1503652261835435us), taking countermeasures.
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=3.42 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.145 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=4.97 ms
64 bytes from 172.17.0.1: icmp_seq=4 ttl=64 time=0.199 ms