vrf.txt 14.4 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Virtual Routing and Forwarding (VRF)
====================================
The VRF device combined with ip rules provides the ability to create virtual
routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
Linux network stack. One use case is the multi-tenancy problem where each
tenant has their own unique routing tables and in the very least need
different default gateways.

Processes can be "VRF aware" by binding a socket to the VRF device. Packets
through the socket then use the routing table associated with the VRF
device. An important feature of the VRF device implementation is that it
impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
(ie., they do not need to be run in each VRF). The design also allows
the use of higher priority ip rules (Policy Based Routing, PBR) to take
precedence over the VRF device rules directing specific traffic as desired.

In addition, VRF devices allow VRFs to be nested within namespaces. For
D
David Ahern 已提交
18 19 20
example network namespaces provide separation of network interfaces at the
device layer, VLANs on the interfaces within a namespace provide L2 separation
and then VRF devices provide L3 separation.
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Design
------
A VRF device is created with an associated route table. Network interfaces
are then enslaved to a VRF device:

         +-----------------------------+
         |           vrf-blue          |  ===> route table 10
         +-----------------------------+
            |        |            |
         +------+ +------+     +-------------+
         | eth1 | | eth2 | ... |    bond1    |
         +------+ +------+     +-------------+
                                  |       |
                              +------+ +------+
                              | eth8 | | eth9 |
                              +------+ +------+

Packets received on an enslaved device and are switched to the VRF device
D
David Ahern 已提交
40 41 42 43 44 45 46
in the IPv4 and IPv6 processing stacks giving the impression that packets
flow through the VRF device. Similarly on egress routing rules are used to
send packets to the VRF device driver before getting sent out the actual
interface. This allows tcpdump on a VRF device to capture all packets into
and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
applied using the VRF device to specify rules that apply to the VRF domain
as a whole.
47 48 49 50 51

[1] Packets in the forwarded state do not flow through the device, so those
    packets are not seen by tcpdump. Will revisit this limitation in a
    future release.

D
David Ahern 已提交
52 53 54 55
[2] Iptables on ingress supports PREROUTING with skb->dev set to the real
    ingress device and both INPUT and PREROUTING rules with skb->dev set to
    the VRF device. For egress POSTROUTING and OUTPUT rules can be written
    using either the VRF device or real egress device.
56 57 58 59 60 61 62

Setup
-----
1. VRF device is created with an association to a FIB table.
   e.g, ip link add vrf-blue type vrf table 10
        ip link set dev vrf-blue up

D
David Ahern 已提交
63 64 65 66 67 68 69
2. An l3mdev FIB rule directs lookups to the table associated with the device.
   A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
   l3mdev rule for IPv4 and IPv6 when the first device is created with a
   default preference of 1000. Users may delete the rule if desired and add
   with a different priority or install per-VRF rules.

   Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
70 71 72
       ip ru add oif vrf-blue table 10
       ip ru add iif vrf-blue table 10

D
David Ahern 已提交
73 74
3. Set the default route for the table (and hence default route for the VRF).
       ip route add table 10 unreachable default
75

D
David Ahern 已提交
76 77
4. Enslave L3 interfaces to a VRF device.
       ip link set dev eth1 master vrf-blue
78 79 80

   Local and connected routes for enslaved devices are automatically moved to
   the table associated with VRF device. Any additional routes depending on
D
David Ahern 已提交
81 82 83 84 85 86
   the enslaved device are dropped and will need to be reinserted to the VRF
   FIB table following the enslavement.

   The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
   addresses as VRF enslavement changes.
       sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
87

D
David Ahern 已提交
88 89
5. Additional VRF routes are added to associated table.
       ip route add table 10 ...
90 91 92 93 94 95 96 97 98 99 100


Applications
------------
Applications that are to work within a VRF need to bind their socket to the
VRF device:

    setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);

or to specify the output device using cmsg and IP_PKTINFO.

101 102 103
TCP & UDP services running in the default VRF context (ie., not bound
to any VRF device) can work across all VRF domains by enabling the
tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
D
David Ahern 已提交
104
    sysctl -w net.ipv4.tcp_l3mdev_accept=1
105
    sysctl -w net.ipv4.udp_l3mdev_accept=1
106

D
David Ahern 已提交
107 108 109 110 111 112
netfilter rules on the VRF device can be used to limit access to services
running in the default VRF context as well.

The default VRF does not have limited scope with respect to port bindings.
That is, if a process does a wildcard bind to a port in the default VRF it
owns the port across all VRF domains within the network namespace.
113 114 115 116 117

################################################################################

Using iproute2 for VRFs
=======================
D
David Ahern 已提交
118 119 120
iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
section lists both commands where appropriate -- with the vrf keyword and the
older form without it.
121 122 123 124 125 126

1. Create a VRF

   To instantiate a VRF device and associate it with a table:
       $ ip link add dev NAME type vrf table ID

D
David Ahern 已提交
127 128 129
   As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
   covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
   device create.
130 131 132 133 134 135 136 137 138

2. List VRFs

   To list VRFs that have been created:
       $ ip [-d] link show type vrf
         NOTE: The -d option is needed to show the table id

   For example:
   $ ip -d link show type vrf
D
David Ahern 已提交
139
   11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
140 141
       link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
       vrf table 1 addrgenmode eui64
D
David Ahern 已提交
142
   12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
143 144
       link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
       vrf table 10 addrgenmode eui64
D
David Ahern 已提交
145
   13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
146 147
       link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
       vrf table 66 addrgenmode eui64
D
David Ahern 已提交
148
   14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
149 150 151 152 153 154 155
       link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
       vrf table 81 addrgenmode eui64


   Or in brief output:

   $ ip -br link show type vrf
D
David Ahern 已提交
156 157 158 159
   mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
   red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
   blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
   green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
160 161 162 163 164 165


3. Assign a Network Interface to a VRF

   Network interfaces are assigned to a VRF by enslaving the netdevice to a
   VRF device:
D
David Ahern 已提交
166
       $ ip link set dev NAME master NAME
167 168 169 170 171

   On enslavement connected and local routes are automatically moved to the
   table associated with the VRF device.

   For example:
D
David Ahern 已提交
172
   $ ip link set dev eth0 master mgmt
173 174 175 176 177 178


4. Show Devices Assigned to a VRF

   To show devices that have been assigned to a specific VRF add the master
   option to the ip command:
D
David Ahern 已提交
179 180
       $ ip link show vrf NAME
       $ ip link show master NAME
181 182

   For example:
D
David Ahern 已提交
183 184
   $ ip link show vrf red
   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
185
       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
D
David Ahern 已提交
186
   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
187
       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
D
David Ahern 已提交
188
   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
189 190 191 192
       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff


   Or using the brief output:
193
   $ ip -br link show vrf red
194 195 196 197 198 199 200 201 202
   eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
   eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
   eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>


5. Show Neighbor Entries for a VRF

   To list neighbor entries associated with devices enslaved to a VRF device
   add the master option to the ip command:
D
David Ahern 已提交
203 204
       $ ip [-6] neigh show vrf NAME
       $ ip [-6] neigh show master NAME
205 206

   For example:
D
David Ahern 已提交
207
   $  ip neigh show vrf red
208 209 210
   10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
   10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE

211 212
   $ ip -6 neigh show vrf red
   2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
213 214 215 216 217 218


6. Show Addresses for a VRF

   To show addresses for interfaces associated with a VRF add the master
   option to the ip command:
D
David Ahern 已提交
219 220
       $ ip addr show vrf NAME
       $ ip addr show master NAME
221 222

   For example:
D
David Ahern 已提交
223 224
   $ ip addr show vrf red
   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
225 226 227 228 229 230 231
       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
       inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
          valid_lft forever preferred_lft forever
       inet6 2002:1::2/120 scope global
          valid_lft forever preferred_lft forever
       inet6 fe80::ff:fe00:202/64 scope link
          valid_lft forever preferred_lft forever
D
David Ahern 已提交
232
   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
233 234 235 236 237 238 239
       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
       inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
          valid_lft forever preferred_lft forever
       inet6 2002:2::2/120 scope global
          valid_lft forever preferred_lft forever
       inet6 fe80::ff:fe00:203/64 scope link
          valid_lft forever preferred_lft forever
D
David Ahern 已提交
240
   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
241 242 243
       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff

   Or in brief format:
D
David Ahern 已提交
244
   $ ip -br addr show vrf red
245 246 247 248 249 250 251 252 253
   eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
   eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
   eth5             DOWN


7. Show Routes for a VRF

   To show routes for a VRF use the ip command to display the table associated
   with the VRF device:
D
David Ahern 已提交
254
       $ ip [-6] route show vrf NAME
255 256 257
       $ ip [-6] route show table ID

   For example:
D
David Ahern 已提交
258
   $ ip route show vrf red
259 260 261 262 263 264 265 266 267 268
   prohibit default
   broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
   10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
   local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
   broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
   broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
   10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
   local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
   broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2

D
David Ahern 已提交
269
   $ ip -6 route show vrf red
270 271 272 273 274 275 276 277 278 279 280 281
   local 2002:1:: dev lo  proto none  metric 0  pref medium
   local 2002:1::2 dev lo  proto none  metric 0  pref medium
   2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
   local 2002:2:: dev lo  proto none  metric 0  pref medium
   local 2002:2::2 dev lo  proto none  metric 0  pref medium
   2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
   local fe80:: dev lo  proto none  metric 0  pref medium
   local fe80:: dev lo  proto none  metric 0  pref medium
   local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
   local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
   fe80::/64 dev eth1  proto kernel  metric 256  pref medium
   fe80::/64 dev eth2  proto kernel  metric 256  pref medium
D
David Ahern 已提交
282
   ff00::/8 dev red  metric 256  pref medium
283 284 285 286 287 288
   ff00::/8 dev eth1  metric 256  pref medium
   ff00::/8 dev eth2  metric 256  pref medium


8. Route Lookup for a VRF

D
David Ahern 已提交
289 290 291
   A test route lookup can be done for a VRF:
       $ ip [-6] route get vrf NAME ADDRESS
       $ ip [-6] route get oif NAME ADDRESS
292 293

   For example:
D
David Ahern 已提交
294 295
   $ ip route get 10.2.1.40 vrf red
   10.2.1.40 dev eth1  table red  src 10.2.1.2
296 297
       cache

D
David Ahern 已提交
298 299
   $ ip -6 route get 2002:1::32 vrf red
   2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317


9. Removing Network Interface from a VRF

   Network interfaces are removed from a VRF by breaking the enslavement to
   the VRF device:
       $ ip link set dev NAME nomaster

   Connected routes are moved back to the default table and local entries are
   moved to the local table.

   For example:
   $ ip link set dev eth0 nomaster

--------------------------------------------------------------------------------

Commands used in this example:

D
David Ahern 已提交
318 319 320 321 322
cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
1  mgmt
10 red
66 blue
81 green
323 324 325 326 327 328 329
EOF

function vrf_create
{
    VRF=$1
    TBID=$2

D
David Ahern 已提交
330 331
    # create VRF device
    ip link add ${VRF} type vrf table ${TBID}
332 333

    if [ "${VRF}" != "mgmt" ]; then
D
David Ahern 已提交
334
        ip route add table ${TBID} unreachable default
335
    fi
D
David Ahern 已提交
336
    ip link set dev ${VRF} up
337 338 339
}

vrf_create mgmt 1
D
David Ahern 已提交
340
ip link set dev eth0 master mgmt
341 342

vrf_create red 10
D
David Ahern 已提交
343 344 345
ip link set dev eth1 master red
ip link set dev eth2 master red
ip link set dev eth5 master red
346 347

vrf_create blue 66
D
David Ahern 已提交
348
ip link set dev eth3 master blue
349 350

vrf_create green 81
D
David Ahern 已提交
351
ip link set dev eth4 master green
352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399


Interface addresses from /etc/network/interfaces:
auto eth0
iface eth0 inet static
      address 10.0.0.2
      netmask 255.255.255.0
      gateway 10.0.0.254

iface eth0 inet6 static
      address 2000:1::2
      netmask 120

auto eth1
iface eth1 inet static
      address 10.2.1.2
      netmask 255.255.255.0

iface eth1 inet6 static
      address 2002:1::2
      netmask 120

auto eth2
iface eth2 inet static
      address 10.2.2.2
      netmask 255.255.255.0

iface eth2 inet6 static
      address 2002:2::2
      netmask 120

auto eth3
iface eth3 inet static
      address 10.2.3.2
      netmask 255.255.255.0

iface eth3 inet6 static
      address 2002:3::2
      netmask 120

auto eth4
iface eth4 inet static
      address 10.2.4.2
      netmask 255.255.255.0

iface eth4 inet6 static
      address 2002:4::2
      netmask 120