1

I have a dedicated server hosted in a local data center and VMware Esxi 7 is installed on it . I recently ordered an IPv6 range and as soon as they told me that the IPv6 range is activated , I noticed that all VMs within VMware Esxi have automatically received an IPv6 without me setting up anything in VMware Esxi or on the guest operating systems such as an Ubuntu 22 VM .

I noticed that my IPv6 connectivity is lost on Ubuntu 22 VM (or any other VMs) after exactly 30 minutes! After I reboot the VM , IPv6 starts working and stops working again at exactly 30 minutes.

This is the output of ip -6 route command in Ubuntu 22 VM when IPv6 is working :

root@testsocat:~# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2a05:1a18:0:12::/64 dev ens160 proto ra metric 1024 expires 2591592sec pref medium
fe80::/64 dev ens160 proto kernel metric 256 pref medium
default via fe80::82e0:1dff:fe6f:3e00 dev ens160 proto ra metric 1024 expires 1392sec mtu 1500 pref medium

This is the output of ip -6 route command in Ubuntu 22 VM when IPv6 is not working and has disappeared after 30 minutes :

root@testsocat:~# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2a05:1a18:0:12::/64 dev ens160 proto ra metric 1024 expires 2589957sec pref medium
fe80::/64 dev ens160 proto kernel metric 256 pref medium

As you can see this entry is removed after 30 minutes :

default via fe80::82e0:1dff:fe6f:3e00 dev ens160 proto ra metric 1024 expires 1392sec mtu 1500 pref medium

I then manually added the gateway route back with this command :

ip -6 route add default via fe80::82e0:1dff:fe6f:3e00 dev ens160

By doing this , the IPv6 starts working again and doesn't stop working even after 30 minutes !

The data center is insisting their IPv6 range configuration is OK and problem is on our side . However , I have tested this on multiple VMs ( Ubuntu , Centos and Windows ) . It happens on all of them . Also I don't see any settings on VMware that can cause IPv6 addresses to stop working after 30 minutes .

Can anyone help me understand what is wrong here ? Why does the automatically assigned IPv6 disappears from VMs at exactly 30 minutes? Why does it work again when I reboot the VM ? Why doesn't it stop working when I manually add the gateway route back in the guest OS ?

I have another dedicated server with the same setup in another datacenter that has an IPv6 range but doesn't have this issue . Have they messed up any parameters while configuring my IPv6 range?

I would be really thankful if you could help me understand what's wrong and fix it . I have been dealing with this for days now .

EDIT 1 : Using "rdisc6 ens160" , I was able to get the default route to show up again . Here is the output of the command in full :

root@testsocat:~# rdisc6 ens160
Soliciting ff02::2 (ff02::2) on ens160...

Hop limit                 :           64 (      0x40)
Stateful address conf.    :           No
Stateful other conf.      :           No
Mobile home agent         :           No
Router preference         :       medium
Neighbor discovery proxy  :           No
Router lifetime           :         1800 (0x00000708) seconds
Reachable time            :  unspecified (0x00000000)
Retransmit time           :  unspecified (0x00000000)
 Source link-layer address: 80:E0:1D:6F:3E:00
 MTU                      :         1500 bytes (valid)
 Prefix                   : 2a05:1a18:0:12::/64
  On-link                 :          Yes
  Autonomous address conf.:          Yes
  Valid time              :      2592000 (0x00278d00) seconds
  Pref. time              :       604800 (0x00093a80) seconds
 from fe80::82e0:1dff:fe6f:3e00
root@testsocat:~# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2a05:1a18:0:12::/64 dev ens160 proto ra metric 1024 expires 2591981sec pref medium
fe80::/64 dev ens160 proto kernel metric 256 pref medium
default via fe80::82e0:1dff:fe6f:3e00 dev ens160 proto ra metric 1024 expires 1781sec mtu 1500 pref medium

EDIT 2 : using "tshark -n -i ens160 -f icmp6 -Y "icmpv6.type == 134" -VO "frame,icmpv6" , This is what I got :

 ** (tshark:5007) 00:28:32.821551 [Main MESSAGE] -- Capture started.
 ** (tshark:5007) 00:28:32.822018 [Main MESSAGE] -- File: "/tmp/wireshark_ens160                                                                                                                                                             7OI6P2.pcapng"
Frame 2: 118 bytes on wire (944 bits), 118 bytes captured (944 bits) on interfac                                                                                                                                                             e ens160, id 0
    Interface id: 0 (ens160)
        Interface name: ens160
    Encapsulation type: Ethernet (1)
    Arrival Time: Jul 10, 2024 00:29:25.087268913 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1720571365.087268913 seconds
    [Time delta from previous captured frame: 0.002588177 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.002588177 seconds]
    Frame Number: 2
    Frame Length: 118 bytes (944 bits)
    Capture Length: 118 bytes (944 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ipv6:icmpv6]
Ethernet II, Src: 80:e0:1d:6f:3e:00, Dst: 00:0c:29:ea:8d:6f
Internet Protocol Version 6, Src: fe80::82e0:1dff:fe6f:3e00, Dst: fe80::6f36:a5e                                                                                                                                                             0:992f:72bd
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xff54 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x00, Prf (Default Router Preference): Medium
        0... .... = Managed address configuration: Not set
        .0.. .... = Other configuration: Not set
        ..0. .... = Home Agent: Not set
        ...0 0... = Prf (Default Router Preference): Medium (0)
        .... .0.. = Proxy: Not set
        .... ..0. = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 80:e0:1d:6f:3e:00)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: 80:e0:1d:6f:3e:00
    ICMPv6 Option (MTU : 1500)
        Type: MTU (5)
        Length: 1 (8 bytes)
        Reserved
        MTU: 1500
    ICMPv6 Option (Prefix information : 2a05:1a18:0:12::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 2592000
        Preferred Lifetime: 604800
        Reserved
        Prefix: 2a05:1a18:0:12::

EDIT 3 : A second RA packet has arrived ! the timestamp is interesting . First one was sent at 00:29:25 , Second one is sent at 01:04:24 . There is a 35 minutes gap between them and router lifetime is 30 minutes !

Frame 45: 118 bytes on wire (944 bits), 118 bytes captured (944 bits) on interface ens160, id 0
    Interface id: 0 (ens160)
        Interface name: ens160
    Encapsulation type: Ethernet (1)
    Arrival Time: Jul 10, 2024 01:04:24.430745189 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1720573464.430745189 seconds
    [Time delta from previous captured frame: 0.002469817 seconds]
    [Time delta from previous displayed frame: 2099.343476276 seconds]
    [Time since reference or first frame: 2099.346064453 seconds]
    Frame Number: 45
    Frame Length: 118 bytes (944 bits)
    Capture Length: 118 bytes (944 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ipv6:icmpv6]
Ethernet II, Src: 80:e0:1d:6f:3e:00, Dst: 00:0c:29:12:f4:07
Internet Protocol Version 6, Src: fe80::82e0:1dff:fe6f:3e00, Dst: fe80::905f:89e5:45e4:2838
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0x97f7 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x00, Prf (Default Router Preference): Medium
        0... .... = Managed address configuration: Not set
        .0.. .... = Other configuration: Not set
        ..0. .... = Home Agent: Not set
        ...0 0... = Prf (Default Router Preference): Medium (0)
        .... .0.. = Proxy: Not set
        .... ..0. = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 80:e0:1d:6f:3e:00)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: 80:e0:1d:6f:3e:00
    ICMPv6 Option (MTU : 1500)
        Type: MTU (5)
        Length: 1 (8 bytes)
        Reserved
        MTU: 1500
    ICMPv6 Option (Prefix information : 2a05:1a18:0:12::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 2592000
        Preferred Lifetime: 604800
        Reserved
        Prefix: 2a05:1a18:0:12::

EDIT 4 : OK , I'm now confused ! the RA packets 3 and 4 have arrived and they are sent sooner this time . third packet arrived at 01:26:15 ( 22 minutes gap between this one and previous one ) and the fourth packet arrived at 01:30:35 ( 4 minutes after the 3rd one ) .

Frame 50: 118 bytes on wire (944 bits), 118 bytes captured (944 bits) on interface ens160, id 0
    Interface id: 0 (ens160)
        Interface name: ens160
    Encapsulation type: Ethernet (1)
    Arrival Time: Jul 10, 2024 01:26:15.149204151 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1720574775.149204151 seconds
    [Time delta from previous captured frame: 0.001792901 seconds]
    [Time delta from previous displayed frame: 1310.718458962 seconds]
    [Time since reference or first frame: 3410.064523415 seconds]
    Frame Number: 50
    Frame Length: 118 bytes (944 bits)
    Capture Length: 118 bytes (944 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ipv6:icmpv6]
Ethernet II, Src: 80:e0:1d:6f:3e:00, Dst: 00:0c:29:78:ad:1b
Internet Protocol Version 6, Src: fe80::82e0:1dff:fe6f:3e00, Dst: fe80::402b:9e4e:408f:3ae4
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xc66b [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x00, Prf (Default Router Preference): Medium
        0... .... = Managed address configuration: Not set
        .0.. .... = Other configuration: Not set
        ..0. .... = Home Agent: Not set
        ...0 0... = Prf (Default Router Preference): Medium (0)
        .... .0.. = Proxy: Not set
        .... ..0. = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 80:e0:1d:6f:3e:00)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: 80:e0:1d:6f:3e:00
    ICMPv6 Option (MTU : 1500)
        Type: MTU (5)
        Length: 1 (8 bytes)
        Reserved
        MTU: 1500
    ICMPv6 Option (Prefix information : 2a05:1a18:0:12::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 2592000
        Preferred Lifetime: 604800
        Reserved
        Prefix: 2a05:1a18:0:12::

Frame 55: 118 bytes on wire (944 bits), 118 bytes captured (944 bits) on interface ens160, id 0
    Interface id: 0 (ens160)
        Interface name: ens160
    Encapsulation type: Ethernet (1)
    Arrival Time: Jul 10, 2024 01:30:35.107428556 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1720575035.107428556 seconds
    [Time delta from previous captured frame: 0.002612786 seconds]
    [Time delta from previous displayed frame: 259.958224405 seconds]
    [Time since reference or first frame: 3670.022747820 seconds]
    Frame Number: 55
    Frame Length: 118 bytes (944 bits)
    Capture Length: 118 bytes (944 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ipv6:icmpv6]
Ethernet II, Src: 80:e0:1d:6f:3e:00, Dst: 00:0c:29:ea:8d:6f
Internet Protocol Version 6, Src: fe80::82e0:1dff:fe6f:3e00, Dst: fe80::6f36:a5e0:992f:72bd
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xff54 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0x00, Prf (Default Router Preference): Medium
        0... .... = Managed address configuration: Not set
        .0.. .... = Other configuration: Not set
        ..0. .... = Home Agent: Not set
        ...0 0... = Prf (Default Router Preference): Medium (0)
        .... .0.. = Proxy: Not set
        .... ..0. = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 80:e0:1d:6f:3e:00)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: 80:e0:1d:6f:3e:00
    ICMPv6 Option (MTU : 1500)
        Type: MTU (5)
        Length: 1 (8 bytes)
        Reserved
        MTU: 1500
    ICMPv6 Option (Prefix information : 2a05:1a18:0:12::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 2592000
        Preferred Lifetime: 604800
        Reserved
        Prefix: 2a05:1a18:0:12::
Improve this question
8
  • Can you install tshark (Wireshark CLI) and do a capture for tshark -n -i ens160 -f icmp6 -Y "icmpv6.type == 134" -VO "frame,icmpv6"? (Yes the icmp6 and icmpv6 are spelled differently.) Leave it running for the half hour, and note down when the "Router Advertisement" packets arrive and what "Router Lifetime" they indicate – the datacenter needs to be sending them every 1/2~1/3 of that lifetime.
    – u1686_grawity
    Commented Jul 10 at 0:13
  • Can you also install ndisc6 and test whether manually requesting a Router Advertisement using rdisc6 ens160 causes the "default" route to show up automatically? (Basically, if the route doesn't show up even if RAs arrive and have a nonzero Lifetime, it's likely your server's config issue; whereas if RAs don't arrive on schedule but manually requesting one does cause the route to show up, that's likely a network issue.)
    – u1686_grawity
    Commented Jul 10 at 0:15
  • @u1686_grawity thank you for your answer . I ran rdisc6 ens160 and posted the output in the original post under EDIT 1 section . please have a look .
    – lostsoul6
    Commented Jul 10 at 0:27
  • @u1686_grawity also posted the tshark output under EDIT 2 in the original post . please have a look .
    – lostsoul6
    Commented Jul 10 at 0:39
  • 1
    @u1686_grawity thank you so much for your help , They finally solved my issue . They told me the issue was that their Cisco routers had global configuration for "ipv6 nd ra-interval" but my range wasn't using that . They said they added it specifically under my IPv6 range settings to override global values and it worked .
    – lostsoul6
    Commented Jul 10 at 12:13

1 Answer 1

Reset to default
2

The default route in IPv6 is set up whenever the host receives a Router Advertisement (RA). The datacenter router broadcasts RAs at a certain interval – e.g. if the RA has a "lifetime" of 1 hour then it is broadcast every 20–40 minutes, and without any RAs your default route would expire in 1 hour.

Initially, when the interface just comes up, the host asks for an immediate RA by sending a Router Solicitation, but from that point it expects RAs to arrive on schedule. That is – hosts do not send periodic Solicitations; instead the router is supposed to periodically broadcast a new RA before the "lifetime" of the previous RA ends, whether any host asked for it or not.

Things to try:

  • Install ndisc6 and run rdisc6 to send off a Router Solicitation. If this gets a RA in response, and if your host immediately sets up a default route again, then it mostly rules out any host-side issues – but if the host ignores the received RA, it might be a firewall or sysctl issue.

    rdisc6 ens160

    (Don't crontab this except as a last resort.)

  • Install tshark (Wireshark CLI) to capture all Router Advertisements that arrive, and note the relative time between them. (This could also be done using tcpdump, but I don't remember the libpcap filter for RAs only, so I use tshark because it also supports Wireshark display filters.)

    tshark -n -i ens160 -f icmp6 -Y "icmpv6.type == 134"

    For a more detailed view that shows the contents of each RA:

    tshark -n -i ens160 -f icmp6 -Y "icmpv6.type == 134" -VO "frame,icmpv6"

    As mentioned, you should expect to see unsolicited RAs arrive at around every 1/3rd to 2/3rds the Router Lifetime (slightly randomized). If the interval between RAs is too long, that is definitely a router-side problem.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .