背景:本防火墙为职场出口墙,自身作为DHCP服务器为终端分配地址,
上联通过与IDC防火墙建立点到点智能选路的IPSec,下联到交换机再到AP,
用户通过连接SSID,访问IDC侧业务并在AC-Campus(SC1为一层楼正常用户认证的,SC2为二层楼用户认证的)上线。
SSID:HW
IP-POOL:10.14.208.1-10.14.208.127
终端连接HW获取到10.14.208.40,但是无法访问业务。

步骤 1、 本端防火墙上开启用户的反查debug,并抓取到控制器SC2的数据包

开启反查debug
sys
diag
debugging user-manage sso ipv4 10.14.208.40 all 
t m
t d

(1)debug中只有反查的查询消息,但是没有收到控制器发的用户上线消息:
image.png

(2) 防火墙抓包没有收到控制器SC2发的上线消息:
在wireshark打开.cap包后,筛选 udp.srcport == 1819 ,未看到包

步骤 2、 在控制器抓包确认控制器SC2已发出用户上线消息。

步骤 3、 在防火墙查看控制器SC1有对应会话,且mac地址是0000-0000-0000,说明是走ipsec隧道的。但是并没有控制器SC2的对应会话信息。

[职场-diagnose]display firewall session table verbose source inside x.x.x.x(SC1.IP)
2021-05-11 19:48:02.380 +08:00
 Current Total Sessions : 1
 udp  VPN: public --> public  ID: *****
 Zone: untrust --> local  TTL: 00:02:00  Left: 00:00:02
 Recv Interface: GigabitEthernet1/0/0
 Interface: InLoopBack0  NextHop: 127.0.0.1  MAC: 0000-0000-0000
 <--packets: 9 bytes: 396 --> packets: 9 bytes: 1,100
 SC1.IP:1819 --> (FW-MGMT.IP):8001 PolicyName: Radius_SC_to_MGMT

[职场-diagnose]display firewall session table verbose source inside x.x.x.x(SC2.IP)
2021-05-11 19:48:17.820 +08:00
 Current Total Sessions : 0

步骤 4、 查看本职场防火墙的ipsec隧道信息,符合感兴趣数据流的隧道对端地址公网IP

display ipsec sa 
2021-05-11 21:42:15.140 +08:00

Interface: GigabitEthernet1/0/0

  -----------------------------
  IPSec policy name: "ipsec*****"
  Sequence number  : 1
  Acl group        : ****/IPv4
  Acl rule         : 10
  Mode             : ISAKMP
  -----------------------------
    Connection ID     : *****
    Encapsulation mode: Tunnel
    Holding time      : 0d 12h 24m 17s
    Tunnel local      : 192.168.10.101/4500
    Tunnel remote     : 公网IP/4500
    Flow source       : 终端地址段/255.255.254.0 0/0-65535
    Flow destination  : SC.IP/255.240.0.0 0/0-65535

    [Outbound ESP SAs] 
      SPI: 18500**** (0xb06****)
      Proposal: ESP-ENCRYPT-AES-256 ESP-AUTH-SHA2-256-128
      SA remaining soft duration (kilobytes/sec): 4229492/1294
      SA remaining hard duration (kilobytes/sec): 5015924/1833
      Max sent sequence-number: 834299    
      UDP encapsulation used for NAT traversal: Y 

步骤 5、 登录ipsec隧道对端IDC防火墙排查,发现IDC防火墙已经收到了控制器SC2发的用户上线消息,出接口为UNR等价路由,走GE1/0/8时正常,走GE3/0/9时异常。

HRP_M<˾FW_M>display firewall session table verbose source inside SC1.IP destination inside  职场FW-MGMT.IP
2021-05-11 21:17:38.370 +08:00
 Current Total Sessions : 1
 udp  VPN: public --> public  ID: *****
 Zone: trust --> untrust  TTL: 00:02:00  Left: 00:01:50
 Recv Interface: GigabitEthernet1/0/9
 Interface: GigabitEthernet1/0/8  NextHop: ****  MAC: 0000-0000-0000
 <--packets: 4 bytes: 176 --> packets: 4 bytes: 480
 SC2.IP:1819 --> 职场FW-MGMT.IP:8001 PolicyName: AgileControllerNew_SC_To_

HRP_M<˾FW_M>display firewall session table verbose source inside SC2.IP destination inside  职场FW-MGMT.IP
2021-05-11 21:17:49.050 +08:00
Current Total Sessions : 1
udp  VPN: public --> public  ID: *****
Zone: trust --> untrust  TTL: 00:02:00  Left: 00:01:59
Recv Interface: GigabitEthernet1/0/9
Interface: GigabitEthernet3/0/9  NextHop: *****  MAC: 0000-0000-0000
<--packets: 0 bytes: 0 --> packets: 6113 bytes: 1,033,804
SC2.IP:1819 --> 职场FW-MGMT.IP:8001 PolicyName: AgileControllerNew_SC_To_

HRP_M<˾FW_M>display ip routing-table 职场FW-MGMT.IP    //等价路由,走GE3/0/9这条隧道时不通
2021-05-11 21:18:49.070 +08:00
Route Flags: R - relay, D - download to fib
------------------------------------------------------------------------------
Routing Table : Public
Summary Count : 2
Destination/Mask    Proto   Pre  Cost      Flags NextHop         Interface

   职场FW-MGMT.IP段/23  Unr     70   0           D   ****   GigabitEthernet3/0/9
                    Unr     70   0           D   ****   GigabitEthernet1/0/8


步骤 6、 查看IDC,发现到职场存在两条隧道,两条隧道分别在两个ipsec模板中。但职场却只有一条隧道,分析怀疑是IDC的隧道有残留导致。

IDC的GE1/0/8上对应的隧道:
interface GigabitEthernet1/0/8
undo shutdown
ip address x.x.x.x 255.255.255.248
vrrp vrid 60 virtual-ip 公网地址 255.255.255.192 active
alias WAN
gateway 公网网关 route disable
undo service-manage enable
redirect-reverse next-hop 公网网关
ipsec policy ipsec91213*****


-----------------------------
  IPSec policy name: "ipsec91213****"
  Sequence number  : 10000
  Acl group        : ****/IPv4
  Acl rule         : 160
  Mode             : Template
  -----------------------------
    Connection ID     : 636**
    Encapsulation mode: Tunnel
    Holding time      : 0d 12h 18m 11s
    Tunnel local      : 公网地址/4500
    Tunnel remote     : 职场公网地址/30545
    Flow source       : SC地址段/255.240.0.0 0/0-65535
    Flow destination  : 终端地址段/255.255.254.0 0/0-65535

    [Outbound ESP SAs] 
      SPI: 19987**** (0xbe9e****)
      Proposal: ESP-ENCRYPT-AES-256 ESP-AUTH-SHA2-256-128
      SA remaining soft duration (kilobytes/sec): 0/530824
      SA remaining hard duration (kilobytes/sec): 0/603399
      Max sent sequence-number: 774519
      UDP encapsulation used for NAT traversal: Y
      SA encrypted packets (number/bytes): 774518/384862382

    [Inbound ESP SAs] 
      SPI: 18500**** (0xb06****)
      Proposal: ESP-ENCRYPT-AES-256 ESP-AUTH-SHA2-256-128
      SA remaining soft duration (kilobytes/sec): 0/488488
      SA remaining hard duration (kilobytes/sec): 0/603399
      Max received sequence-number: 661568
      UDP encapsulation used for NAT traversal: Y
      SA decrypted packets (number/bytes): 661567/183957883
      Anti-replay : Enable
      Anti-replay window size: 1024


IDC的GE3/0/9上对应的隧道:
interface GigabitEthernet3/0/9
undo shutdown
ip address x.x.x.x 255.255.255.248
vrrp vrid 70 virtual-ip 公网地址2 255.255.255.224 active
alias  gateway 公网地址2网关 route disable
undo service-manage enable
redirect-reverse next-hop 公网地址2网关
ipsec policy ipsec_yu****

  -----------------------------
  IPSec policy name: "ipsec_yun****"
  Sequence number  : 10000
  Acl group        : ****/IPv4
  Acl rule         : 160
  Mode             : Template
  -----------------------------
    Connection ID     : 6***
    Encapsulation mode: Tunnel
    Holding time      : 0d 19h 52m 55s
    Tunnel local      : x.x.x./4500
    Tunnel remote     : 职场公网/31789
    Flow source       : SC地址段/255.240.0.0 0/0-65535
    Flow destination  : 终端地址段/255.255.254.0 0/0-65535

    [Outbound ESP SAs] 
      SPI: 20125**** (0xbfe****)
      Proposal: ESP-ENCRYPT-AES-256 ESP-AUTH-SHA2-256-128
      SA remaining soft duration (kilobytes/sec): 0/460909
      SA remaining hard duration (kilobytes/sec): 0/557676
      Max sent sequence-number: 2768467
      UDP encapsulation used for NAT traversal: Y
      SA encrypted packets (number/bytes): 2768466/1288507978

    [Inbound ESP SAs] 
      SPI: 19793**** (0xbcc****)
      Proposal: ESP-ENCRYPT-AES-256 ESP-AUTH-SHA2-256-128
      SA remaining soft duration (kilobytes/sec): 0/454861
      SA remaining hard duration (kilobytes/sec): 0/557676
      Max received sequence-number: 2293202
      UDP encapsulation used for NAT traversal: Y
      SA decrypted packets (number/bytes): 2292494/808281229
      Anti-replay : Enable
      Anti-replay window size: 1024


步骤 7、 查看职场防火墙,在5月11日9点17分出现了ispec智能选路链路的切换,职场上老的隧道删除,但是IDC对应的老隧道并没有删除。

May 11 2021 09:17:55+08:00 职场 %%01IPSEC/5/IPSEC_TUNNEL_TEARED_DOWN(l)[1415179]:Vsys public: An IPSec tunnel is teared down. (PolicyName=policyf****, IfIndex=7, SeqNum=4407, RuleNum=10, 
May 11 2021 09:17:58+08:00 职场 %%01IPSEC/5/IPSEC_TUNNEL_ESTABLISHED(l)[1415188]:Vsys public: An IPSec tunnel is established. (PolicyName=ipsec9122*****, IfIndex=7, SeqNum=1, RuleNum=10, SrcIP=职场出口IP, DstIP=IDC公网地址, Slot=11, CpuID=0, State=Normal, Role=Initiator)

IDC只有新的隧道建立的记录,没有老的隧道删除的记录:
May 11 2021 09:20:49+08:00 FW_M %%01IPSEC/5/IPSEC_TUNNEL_ESTABLISHED(l)[521051]:Vsys public: An IPSec tunnel is established. (PolicyName=ipsec9121****, IfIndex=14, SeqNum=10000, RuleNum=160, SrcIP=IDC公网地址, DstIP=职场公网地址, Slot=11, CpuID=0, State=Normal, Role=Responder)

步骤 8、 在IDC上重置下该隧道后,残留的隧道删除,业务测试正常。

HRP_M<˾FW_M> reset ike sa  remote 职场公网地址 
Warning: This operation may delete the corresponding SA, Continue? [Y/N]:y
HRP_M<˾FW_M>
HRP_M<˾FW_M>display ike sa  remote 职场公网地址
残留的隧道被清除

根本原因:

职场是ipsec智能选路,职场的ipsec隧道已经切换了,
但是IDC的ipsec隧道有残留,
导致IDC作为模块方生成了两条内层报文的等价路由,
从而控制器SC2发往防火墙的上线消息在IDC的墙上进入了残留的ipsec隧道中,
上线消息无法到达分支,职场的防火墙无法收到控制器SC2发的用户上线消息,
从而导致部分用户无法上线。

Y.


    Network engineer