VPN & IPsec + Fortinet

A Virtual Private Network (VPN) is a concept that can be used for a secure communication between some nodes in a way that traffic flows through shared/public resources such as in Internet. It can provide Confidentiality, Authentication and Integrity if needed. The concept of VPN can be used for different scenarios such as for Remote Access (one side if Server and another side is Client) or between two sites which is usually called site-to-site (each side is a peer than can brings up the VPN connection or respond to the request of bringing up the connection).

There are several different protocols that VPN can use to meet user needs such as:

  • Internet Protocol Security (IPsec)
  • Transport Layer Security (SSL/TLS)
  • Secure Shell (SSH)

Our focus here is on IPsec which is widely being used specially for site-to-site connection.


IPsec Concepts

IPsec is basically a framework that provides a set of security services at Layer 3 (Network Layer). It can provide following services based on configuration:

  • Peer authentication
  • Data confidentiality
  • Data integrity
  • Data origin authentication

In order IPsec be able to provide above services, it uses many different protocols such as AH (Authentication Header), ESP (Encapsulating Security Payload) and SA (Security Association). What I find very interesting concept in IPsec is the usage of SA which I will explain in more details here.


Security Association (SA)

SA in a very simple term is a Package consist of several agreement (methods and parameters to be used in securing a communication channel) in unidirectional way between 2 entities. As an example, here we can tell what we expect from our IPsec channel such as Only authenticating the packets and its integrity, Only privacy through encryption or having both authentication and encryption and obviously providing what is needed like algorithms, keys and etc.

So in summary we can tell SA groups following parameters:

  • Algorithms and corresponding keys
  • Protocol mode (transport or tunnel)
  • Key management protocol (manual, IKE, KINK or IPSECKEY DNS)
  • SA life time

So far it’s clear that IPsec requires a protocol for handling SA. At the same time we need another protocol which can handle Key management in IPsec as part of SA. There is a protocol that meets our needs which is called ISAKMP (Internet Security Association and Key Management Protocol). Regarding the key management, ISAKMP does not handle the key exchange itself, but rely in another protocol for actual work which is called Internet Key Exchange (IKE). Following figure shows the relationship between them quite well:


So in order to avoid any confusion here, we can simply tell that IKE is the one which handle SA and as part of it also Key exchange. So technically IKE is responsible for secure SA negotiation as well as real data communication by handling the key exchange. There is 2 version of IKE which are IKEv1 and IKEv2. Following is how SA packet looks like:


As I mentioned earlier IPsec provides several services such as confidentiality, authentication and integrity. It meets mentioned features in 2 phase of communication:


Phase-1: known also as IKE phase-1

This phase use the IKE protocol to perform their functions. The IKE protocol authenticates the peer and then negotiates a compatible security policy before establishing the data tunnel. This phase facilitates:

  • IPSec Key Negotiation.
  • IPSec Key Renegotiation.
  • The exchange of control messages for maintaining data management tunnels for next phase.

This phase can negotiate in 2 different modes which are: (only for IKEv1)

Main mode

In main mode, the initiator and recipient send three two-way exchanges (six messages total) to accomplish the following services:

  •  First exchange (messages 1 and 2)—Proposes and accepts the encryption and authentication algorithms.
  •  Second exchange (messages 3 and 4)—Executes a DH exchange, and the initiator and recipient each provide a pseudorandom number.
  •  Third exchange (messages 5 and 6)—Sends and verifies the identities of the initiator and recipient.

The information transmitted in the third exchange of messages is protected by the encryption algorithm established in the first two exchanges. Thus, the participants’ identities are encrypted and therefore not transmitted “in the clear.”

Aggressive Mode

In aggressive mode, the initiator and recipient accomplish the same objectives as with main mode, but in only two exchanges, with a total of three messages:

  •   First message—The initiator proposes the security association (SA), initiates a DH exchange, and sends a pseudorandom number and its IKE identity.one set of the Phase 1 security parameters proposed and then process them.
  •   Second message—The recipient accepts the SA; authenticates the initiator; and sends a pseudorandom number, its IKE identity, and, if using certificates, the recipient’s certificate.
  •   Third message—The initiator authenticates the recipient, confirms the exchange, and, if using certificates, sends the initiator’s certificate.

Because the participants’ identities are exchanged in the clear (in the first two messages), aggressive mode does not provide identity protection.

After the participants have established a secure and authenticated channel, they proceed through Phase 2, in which they negotiate security associations (SAs) to secure the data to be transmitted through the IPsec tunnel.

Although Main mode is more secure, you must select Aggressive mode if there is more than one dialup Phase 1 configuration for the interface IP address, and the remote VPN peer or client is authenticated using an identifier local ID. Aggressive mode might not be as secure as Main mode, but the advantage to Aggressive mode is that it is faster than Main mode (since fewer packets are exchanged). Aggressive mode is typically used for remote access VPNs. But you would also use aggressive mode if one or both peers have dynamic external IP addresses. Descriptions of the peer options in this guide indicate whether Main or Aggressive mode is required.

IKE Phase 1 is successful only when the following are true:

  • Each peer negotiates a matching IKE SA policy.
  • Each peer is authenticated and their identities protected.
  • The Diffie-Hellman exchange is authenticated (the pre-shared secret keys match).


Phase-2: known also as IKE phase-2

This phase is responsible to secure data traffic by using the Authentication Header (AH) protocol or the Encapsulated Security Protocol (ESP) to perform their operations. So basically this phase facilitate following features:

  • Data integrity.
  • Data confidentiality.

There are 2 methods of operation in this phase which are:

  • Tunnel mode: The whole original packet will be encrypted and as a result new IP headers needed.
  • Transport mode: Only the data (payload) is encrypted and original headers are untouched.

It is very important to understand that the format of the IKE packets are same for Phase 1 and Phase 2. We need to keep in mind that if in Phase 2, SA specifies the ESP header in tunnel mode, then the whole packet will be encrypted and in order the packet be able to being routed to the destination, new headers will be added to the packet.

The encryption basically happens in both phase 1 and phase 2. In phase 1 happens to secure the negotiation between 2 parties and in phase 2 happens to encrypt the real data if ESP has been used. However we should keep it in mind that encryption that happens in Phase1 or 2 is from Symmetric type which need a shared key between 2 parties. In most of the case it happens automatically through some algorithms such as Diffie-Helman. There are usually many different encryption algorithms for encryption such as:

  • DES
  • 3DES
  • AES 128/192/256
  • ChaCha20


IPsec in Fortinet Firewall

Most of the above concepts are also exist in Fortinet units. I highlight some of the unique features of VPN in Fortinet which might exist only here.

Types of VPN in Fortigate Unit

There are Route-based and Policy-based which both has Phase 1 and Phase 2.

a. Route-based policy (interface-based vpn):

It creates a virtual IPsec network interface that applies encryption or decryption as needed to any traffic that it carries. Here you create two security policies between the virtual IPsec interface and the interface that connects to the private network. In one policy, the virtual interface is the source. In the other policy, the virtual interface is the destination. This creates bidirectional policies that ensure traffic will flow in both directions over the VPN. Here we have only NAT mode

b. Policy-base policy (Tunnel-mode vpn)

one security policy enables communication in both directions. You enable inbound and outbound traffic as needed within that policy, or create multiple policies of this type to handle different types of traffic differently. For example HTTPS traffic may not require the same level of scanning as FTP traffic. Both Transparent and NAT modes.

The Main difference between 2 above is with Security Policy:

In Route-based VPN we create a regular ACCEPT security policy to permit traffic to flow between the virtual IPsec interface and another network interface. And lastly, configure a static route to allow traffic over the  VPN.

Here in Fortinet, Phase 1 is called the IKE Policy. Phase 2 is called the IPsec Policy.

What is important here is following concept:

Diffie-Hellman algorithm is being used in both Phase 1 and 2 for key exchange. It doesn’t have anything to do with Authentication and Encryption that happens in both Phase. So let’s clear this up:

Authentication and Encryption

We need to assure that only authorized users can have access to VPN network. There are 2 solutions in Fortinet for this purpose: pre-shared key (shared secret) or RSA certificates. We use normally the shared secret for authentication. This shared secret never being sent in line by either side. It being used also for creating the symmetric Encryption key and the Diffie-Hellman to exchange it.

Also we can use some hash algorithm to check the authenticity of messages during the encrypted session such as MD5 or SHA1.



 There are different DH group which each use different module bits. The default group is 14 which provide sufficient protection for stronger cipher with including AES and SHA2.

IKE negotiation TimeOut/expiration

There are several options in configuration of both phase 1 and phase 2 which are related to the concept of re-negotiation that might needed for several reasons such as:


  1. Keylife: simply how long IKE encryption key can be used
  2. Dead Peer Detection (DPD): if this enabled, it will keep sending probe message to the other peer to detect if the communication link between them has an issue and re-establish the IKE negotiation automatically. You can use this option to receive notification whenever a tunnel goes up or down, or to keep the tunnel connection open when no traffic is being generated inside the tunnel.
  3. Perfect forward secrecy (PFS): if enabled it will improve the security by forcing a new DH exchange whenever keylife expires.
  4. Autokey Keep Alive: enabled if you want the tunnel to remain active when no data is being processed.
  5. Auto-negotiate: Enable the option if you want the tunnel to be automatically renegotiated when the tunnel expires.


Security Policies

Here we define some policies to have control over our traffics and letting some special traffics to be sent in VPN channel. Based on what VPN types we have chosen, we need to create corresponding polices. Since the Route-based policy is the one which usually being used, I ignore the Policy-based vpn.

 A route-based VPN

Route-based VPN is created by creating a “virtual IPsec Interface” and then creating a regular Accept policy to allow traffics to flow between “virtual IPsec Interface” and another network interface. For each direction (outgoing & incoming) of VPN connection, we need an accept security policy. In one policy, the “virtual IPsec interface” is the source. In the other policy, the “virtual IPsec interface” is the destination. This creates bidirectional policies that ensure traffic will flow in both directions over the VPN.

Outgoing Traffic:

Incoming Interface:  the interface that connects to the private network behind this FortiGate unit.

Outgoing Interface:  IPsec Interface you configured.

Source:                      the address name that you defined for the private network behind this FortiGate unit.

Destination Address: the address name that you defined for the private network behind the remote peer.

Action: Select ACCEPT.

NAT:  Disable NAT


Incoming Trrafics:

Incoming Interface:  the IPsec Interface you configured.

Outgoing Interface:  the interface that connects to the private network behind this FortiGate unit.

Source:                      the address name that you defined for the private network behind the remote peer.

Destination Address: the address name that you defined for the private network behind this FortiGate unit.

Action: ACCEPT.

NAT:  Disable NAT.



So to have a summary over all above concepts, we can say following general configuration should be performed by both side:

  • Define the Phase 1 parameters that the FortiGate unit needs to authenticate the remote peer and establish a secure connection.
  • Define the Phase 2 parameters that the FortiGate unit needs to create a VPN tunnel with the remote peer.
  • Create security policies to control the permitted services and permitted direction of traffic between the IP source and destination addresses.


%d bloggers like this: