How to install GPFS Cluster on RHEL (Part 1)

GPFS stands for General Parallel File System is a cluster file system developed by IBM. It’s named IBM Storage Scale and previously IBM Spectrum Scale. It’s a cluster file system that provides concurrent access to a single file system or set of file systems from multiple nodes. The nodes can be SAN attached, network attached, a mixture of SAN attached and network attached. This enables high performance access to this common set of data to support a scale-out solution or to provide a high availability platform.

It has many features including data replication, policy based storage management, and multi-site operations, etc,. It can share data within a location (LAN) or across wide area network (WAN) connections.

It supports very huge number of nodes (9500+) as per IBM FAQ page and it’s used by many of the world’s largest commercial companies, as well as some of the supercomputers.

Also, a GPFS cluster can be set up with a mix of cross-platforms such as Linux and AIX. It supports Linux (RHEL, Ubuntu and SUSE), AIX and Windows platforms but the number of nodes supported varies based on the operating system.

It supports 2048 disks in a file system, maximum supported GPFS disk size is 2TB, and size of the filesystem is 2^99 bytes.

In this guide, we will demonstrate how to install GPFS cluster File System 5.1.8 on RHEL system.

  • Part-1 How to Install and Configure GPFS Cluster on RHEL
  • Part-2 How to Create GPFS Filesystem on RHEL
  • Part-3 How to Extend a GPFS Filesystem on RHEL
  • Part-4 How to Export a GPFS Filesystem in RHEL
  • Part-5 How to Upgrade GPFS Cluster on RHEL

Our lab setup:

  • Two Node GPFS cluster with RHEL 8.8
  • Node1 – 2ggpfsnode01 –
  • Node2 – 2ggpfsnode01 –
  • IBM Storage Scale Standard Edition License

Prerequisites for GPFS

  • Minimum three nodes required to setup a GPFS cluster with default Node quorum algorithm.
  • If you want to run a small GPFS cluster (a two-node GPFS cluster system), you need at least three tiebreaker disks, which is called a Node quorum with tiebreaker disks.
  • Two network interfaces are required, one for host communication and another for GPFS cluster intercommunication (Private IP).
  • Make sure the ‘/var’ partition has a maximum size of ’10GB’ as GPFS related binaries and packages will be placed there.
  • All SAN-attached disks should be mapped in a shared mode between nodes, including tiebreaker disks.
  • Download IBM Storage Scale package from Fix Central. It requires active IBM account and license for GPFS.
  • Check kernel compatibility before installing/upgrading a specific version of GPFS on a RHEL system. This information can be found IBM Storage Scale QA page.

1) Configuring /etc/hosts

Although DNS name resolution is in place. It’s recommended to configure hosts files on both nodes to allow faster name resolution (On both the Nodes).

cat /etc/hosts

#######Host IPs Configuration Start############	2ggpfsnode01.local 2ggpfsnode01	2ggpfsnode02.local 2ggpfsnode02
#######Host IPs Configuration  END#############

Similarly configure the hosts file for the private IP (On both the Nodes).

cat /etc/hosts

#######GPFS Private IPs Start########### 2ggpfsnode01-gpfs 2ggpfsnode02-gpfs
#######GPFS Private IPs END#############

2) Configuring Private IP in Network Interface

GPFS cluster communication goes through private IP between nodes, so configure private IP as another network interface as shown below. (On both the Nodes).

vi /etc/sysconfig/network-scripts/ifcfg-ens256


2a) Restarting Network

Once the private IP is configured, restart the network interface using the below command (On both the Nodes).

ifdown ens256
ifup ens256

2b) Checking IP Output

Check if the private IP is configured on another network interface (On both the Nodes).

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:97:13:2e brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe97:132e/64 scope link
       valid_lft forever preferred_lft forever
3: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:97:13:9d brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic noprefixroute ens256
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe97:132e/64 scope link
       valid_lft forever preferred_lft forever

2c) Setup password-less authentication

For any cluster setup, password-less authentication between cluster nodes is mandatory for various operations, so we will configure the same. In GPFS, each gpfs cluster node can run ssh & scp commands on all other nodes using root user to allow remote administration of other nodes.

[root@2ggpfsnode01 ~]# ssh-copy-id root@2ggpfsnode01   #For Self on Node-1
[root@2ggpfsnode01 ~]# ssh-copy-id root@2ggpfsnode02   #For Other Node (From Node-1 to Node-2)
[root@2ggpfsnode02 ~]# ssh-copy-id root@2ggpfsnode01   #For Other Node (From Node-2 to Node-1)
[root@2ggpfsnode02 ~]# ssh-copy-id root@2ggpfsnode02   #For Self on Node-2

2d) Validating password-less authentication

Check if password-less authentication is working as expected for self and other nodes.

[root@2ggpfsnode01 ~]# ssh 2ggpfsnode01 date
Sat Nov 18 08:13:20 IST 2023

[root@2ggpfsnode01 ~]# ssh 2ggpfsnode02 date
Sat Nov 18 08:13:21 IST 2023

[root@2ggpfsnode02 ~]# ssh 2ggpfsnode01 date
Sat Nov 18 08:13:25 IST 2023

[root@2ggpfsnode02 ~]# ssh 2ggpfsnode02 date
Sat Nov 18 08:13:27 IST 2023

3) Installing GPFS RPMs

Now the environment is ready, let’s proceed with the GPFS installation. It can be installed in two methods like Manual or installation toolkit, but in this article we will show you the manual method.

Install the following prerequisite packages using the yum package manager.

# yum install kernel-devel cpp gcc gcc-c++ binutils elfutils-libelf-devel make kernel-headers nfs-utils ethtool python3 perl rpm-build

Extracting the archive file:

# tar -xf Scale_std_install_5.1.8.0_x86_64.tar.gz

Verify that the self-extracting program has executable permissions. Otherwise, you can make it executable by issuing the ‘chmod +x’ command.

# ls -lh
-rw-r--r-- 1 root root 2.0K Nov 18 07:00 README
-rw-r----- 1 root root 1.1G Nov 18 07:20 Scale_std_install_5.1.8.0_x86_64.tar.gz
-rw-r--r-- 1 root root 1.6K Nov 18 07:25 SpectrumScale_public_key.pgp
-rwxr-xr-x 1 root root 1.1G Nov 18 07:25 Spectrum_Scale_Standard-

When running the below command. Initially, it extracts License Acceptance Process Tools to /usr/lpp/mmfs/ and invokes it, which prompts for License Acceptance, input '1' and hit 'Enter' to extract the required product RPMs to /usr/lpp/mmfs/ It checks if the same dir was previously created during another extraction. If yes, it will be removed to avoid conflicts with the ones being extracted. Also, it removes License Acceptance Process Tools once product RPMs extracted.

Also, it provides an instruction to install and configure IBM Storage Scale through the Toolkit method.

# sh Spectrum_Scale_Standard-

Navigate to the RPM file location in order to install it.

# cd /usr/lpp/mmfs/

Install the GPFS RPMs now.

# rpm -ivh gpfs.base*.rpm gpfs.gpl*rpm gpfs.license.std*.rpm gpfs.gskit*rpm gpfs.msg*rpm*rpm

3a) Exporting PATH environment Variable

Export the PATH environment variable for the root user on each node as shown below, this allows a user to execute IBM Storage Scale commands without changing directory to '/usr/lpp/mmfs/bin'.

# echo "export PATH=$PATH:$HOME/bin:/usr/lpp/mmfs/bin" > /root/.bash_profile
# source /root/.bash_profile

3b) Installing GPFS Portability Layer

Run the below command to build the GPFS portability layer. The GPFS portability layer is a loadable kernel module that allows the GPFS daemon to interact with the operating system.

This command must be executed every time a new kernel is installed and can be automated using the 'autoBuildGPL' option, which will be shown to you in this article or the next one.

# mmbuildgpl

3c) Creating GPFS cluster

This is a very small setup (a two-node cluster), so we’ll add both nodes with the role of cluster manager and the quorum manager. To do so, add the GPFS hostname to a text file as shown below.


In the following file we will only add ‘Node Name’ and ‘Node Designations’ and the rest of the parameters can be added as part of GPFS cluster creation command.

# cd /usr/lpp/mmfs

# echo "2ggpfsnode01-gpfs:manager-quorum
2ggpfsnode02-gpfs:manager-quorum" > nodefile.txt

Now everything is ready. Finally, run the mmcrcluster command to create a cluster.

# mmcrcluster -N /usr/lpp/mmfs/nodefile.txt --ccr-enable -r /usr/bin/ssh -R /usr/bin/scp -C 2gtest-cluster -A

mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Finalizing the cluster data structures ...
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmcrcluster: [I] The cluster was created with the tscCmdAllowRemoteConnections configuration parameter set to "no". If a remote cluster is established with another cluster whose release level (minReleaseLevel) is less than, change the value of tscCmdAllowRemoteConnections in this cluster to "yes".
mmcrcluster: Propagating the cluster configuration data to all
  affected nodes. This is an asynchronous process.


  • /usr/lpp/mmfs/nodefile.txt – This file contains node information’s.
  • –ccr-enable – Cluster Configuration Repository (CCR) to maintain cluster configuration information.
  • -r – RemoteShellCommand (/usr/bin/ssh)
  • -R – RemoteFileCopy (/usr/bin/scp)
  • -C – ClusterName (2gtest-cluster)
  • -N – Input a Node name and node designations or Path of the file that contains node & designations information.
  • -A – Specifies that GPFS daemons are to be automatically started when nodes come up.

To verify the cluster information created in the above steps, run:

# mmlscluster
GPFS Cluster Installation

3d) Creating Tiebreaker Disks

As discussed in the beginning of the article. When running on small GPFS clusters, you might want to have the cluster remain online with only one surviving node. To achieve this, you need to add a tiebreaker disk to the quorum configuration. Node quorum with tiebreaker disks allows you to run with as little as one quorum node available as long as you have access to a majority of the quorum disks.

It won’t hold any data, so get '3x1 GB' disks from SAN (Storage Area Network) in shared mode for both nodes and follow below steps.

Get a LUN ID’s from the Storage Team and scan the SCSI disks (On both the Nodes).

# for host in `ls /sys/class/scsi_host`; do echo "Scanning $host...Completed"; echo "- - -" > /sys/class/scsi_host/$host/scan; done

After the scan, check if the given LUNs are discovered at the OS level.

# lsscsi --scsi --size | grep -i [Last_Five_Digit_of_LUN]

Create 'NSD' file for Tiebreaker Disks as shown below. Please make sure that you are using the correct block devices.

# echo "/dev/sde:::::tiebreak1
/dev/sdg:::::tiebreak3" > /usr/lpp/mmfs/tiebreaker_Disks.nsd

Create Network Shared Disks (NSDs) used by GPFS using the mmcrnsd command as shown below:

# mmcrnsd -F /usr/lpp/mmfs/tiebreaker_Disks.nsd

mmcrnsd: Processing disk sde
mmcrnsd: Processing disk sdf
mmcrnsd: Processing disk sdg
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes. This is an asynchronous process.

If the mmcrnsd command executes successfully, a new line is created and the user inputs are hashed out as shown below.

# cat /usr/lpp/mmfs/tiebreaker_Disks.nsd

# /dev/sde:::::tiebreaker1
# /dev/sdf:::::tiebreaker2
# /dev/sdg:::::tiebreaker3

Run the mmchconfig command to add the tiebreaker disks information in the GPFS cluster configuration.

# mmchconfig tiebreakerDisks="tiebreaker1;tiebreaker2;tiebreaker3"

mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
  affected nodes. This is an asynchronous process.

3e) Starting Cluster

GPFS cluster can be started using the mmstartup command. Use '-a' switch to start the cluster in all node simultaneously.

mmstartup -a

Thu Nov 18 10:45:22 +04 2023: mmstartup: Starting GPFS ...

The mmgetstate command displays the state of the GPFS daemon on one or more nodes. The nodes have not yet joined and formed the cluster, that’s why it shows as 'arbitrating'.

# mmgetstate -aL

Node number  Node name 		Quorum  Nodes up  Total nodes  GPFS state    Remarks
       	  1  2ggpfsnode01-gpfs    			       arbitrating   quorum node
          2  2ggpfsnode02-gpfs    			       arbitrating   quorum node

It’s now joined successfully and GPFS state shows as 'active':

# mmgetstate -aL

Node number  Node name 		Quorum  Nodes up  Total nodes  GPFS state    Remarks
       	  1  2ggpfsnode01-gpfs    1*         2          2      active        quorum node
          2  2ggpfsnode02-gpfs    1*         2          2      active        quorum node

3f) Accepting License

Use the mmchlicense command to designate licenses as needed. Also, it controls the type of GPFS license associated with the nodes in the cluster.

# mmchlicense server --accept -N 2ggpfsnode01-gpfs,2ggpfsnode02-gpfs

The following nodes will be designated as possessing server licenses:
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes. This is an asynchronous process.

Final Thoughts

I hope you learned how to install and configure a GPFS Cluster File System on a RHEL system.

In this article, we covered GPFS Cluster installation and configuration, including Cluster Creation, NSD Creation and Tiebreaker disks addition.

If you have any questions or feedback, feel free to comment below.

Leave a Reply

Your email address will not be published. Required fields are marked *