一、HugePages（标准大页）

1、什么是标准大页？

HugePages is a feature integrated into the Linux kernel with release 2.6 and later. This feature basically provides the alternative to the 4K page size (16K for IA64) providing bigger pages.

HugePages in 2.4 Kernels

The HugePages feature is backported to some 2.4 kernels. Kernel versions 2.4.21-* has this feature (See Note 311504.1 for the distributions with 2.4.21 kernels) but it is implemented in a different way. The feature is completely available. The difference from 2.6 implementation is the organization within the source code and the kernel parameters that are used for configuring HugePages. See Parameters/Setup section below.

2、如何查看Hugepagesize

$ grep Hugepagesize /proc/meminfo

HW Platform	Source Code Tree	Kernel 2.4	Kernel 2.6 and later
Linux x86 (IA32)	i386	4 MB	2 MB
Linux x86-64 (AMD64, EM64T)	x86_64	2 MB	2 MB
Linux Itanium (IA64)	ia64	256 MB	256 MB
IBM Power Based Linux (PPC64)	ppc64/powerpc	N/A **	16 MB
IBM zSeries Based Linux	s390	N/A	1 MB
IBM S/390 Based Linux	s390	N/A	N/A

3、HugePages and Oracle 11g Automatic Memory Management (AMM)

The AMM and HugePages are not compatible. One needs to disable AMM on 11g to be able to use HugePages. See Document 749851.1 for further information.

4、如何设置HugePages

Kernel Version 2.4

The kernel parameter used for HugePages is vm.hugetlb_pool which is based on MB of memory. RHEL3, Asianux 1.0, SLES8 (Service Pack 3 and over) are examples of distributions with the 2.4 kernels with HugePages support. For the configuration, follow steps below:

1. Start database instance(s)
2. Calculate hugetlb_pool using script from Note 401749.1
3. Shutdown database instances 
4. Set kernel parameter:
         # sysctl -w vm.hugetlb_pool=<value from above>
    and make sure that the parameter is persistent to reboots. e.g. On Asianux 1.0 by editing /etc/sysctl.conf adding/modifying as below:
        vm.hugetlb_pool=<value from above>

5. Check available hugepages:
        $ grep Huge /proc/meminfo

6. Restart database instances
7. Check available hugepages:
        $ grep Huge /proc/meminfo

Notes:

If the setting of hugetlb_pool is not effective, you will need to reboot the server to make HugePages allocation during system startup.
The HugePages are allocated in a lazy fashion, so the "Hugepages_Free" count drops as the pages get touched and are backed by physical memory. The idea is that it's more efficient in the sense that you don't use memory you don't touch.
If you had set the instance initialization parameter PRE_PAGE_SGA=TRUE (for suitable settings see Document 30793.1), all of the pages would be allocated from HugePages up front.

Kernel Version 2.6 and later

The kernel parameter used for HugePages is vm.nr_hugepages which is based on the number of the pages. SLES9, RHEL4 and Asianux 2.0 are  examples of distributions with the 2.6 kernel. For the configuration, follow steps below:

1. Start database instance(s)
2. Calculate nr_hugepages using script from Document 401749.1
3. Shutdown database instances 
4. Set kernel parameter:
        # sysctl -w vm.nr_hugepages=<value from above>
and make sure that the parameter is persistent to reboots. e.g. On SLES9:
        # chkconfig boot.sysctl on

5. Check available hugepages:
        $ grep Huge /proc/meminfo

6. Restart database instances
7. Check available hugepages:
        $ grep Huge /proc/meminfo

Notes:

If the setting of nr_hugepages is not effective, you will need to reboot the server to make HugePages allocation during system startup.
The HugePages are allocated in a lazy fashion, so the "Hugepages_Free" count drops as the pages get touched and are backed by physical memory. The idea is that it's more efficient in the sense that you don't use memory you don't touch.
If you had set the instance initialization parameter PRE_PAGE_SGA=TRUE (for suitable settings see Document  30793.1), all of the pages would be allocated from HugePages up front.
Notes on HugePages in General
The userspace application that employs HugePages should be aware of permission implications. Permissions HugePages segments in memory can strictly impose certain requirements. e.g. Per Bug 6620371 on Linux x86-64 port of Oracle RDBMS until 11g was setting the shared memory flags to hugetlb, read and write by default. But that shall depend on the configuration environment and with Patch 6620371 on 10.2 and with 11g, the read and write permissions are set based on the internal context.

Calculate hugetlb_pool/nr_hugepages using script from Document 401749.1

[oracle@p11g01 ~]$ cat /home/oracle/hugepages_settings.sh 
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# on Oracle Linux
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# http://support.oracle.com

# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed..."

read

# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
    echo "The hugepages may not be supported in the system where the script is being executed."
    exit 1
fi

# Initialize the counter
NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
    echo "***********"
    echo "** ERROR **"
    echo "***********"
    echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
    exit 1
fi

# Finish with results
case $KERN in
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.14') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.18') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '5.4') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    *) echo "Kernel version $KERN is not supported by this script (yet). Exiting." ;;
esac

# End

二、Transparent HugePages（透明大页）

1、什么是Transparent HugePages（默认开启）

Starting with RedHat6, OL6, SLES11, and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management.  Transparent HugePages are similar to the HugePages that have been available in previous Linux releases.  The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time.

2、开启Transparent HugePages会有什么影响

Starting with SLES11, RHEL6, OL6, and UEK2 Kernels, Transparent HugePages are introduced and enabled by default, and this can cause node reboots and performance problems.
Node reboots.

The ocssd.log may show some of the threads are blocked (but this does not show all the time):

2013-05-01 14:30:45.255: [    CSSD][224204544]clssscMonitorThreads clssnmvKillBlockThread not scheduled for 7500 msecs
2013-05-01 14:30:46.945: [    CSSD][224204544]clssscMonitorThreads clssnmvWorkerThread not scheduled for 8030 msecs

3、Oracle建议关闭Transparent HugePages

Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.

4、如何关闭Transparent HugePages（透明大页）

For RedHat 6, Oracle Linux 6, SLES 11 and UEK2 kernels please have at "ALERT: Disable Transparent HugePages on SLES11, RHEL6, Oracle Linux 6 and UEK2 Kernels (Doc ID 1557478.1)"

The following instruction are for Redhat 6 or lower, Oracle Linux 6 or lower, and SLES 11 or lower:

To check if the Transparent HugePages are enabled in your server execute the following:

Default/Enabled setting is  [always]:

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] never

Note 1: For RHEL kernel, the file path is different from above:

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

[always] never
Please modify this file accordingly.

Note 2: For UEK2 kernel, as of 2.6.39-400.116.0 Transparent Hugepages has been removed from the kernel.  If it is not compiled into the kernel then /sys/kernel/mm/transparent_hugepage will not exist.
** Disabled setting is [never]:**

# cat /sys/kernel/mm/transparent_hugepage/enabled

always [never]

If “enabled” is NOT set to “[never]”, the Transparent HugePages are being used.

You can also issue:

# grep AnonHugePages /proc/meminfo

If the output contains a line like “AnonHugepages: xxxx kB”, with a value > 0kB, the kernel is using Transparent HugePages.

Because the kernel currently uses Transparent HugePages only for the anonymous memory blocks like stack and heap, the value of AnonHugepages in /proc/meminfo is the current amount of Transparent HugePages that the kernel is using.
To disable Transparent HugePages boot time either one of the following 2 methods may be used:

Add the following to the kernel boot line in /etc/grub.conf (a symlink to /boot/grub/grub.conf) and reboot the server (this is the preferred method):

transparent_hugepage=never

Once modified the line will read similar to the following example:

title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
        root (hd0,0)
       ** kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never**
        initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img

OR

Add the following lines in /etc/rc.local and reboot the server (this still can be done on Redhat 7 although rc.local is being deprecated):

cat <<EOF >> /etc/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

Please change the file path for RHEL kernel to /sys/kernel/mm/redhat_transparent_hugepage/ accordingly.

===========================================================================

The following comment was received about SLES11 SP3.  Please note that this is not tested, so this is provided only as a talking point to your sys admin who knows SLES11 SP3 and who should also contact SLES for correct information.

in SLES11 SP3 with YAST/GRUB boot loader /etc/grub.conf overrided usind YAST. parameter transparent_hugepage=never must be set in YAST-bootloader- Edit settings - in line: optional kernel parameter [before showopts].

_Oracle Linux 7: How to Disable Transparent HugePages for RHCK Kernel? (Doc ID 2066217.1)    _

SOLUTION

“tuned.service” on OL7 set the_ transparent_hugepage_ to always by default, even if it is disabled in grub kernel command line.

# grep transparent_hugepage /boot/grub2/grub.cfg

linux16 /vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/vgsystem-root ro  rd.lvm.lv=vgsystem/swap rd.lvm.lv=vgsystem/root rhgb quiet numa=off **transparent_hugepage=never     **

Verify the THP (Transparent hugepage) mode

# uname -r

3.10.0-229.el7.x86_64

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never                                 ← **THP is enabled **

Disable it globally on throughput-performance

1. **Take a backup of “/usr/lib/tuned/throughput-performance/tuned.conf”, then change “transparent_hugepages=always” to “transparent_hugepages=never”. **

   #cp /usr/lib/tuned/throughput-performance/tuned.conf /usr/lib/tuned/throughput-performance/tuned.conf.bkp_original
   #vi /usr/lib/tuned/throughput-performance/tuned.conf
   

[vm]
 transparent_hugepages=always
   to 
 [vm]
 transparent_hugepages=never 
2. ** Reboot the server and verify the outcome.   **
#cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]                          ← THP is disabled

Disable it on active tuned profile

1. ** First Identify which profile is active.     **

   # tuned-adm active
   

   Current active profile: virtual-guest            ← Virtual-guest is the active profile

2. **Now edit “/usr/lib/tuned/virtual-guest/tuned.conf” file and append "transparent_hugepages=never" in_ vm _section to disable THP.  **

   #cp /usr/lib/tuned/virtual-guest/tuned.conf /usr/lib/tuned/virtual-guest/tuned.conf.bkp_original
   #vi /usr/lib/tuned/virtual-guest/tuned.conf
   

   [main]
   include=throughput-performance
   **[vm]
   transparent_hugepages=never                       **

3. Reboot the server and verify the outcome.

   #cat /sys/kernel/mm/transparent_hugepage/enabled
   always madvise [never]                            ← THP Disabled

Transparent Huge Pages cannot be enabled or disabled on a running machine and requires a reboot.

THP status in other UEK kernel versions

• UEK3 disabled by default .

  [root@ol6 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-3.8.13-118.6.2.el6uek.x86_64
  # CONFIG_TRANSPARENT_HUGEPAGE is not set

• UEK4 enabled by default.

  [root@ol7 ~]# grep -i CONFIG_TRANSPARENT_HUGEPAGE /boot/config-4.1.12-37.2.2.el7uek.x86_64
  CONFIG_TRANSPARENT_HUGEPAGE=y
  CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y