关于ORACLE 11GR2 RAC部署需要注意的点

1.NTPD和CTSSD

For 11gR2, Cluster Time Synchronization Daemon (CTSSD) can be used in place of NTPD.  CTSSD will synchronize time with a reference node in the cluster when an NTPD is not found to be configured.  Should you require synchronization from an external time source you must use NTPD which will cause CTSSD to run in "observer" mode.  However, if NTP is running, then it must to be configured with the slewing option as documented in Document 551704.1.

Note: Since Oracle Linux 7 ntpd is replaced by chronyd as the default network time protocol daemon

Note : Hangheck timer is not required starting with Oracle Clusterware 11gR2(CTSSD)

The Hangcheck-timer kernel module is required for both Oracle 10gR2 and 11gR1 RAC on Linux. Assuming the default setting of "CSS misscount" is set to either 30 or 60 seconds, the recommended hangcheck-timer settings are: hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1,  see Document 726833.1.  For 11gR2, hangcheck-timer is no longer required as this functionality has been built into the Grid Infrastructure stack.

2.HugePages（标准大页）和 Transparent HugePages（透明大页）

主要用于Large Page Size，大内存。

HugePages:

Implement HugePages for the database instances.  HugePages provides a method to have larger page sizes and it will lock the SGA into physical memory thus eliminating the need for system page table lookups for the SGA.  This is especially important on systems with high memory allocations due to eliminate the management overhead associated with the such configurations.  HugePages is also recommended in environments where server stability issues are evident – various spins/hangs, which are not attributed to Oracle Clusterware or other known OS issues.  To compute size, see Document 401749.1. For more information, see Document 361323.1. In 11gR1 and above, you must disable AMM to use HugePages as directed in Document 749851.1.

**Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)    **

Transparent HugePages:

If you are running RedHat/OEL 6/7, SLES 11/12 or UEK2 kernels, be sure to disable Transparent HugePages to prevent performance problems and Node/Instance evictions.  Our long standing recomendation of using general HugePages still stands (THP is a different implementation than general HugePages).  See Document 1557478.1 for additional details.

**ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and UEK2 and above (Doc ID 1557478.1)    **

If you want to use HugePages make sure that both MEMORY_TARGET / MEMORY_MAX_TARGET initialization parameters are unset (i.e. using “ALTER SYSTEM RESET”) for the database instance.

3.Address Space Layout Randomization(ASLR)

Disable ASLR (Address Space Layout Randomization) on OEL/RHEL 5, 6 and 7 and Suse 10 and above to prevent process startup errors.  ASLR  is a feature designed to load shared memory objects in random addresses. In Oracle, multiple processes map a shared memory object at the same address across the processes.  With ASLR turned on Oracle cannot guarantee the availability of this shared memory address.  This conflict in the address space means that a process trying to attach a shared memory object to a specific address may not be able to do so, resulting in a failure in shmat subroutine.  See Document 1345364.1 for additional details.

Note:  "In RHEL/OEL 7 exec-shield is not modifiable anymore, so changing the exec-shield parameter produces an error."

**You need to reboot for kernel.exec-shield parameter to take effect. **

4.vm.min_free_kbytes

Set the vm.min_free_kbytes kernel parameter to reserve 512MB to allow the OS to reclaim memory faster and avoid LowMem pressure.  See Document 452326.1, Document 452000.1 and Document 1367153.1 for additional information.

Notes：该参数表示Linux VM最低保留多少的空闲内存空间，当可用的内存低于配置参数时，系统会进行cache内存的回收，来进行内存的释放。

★ 知识点
※ 用途：合理设置该值有助于Linux系统更有效地回收内存
※ 注意：对于内存较小（≤32GB）的环境不建议设置，保持默认即可
※ 危险：不要对正在运行的环境进行设置
※ 有修改操作，非原版
※ vm.min_free_kbytes参数的单位是：KB
※ 内存水位
    min水位：下的内存是保留给内核使用的；当到达min，会触发内存的direct reclaim
    low水位：比min高一些，当内存可用量小于low的时候，会触发 kswapd回收内存
    high水位：继续睡眠
※ 内存回收方式
    direct reclaim : 触发min水位线时执行
    kswapd reclaim : 触发low水位线时执行

On servers with High Physical Memory, the parameter vm.min_free_kbytes should be set in the order of 0.4% of total Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers reducing the probability of a low-buffer-space conditions.

*** For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1073742 ***

On NUMA Enabled Systems, the value of vm.min_free_kbytes should be multiplied by the number of NUMA nodes since the value is to be split across all the nodes.

On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here ‘n’ is the number of NUMA nodes.

NUMA(Non Uniform Memory Access Architecture，非统一内存访问)把一台计算机分成多个节点(node)，每个节点内部拥有多个CPU，节点内部使用共有的内存控制器，节点之间是通过互联模块进行连接和信息交互。因此节点的所有内存对于本节点所有的CPU都是等同的，对于其他节点中的所有CPU都不同。虽然每个CPU可以访问整个系统内存，但是访问本地节点的内存速度是最快的(不经过互联模块)，访问非本地节点的内存速度较慢(需要经过互联模块)。
Oracle Database对NUMA的支持

Oracle从11.2开始，默认不会开启NUMA支持，也不会硬件平台以及操作系统是否支持NUMA。要开启NUMA支持，需要手工进行配置。

在11.2中，开启NUMA的支持只需要设置一个隐含参数：

_enable_NUMA_support=TRUE
然后重启数据库，如果设置成功，在alert log就会出现：

...NUMA system found and support enabled...
类似这样的信息。

在12c中，除了要设置上述这个参数外，还需要设置另外一个参数：

_px_numa_support_enabled=TRUE
开启NUMA支持理论上可以提升一定的性能，但是在生产环境中开启NUMA一定要经过严格的测试。

5、vm.swappiness

On kernel revision 2.6.18 and below set the kernel parameter vm.swappiness=100.  Stress testing has shown that vm.swappiness = 100 (default = 60) on kernel version  2.6.18 or lower can reduce or delay node evictions under conditions of heavy memory pressure due to large numbers of client connections or during login storms.

swappiness的值的大小对如何使用swap分区是有着很大的联系的。swappiness=0的时候表示最大限度使用物理内存，然后才是 swap空间，swappiness＝100的时候表示积极的使用swap分区，并且把内存上的数据及时的搬运到swap空间里面。

cat /proc/sys/vm/swappiness

对于内核小于2.6.18的linux系统，默认为60，Oracle建议设置 vm.swappiness = 100；

对于内核大于2.6.18的linux系统，默认为30，建议设置为10；

6、OS package

quired 32-bit and 64-bit OS packages are installed and system prerequisites for your particular release of Oracle have been properly implemented.  This information is documented in Document 169706.1 as well as the install guides for your particular release.

**Oracle Database (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2) (Doc ID 169706.1)    **

**Requirements for Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) (Doc ID 1441282.1)    **

**Requirements for Installing Oracle 11.2.0.4 RDBMS on OL7 or RHEL7 64-bit (x86-64) (Doc ID 1962100.1)    **

7、SeLinux

For pre-11.2.0.2 installations, SELinux must be disabled. For 11.2.0.2, SELinux is supported but the recommendation (if possible) is to run with SELinux disabled. See Bug 9746474.

8、avahi-daemon

Disable the AVAHI daemon to prevent startup issues with the CSSD daemon, see Document 1501093.1 for additional details.

**CSSD Fails to Join the Cluster After Private Network Recovered if avahi Daemon is up and Running (Doc ID 1501093.1)    **

CAUSE

Third party zeroconf mDNS process is up and running, in this case, it’s avahi on Linux

SOLUTION

The solution is to shutdown and disable 3rd party zeroconf mDNS process, in this case, avahi-daemon

To shut it down, as root:

# /etc/init.d/avahi-daemon stop

To disable it, as root:

# /sbin/chkconfig  avahi-daemon off

On Oracle Linux/Redhat Linux, “NOZEROCONF=yes” must be included in /etc/sysconfig/network

Once avahi is disabled, restart the stack or reboot the node.