See Metalink 465001.1 for raw device configuration for RHEL 5. It's not explicitly said, but all disks used must be partitioned! Otherwise, if you don't partition the OCR disk, the root.sh from clusterware installation will fail (fail with “Failed to upgrade Oracle Cluster Registry configuration” error, with “Failed to call clsssinit” in log.)
The ASMLib will also refuse to stamp whole disk, a partition is required.
For OS configuration, see Metalink 421308.1. However, the sysctl parameters listed there do not exist on RHEL 5.1, you will have to use:
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304
net.ipv4.tcp_wmem = 262144 262144 262144
You will also need to install one more rpm package (kernel-headers from CD1), but this will indicate the rpm command anyway.
As for network, remember that the localhost entries must be at the end of hosts file, so OUI will resolve current server with its name, instead of "localhost". (You would see this in the "Specify cluster configuration" screen.) If the node is listed as "localHost" (note the capital H), check your /etc/hosts for errors and that you can ping all the nodes listed (check also for typing errors).
Before running the root.sh script, read Metalink 414163.1. You will have to edit vipca and srvctl scripts (remove LD_ASSUME_KERNEL setting) and manually configure public/interconnect interfaces, finally running vipca interactively from root X session.
If you ran root.sh without these changes, remove the /etc/oracle/ocr.loc, remove the entries from /etc/inittab and run telinit so the root.sh can configure the OCR from scratch again.
Hope this helps:-)