Problem statement: Database creation in a ODA DB-system
(version 19.30) was failing due to insufficient free memory.
DCS-10045:Validation
error encountered: DB Shape validation on memory for DB creation/iRestore
failed: DCS-10045:Validation error encountered: Available Memory is less than
SGA Size { Available : 380MB and SGA Size : 7782MB } on <hostname>.
Inside the DBSystem
only free Huge page was 212 which is approximately 400 MB and is exactly matching
what we have received as an error.
cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 4083
HugePages_Free: 212
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 8361984 kB
Action Taken:We attempted to increase huge page
manually using the configuration file /etc/systcl.conf from vm.nr_hugepages=4083
to vm.nr_hugepages=8000.It was not taking into effect even after executing
sysctl -p and still change was not reflected and HugePages_Total was showing
5929.DBSystem node was rebooted and after that ssh to node was failing with the
below error
ssh: connect to host <hostname>
port 22: No route to host
Troubleshooting: First of all,I must say above actions
were taken harshly and never ever try that in a DB systems. Very first thing we
should have done is to know DB-Systems detail using below command
odacli describe-dbsystem -n vmdb01
DB System details
--------------------------------------------------------------------------------
ID: 2b53b0dc-2d27-47e7-be09-85fdde8310fa
Name: vmdb01
Image version: 23.26.1.0.0
Current version: 23.26.1.0.0
Shape: dbs2
Cluster name: vmcl
Grid version: 23.26.1.0.0
NUMA enabled: YES
Status: CONFIGURED
Created: 2026-03-19 12:34:02 MST
Updated: 2026-03-19 13:21:32 MST
User Information
--------------------------
Multi User Status: Multi User
Access (MUA)
CPU Pool
--------------------------
Name: cpupool2c
Number of cores: 2
Host: oda02a
Effective CPU set: 1,17,33,49
Online CPUs: 1, 17, 33, 49
Offline CPUs: NONE
Host: oda02b
Effective CPU set: 1,17,33,49
Online CPUs: 1, 17, 33, 49
Offline CPUs: NONE
VM Storage
--------------------------
Disk group: DATA
Volume name: S8E0A8E79D
Volume device: /dev/asm/s8e0a8e79d-364
Size: 402.00 GB
Used: 132.15 GB
Free: 269.85 GB
Usage: 32.87%
Mount Point: /u05/app/sharedrepo/vmdb01
Redundancy: Mirror
Acc Volume name: AS8E0A8E79D
Acc Volume device:
/dev/asm/as8e0a8e79d-163
Acc Volume size: 1.61 GB
VMs
--------------------------
Host: oda02a
VM Name: x3e0a3e89d
VM Host Name: db01.com
VM image path: /u05/app/sharedrepo/vmdb01/.ACFS/snaps/vm_x3e0a3e89d/x3e0a3e89d
Number of cores: 2
Memory: 16.00 GB
Target State: ONLINE
Current State: ONLINE
Host: oda02b
VM Name: y3e0a3e89d
VM Host Name: db02.com
VM image path: /u05/app/sharedrepo/vmdb01/.ACFS/snaps/vm_y3e0a3e89d/y3e0a3e89d
Number of cores: 2
Memory: 16.00 GB
Target State: ONLINE
Current State: ONLINE
VNetworks
--------------------------
Host: oda02a
VM Name: x3e0a3e89d
Public: 10. / 255.255.254.0 / enp0s3 / BRIDGE(pubnet)
ASM: 192. / 255.255.255.128 / enp0s4 /
BRIDGE(privasm) VLAN(icbond0.100)
Interconnect: 192.**.**.**
/ 255.255.255.252 / enp0s5 / BRIDGE(privnet1) VLAN(icbond0.101)
Host: oda02b
VM Name: y3e0a3e89d
Public: 10. / 255.255.254.0 / enp0s3 / BRIDGE(pubnet)
ASM: 192.**.**.** / 255.255.255.128 / enp0s4 /
BRIDGE(privasm) VLAN(icbond0.100)
Interconnect: 192.**.**.**/ 255.255.255.252 / enp0s5 /
BRIDGE(privnet1) VLAN(icbond0.101)
Extra VNetworks
--------------------------
Host: oda02a
VM Name: x3e0a3e89d
pubnet: 10. / 255.255.254.0 / PUBLIC
Host: oda02b
VM Name: y3e0a3e89d
pubnet: 10. / 255.255.254.0 / PUBLIC
Databases
--------------------------
Name: cdbtest
Resource ID: 24883518-c164-40a7-970a-2f52f8a28e30
Unique name: cdbtest
Database ID: 970363228
Domain name: .com
Status: CONFIGURED
DB Home ID: a6c30871-762d-412e-939c-b6fbe7cdf7ff
Shape: odb2
Version: 23.26.1.0.0
Edition: EE
Type: SI
Role: PRIMARY
Class: OLTP
Storage: ASM
Redundancy:
Target node name: db01
Character set: AL32UTF8
NLS character set: AL16UTF16
Language: AMERICAN
Territory: AMERICA
Console enabled: false
High Availability: false
Associated networks:
Public-network
Backup config ID:
Level 0 Backup Day: sunday
Autobackup enabled: false
TDE enabled: false
CDB type: true
PDB name: pdbtest
PDB admin user: pdbadmin
Every vm maintains
its configuration file into a XML file under ACFS file storage where actual VM
image is located.
Use below
command to find running VM in the bare metal node like below
virsh
list --all
Id
Name State
----------------------------
7 x3e0a3e89d running
Then dump
contents of the XML file like below
virsh
dumpxml x3e0a3e89d > /root/x3e0a3e89d.xml
Under
html tag sourcefile we can see the location of the xml file
Note:- The virsh program is the main interface for managing
virsh guest domains. The program can be used to create, pause, and shutdown
domains. It can also be used to list current domains.virsh operations rely upon
libvirtd library.
grep -i memory
/root/x3e0a3e89d.xml
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<memoryBacking>
</memoryBacking>
<cell id='0' cpus='0-1' memory='8388608'
unit='KiB'>
<cell id='1' cpus='2-3' memory='8388608'
unit='KiB'>
Now we will
update actual xml configuration using below command where we are doubling memory
in VM level also in VM node level
virsh
edit x3e0a3e89d
Increase the
memory value for the parameters per below
<memory
unit='KiB'>33554432</memory>
<currentMemory
unit='KiB'>33554432</currentMemory>
<cell id='0'
cpus='0-1' memory='16777216' unit='KiB'>
<cell id='1'
cpus='2-3' memory='16777216' unit='KiB'>
Now shutdown
using below command
virsh
destroy x3e0a3e89d
Domain
'x3e0a3e89d' destroyed
Now below
command removes the VM definition from /etc/libvirt/qemu/x3e0a3e89d.xml but
does not delete the virtual disks bydefault
virsh
undefine x3e0a3e89d
Domain
'x3e0a3e89d' has been undefined
Done same from other
node
virsh
destroy y3e0a3e89d
Domain
'y3e0a3e89d' destroyed
virsh
undefine y3e0a3e89d
Domain
'y3e0a3e89d' has been undefined
Now stop the db
system and then start the db-system
odacli
stop-dbsystem -n vmdb01
odacli
start-dbsystem -n vmdb01
At this stage
we were able to login in the DBsystem but dcs agent was down
odacli
ping-agent
DCS-10033:Service
DCS agent is down.
Restarted the
agent using below command from root
systemctl
stop initdcsagent
systemctl
start initdcsagent
Now change what
we made in vm level is not reflected in ODA repository.SO to sync that we executed
below command to increase memory without changing the shape of the db system
odacli
modify-dbsystem -m 32G -n vmdb01
So please
mind that if we face low hugepage memory issue in dbsystem do not just change
systcl.conf in vm nodes.
Stop dbsystem
from both nodes
Use above
command if you just want to increase memory without changing shape.That will
take care and distribute equally new memory settings
You can also
increase the shape of the dbsystem like below
odacli
modify-dbsystem -n vmdb01-s dbs6
I hope you have
learnt something useful…
Reference:- ODA ODACLI UPDATE-OSPARAMETERS or OAKCLI
RECONFIGURE OSPARAMS to Calculate, Update and Set Kernel / Memory Values
Including Hugepages
KB593630
