ORA-27103 [ ] and ORA-6544 [pevm_peruws_callback-1] [27103] in Alert log

-
In a SPARC Solaris system (Ldom) which runs multiple database instances on 12.2.0.2, the alerts logs were flooded with ORA-27103 [ ] and ORA-6544 [pevm_peruws_callback-1] [27103] error messages.
Database operations were impacted, new connection requests were starting to timeout.
 2020-06-01T12:47:06.635714+05:30  
 Errors in file /u01/app/oracle/diag/rdbms/memdb/memdb1/trace/memdb1_dbw0_19325.trc:  
 ORA-27103: internal error  
 Additional information: 11818  
 Additional information: 9  
 ORA-27103: internal error  
 SVR4 Error: 11: Resource temporarily unavailable  
 Additional information: 7175  
 Additional information: 1114112  
 Errors in file /u01/app/oracle/diag/rdbms/memdb/memdb1/trace/memdb1_dbw0_19325.trc (incident=173114) (PDBNAME=CDB$ROOT):  
 ORA-27103 [] [] [] [] [] [] [] [] [] [] [] []  
 Incident details in: /u01/app/oracle/diag/rdbms/memdb/memdb1/incident/incdir_173114/memdb1_dbw0_19325_i173114.trc  

 2020-06-01T12:47:06.651393+05:30  
 Errors in file /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/trace/memadb1_ora_23750.trc (incident=762431) (PDBNAME=PRODMEMA):  
 ORA-6544 [pevm_peruws_callback-1] [27103] [] [] [] [] [] [] [] [] [] []  
 2020-06-01T12:47:06.651417+05:30  
 Errors in file /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/trace/memadb1_j006_3204.trc (incident=762727) (PDBNAME=PRODMEMA):  
 ORA-6544 [pevm_peruws_callback-1] [27103] [] [] [] [] [] [] [] [] [] []  
 PRODONL(6):Incident details in: /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/incident/incdir_762727/memadb1_j006_3204_i762727.trc
Server has 140 GB of memory allocated from which 85 GB was allocated to Oracle via project file. (/etc/projects). This was based on classic 60:40 ratio (60% for DB and 40% for OS)
Further digging showed high paging on the server and the below snap from the OS log (/var/adm/messages) confirmed the memory exhaustion. An external diagnosis process which was running has caused significant memory usage resulting the above high paging.
 Jun 1 12:47:06 dbserva1 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 23928 (oracle)  
 Jun 1 12:47:06 dbserva1 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 23929 (java)  
 Jun 1 12:47:36 dbserva1 fbt: [ID 795213 kern.warning] WARNING: couldn't allocate FBT table for module oracleacfs  
 Jun 1 12:47:42 dbserva1 last message repeated 13 times  
 Jun 1 12:47:46 dbserva1 genunix: [ID 603404 kern.notice] NOTICE: core_log: oracle[23938] core dumped: /var/cores/core_dbserva1_oracle_54321_54321_1590995856_23938  
 Jun 1 12:47:55 dbserva1 genunix: [ID 780570 kern.info] modload of socketmod/sockrds failed  
 Jun 1 12:48:01 dbserva1 last message repeated 89 times

Comments

Post a Comment

Popular posts from this blog

ORA-16433: The database or pluggable database must be opened in read/write

-
Image
ORA-16433 is an error returned when a database cannot be opened. This can be a result of multiple conditions including, The inconsistency of the database Physical corruption Loss of dependent files or access issues etc.. More often than not the solution is to recover from available backups, but there can be situations where backups are not available, specially on UAT or Test setups. Although not the ideal scenario, in this type of situation you might want to open the databases even with inconsistency.  In Comes --> _ALLOW_RESETLOGS_CORRUPTION First, if you are in a position to use _ALLOW_RESETLOGS_CORRUPTION=TRUE, that means it has to be a critical moment, and when you come across more ora- errors while you are trying to reopen the databsae, that can be annoying. Note - _ALLOW_RESETLOGS_CORRUPTION=TRUE must only be used after Oracle support confirmation as a last resort. Also make sure to take cold backup before proceeding. Basic steps to be followed to resolve ORA 16433 after
Read more

Oracle Multitenant - Create new service for PDB using DBMS_SERVICE

-
Image
When an Oracle PDB is created it starts up a default service with the PDB name. For direct connections to    PDB this service can be used. But there are situations where new service name is required in PDB level, common example is after a PDB restoration/clone, in order to enable application connections without changes in connection string. Setting service_names parameter in PDB level is not possible hence it is not an option in multitenant setup. How to create a new PDB service on Oracle 12C Check the current services, here only the default service is running, SQL> show pdbs; CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 3 NFACTORPDB READ WRITE NO SQL> select service_id, name, pdb from dba_Services; SERVICE_ID NAME PDB ---------- ---------------------------------------- ---------- 7 nfactorpdb NFACTORPDB SQL> Create
Read more

Wait for unread message on broadcast channel - Blocking Sessions

-
Image
What is Wait for unread message on broadcast channel wait event? Wait for unread message on broadcast channel is a wait event where source destination communication is involved and the database is waiting for a reply from remote party. Mostly this wait is seen related to data pump export (expdp) or import (impdp) operations. Although categorized in idle wait class "Wait for unread message on broadcast channel" can become the blocking session for some other sessions. Common causes for stuck sessions waiting on this wait event includes, Broken databases links, where remote database is not responding. Abnormally killed remote processes causing the sessions to wait. When a session is waiting on "Wait for unread message on broadcast channel" it can become a blocking session for dependent sessions. Most probably these sessions will be waiting on "library cache pin" or related concurrency wait events. The blocking_session column of (g)v$session view
Read more