ORA-27103 [ ] and ORA-6544 [pevm_peruws_callback-1] [27103] in Alert log

-
In a SPARC Solaris system (Ldom) which runs multiple database instances on 12.2.0.2, the alerts logs were flooded with ORA-27103 [ ] and ORA-6544 [pevm_peruws_callback-1] [27103] error messages.
Database operations were impacted, new connection requests were starting to timeout.
 2020-06-01T12:47:06.635714+05:30  
 Errors in file /u01/app/oracle/diag/rdbms/memdb/memdb1/trace/memdb1_dbw0_19325.trc:  
 ORA-27103: internal error  
 Additional information: 11818  
 Additional information: 9  
 ORA-27103: internal error  
 SVR4 Error: 11: Resource temporarily unavailable  
 Additional information: 7175  
 Additional information: 1114112  
 Errors in file /u01/app/oracle/diag/rdbms/memdb/memdb1/trace/memdb1_dbw0_19325.trc (incident=173114) (PDBNAME=CDB$ROOT):  
 ORA-27103 [] [] [] [] [] [] [] [] [] [] [] []  
 Incident details in: /u01/app/oracle/diag/rdbms/memdb/memdb1/incident/incdir_173114/memdb1_dbw0_19325_i173114.trc  

 2020-06-01T12:47:06.651393+05:30  
 Errors in file /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/trace/memadb1_ora_23750.trc (incident=762431) (PDBNAME=PRODMEMA):  
 ORA-6544 [pevm_peruws_callback-1] [27103] [] [] [] [] [] [] [] [] [] []  
 2020-06-01T12:47:06.651417+05:30  
 Errors in file /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/trace/memadb1_j006_3204.trc (incident=762727) (PDBNAME=PRODMEMA):  
 ORA-6544 [pevm_peruws_callback-1] [27103] [] [] [] [] [] [] [] [] [] []  
 PRODONL(6):Incident details in: /u01/app/oracle/product/12.2.0/diag/rdbms/memadb/memadb1/incident/incdir_762727/memadb1_j006_3204_i762727.trc
Server has 140 GB of memory allocated from which 85 GB was allocated to Oracle via project file. (/etc/projects). This was based on classic 60:40 ratio (60% for DB and 40% for OS)
Further digging showed high paging on the server and the below snap from the OS log (/var/adm/messages) confirmed the memory exhaustion. An external diagnosis process which was running has caused significant memory usage resulting the above high paging.
 Jun 1 12:47:06 dbserva1 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 23928 (oracle)  
 Jun 1 12:47:06 dbserva1 genunix: [ID 470503 kern.warning] WARNING: Sorry, no swap space to grow stack for pid 23929 (java)  
 Jun 1 12:47:36 dbserva1 fbt: [ID 795213 kern.warning] WARNING: couldn't allocate FBT table for module oracleacfs  
 Jun 1 12:47:42 dbserva1 last message repeated 13 times  
 Jun 1 12:47:46 dbserva1 genunix: [ID 603404 kern.notice] NOTICE: core_log: oracle[23938] core dumped: /var/cores/core_dbserva1_oracle_54321_54321_1590995856_23938  
 Jun 1 12:47:55 dbserva1 genunix: [ID 780570 kern.info] modload of socketmod/sockrds failed  
 Jun 1 12:48:01 dbserva1 last message repeated 89 times

Comments

Post a Comment

Popular posts from this blog

ORA-16433: The database or pluggable database must be opened in read/write

-
Image
ORA-16433 is an error returned when a database cannot be opened. This can be a result of multiple conditions including, The inconsistency of the database Physical corruption Loss of dependent files or access issues etc.. More often than not the solution is to recover from available backups, but there can be situations where backups are not available, specially on UAT or Test setups. Although not the ideal scenario, in this type of situation you might want to open the databases even with inconsistency.  In Comes --> _ALLOW_RESETLOGS_CORRUPTION First, if you are in a position to use _ALLOW_RESETLOGS_CORRUPTION=TRUE, that means it has to be a critical moment, and when you come across more ora- errors while you are trying to reopen the databsae, that can be annoying. Note - _ALLOW_RESETLOGS_CORRUPTION=TRUE must only be used after Oracle support confirmation as a last resort. Also make sure to take cold backup before proceeding. Basic steps to be followed to resolve ORA 16433 after
Read more

Wait for unread message on broadcast channel - Blocking Sessions

-
Image
What is Wait for unread message on broadcast channel wait event? Wait for unread message on broadcast channel is a wait event where source destination communication is involved and the database is waiting for a reply from remote party. Mostly this wait is seen related to data pump export (expdp) or import (impdp) operations. Although categorized in idle wait class "Wait for unread message on broadcast channel" can become the blocking session for some other sessions. Common causes for stuck sessions waiting on this wait event includes, Broken databases links, where remote database is not responding. Abnormally killed remote processes causing the sessions to wait. When a session is waiting on "Wait for unread message on broadcast channel" it can become a blocking session for dependent sessions. Most probably these sessions will be waiting on "library cache pin" or related concurrency wait events. The blocking_session column of (g)v$session view
Read more

ORA-14126 - While splitting max value partition

-
Oracle version -- Oracle 9i to 11g While splitting partitions, SQL> alter TABLE ARBOR.CDR_BILLED split PARTITION CDR_BILLED_MAXVALUE at (TO_DATE('2014-09-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) into (partition CDR_BILLED_01SEP14,partition CDR_BILLED_MAXVALUE) TABLESPACE "C01_CDR_BILL" update global indexes; 2 into (partition CDR_BILLED_01SEP14,partition CDR_BILLED_MAXVALUE) TABLESPACE "C01_CDR_BILL" update global indexes * ERROR at line 2: ORA-14126: only a <parallel clause> may follow description(s) of resulting partitions 4:58:56 PM The reason for above error is while splitting max value partition, it is not allowed to explicitly provide a tablespace. The splitted partition will be reside on the same tablespace as the max value partitions. Below can be used without the tablespace clause to split the max value partition, alter TABLE ARBOR.CDR_BILLED split PARTITION CDR_BILLED_
Read more