Solaris Linux iostat in DBAs view

-

Background

When troubleshooting  any database performance issue, we need to understand that it can happen in two possible sequence,

1. Additional large utilization (load) on database server causing the delay in normal database operation. Which can be from newly added application component, query that is running abnormally (e.g plan change),  new parallel execution etc..

2. A change or suboptimal operation on underlying hardware/software layer that causes the default load of the database to take more time than it usually does. Here database system is not the culprit but the victim. It is trying its best to perform as it used to, but a bottleneck in different layer that is not in the control of database is causing the delay.

Example

Below is a demonstration using iostat command in Solaris environment running Oracle database, which shows the metric differences while database system is experiencing performance delays. In both situation the result is the same, distorted db performance where queries taking more time to execute, transaction timing out in applications etc.. but the cause leading to the result is completely different.

Case 1 - iostat during new query executing with parallel degree of 24

solaris iostat output explained

Here in highlighted part of the iostat output, we can observe very large amount of read I/O (kr/s) with average service time +35 ms (asvc_t) and disk busy percentage (%b) around 100%.
All these iostat outputs indicate overloaded disk sub system, resulted from IO heavy query running with parallel 24.
So in this type of scenario next steps for resolution would be to look at query optimization reducing the actual IO load. 

Case 2 - iostat during delay failure of SAN storage controller affecting set of disks


Linux iostat output explained


Here when we look at the highlighted outputs, we see almost zero IO read requests (kr/s) or IO write requests (kw/s) but still some disks show busy percentage (%b) of100%. Some queries were impacted as they were not completing.
Here when providing the solution we need to understand that database is impacted by underlying infrastructure and need to widen troubleshooting beyond the database. Above example iostat output was during a SAN storage controller failure which impacted some of the disks.

Comments

Post a Comment

Popular posts from this blog

ORA-16433: The database or pluggable database must be opened in read/write

-
Image
ORA-16433 is an error returned when a database cannot be opened. This can be a result of multiple conditions including, The inconsistency of the database Physical corruption Loss of dependent files or access issues etc.. More often than not the solution is to recover from available backups, but there can be situations where backups are not available, specially on UAT or Test setups. Although not the ideal scenario, in this type of situation you might want to open the databases even with inconsistency.  In Comes --> _ALLOW_RESETLOGS_CORRUPTION First, if you are in a position to use _ALLOW_RESETLOGS_CORRUPTION=TRUE, that means it has to be a critical moment, and when you come across more ora- errors while you are trying to reopen the databsae, that can be annoying. Note - _ALLOW_RESETLOGS_CORRUPTION=TRUE must only be used after Oracle support confirmation as a last resort. Also make sure to take cold backup before proceeding. Basic steps to be followed to resolve ORA 16433 after
Read more

Wait for unread message on broadcast channel - Blocking Sessions

-
Image
What is Wait for unread message on broadcast channel wait event? Wait for unread message on broadcast channel is a wait event where source destination communication is involved and the database is waiting for a reply from remote party. Mostly this wait is seen related to data pump export (expdp) or import (impdp) operations. Although categorized in idle wait class "Wait for unread message on broadcast channel" can become the blocking session for some other sessions. Common causes for stuck sessions waiting on this wait event includes, Broken databases links, where remote database is not responding. Abnormally killed remote processes causing the sessions to wait. When a session is waiting on "Wait for unread message on broadcast channel" it can become a blocking session for dependent sessions. Most probably these sessions will be waiting on "library cache pin" or related concurrency wait events. The blocking_session column of (g)v$session view
Read more

ORA-14126 - While splitting max value partition

-
Oracle version -- Oracle 9i to 11g While splitting partitions, SQL> alter TABLE ARBOR.CDR_BILLED split PARTITION CDR_BILLED_MAXVALUE at (TO_DATE('2014-09-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) into (partition CDR_BILLED_01SEP14,partition CDR_BILLED_MAXVALUE) TABLESPACE "C01_CDR_BILL" update global indexes; 2 into (partition CDR_BILLED_01SEP14,partition CDR_BILLED_MAXVALUE) TABLESPACE "C01_CDR_BILL" update global indexes * ERROR at line 2: ORA-14126: only a <parallel clause> may follow description(s) of resulting partitions 4:58:56 PM The reason for above error is while splitting max value partition, it is not allowed to explicitly provide a tablespace. The splitted partition will be reside on the same tablespace as the max value partitions. Below can be used without the tablespace clause to split the max value partition, alter TABLE ARBOR.CDR_BILLED split PARTITION CDR_BILLED_
Read more