Tuesday, March 03, 2009

Discovering the I/O pattern of a production database through mining STATSPACK data

1. Total I/O in every hour during weekdays

Total I/O is defined as the number of physical reads plus the number of physical writes. They can be obtained throught the following two scripts, respectively.


---- Sample Script to obtain physical reads ---------
select
to_char(sn.snap_time, 'YYYY-MM-DD HH24') start_time,
sum(b.value - a.value) val
from perfstat.stats$sysstat a,
perfstat.stats$sysstat b,
perfstat.stats$snapshot sn
where sn.snap_id = a.snap_id
and a.snap_id = b.snap_id -1
and a.statistic# = 42
and b.statistic# = 42
and snap_time >= to_date('2009-01-26', 'YYYY-MM-DD')
and snap_time < to_date('2009-01-31', 'YYYY-MM-DD')
group by to_char(sn.snap_time, 'YYYY-MM-DD HH24')
;
---- END of Sample Script to obtain physical reads ---------

---- Sample Script to obtain physical writes---------

select
to_char(sn.snap_time, 'YYYY-MM-DD HH24') start_time,
sum(b.value - a.value) val
from perfstat.stats$sysstat a,
perfstat.stats$sysstat b,
perfstat.stats$snapshot sn
where sn.snap_id = a.snap_id
and a.snap_id = b.snap_id -1
and a.statistic# = 46
and b.statistic# = 46
and snap_time >= to_date('2009-01-26', 'YYYY-MM-DD')
and snap_time < to_date('2009-01-31', 'YYYY-MM-DD')
group by to_char(sn.snap_time, 'YYYY-MM-DD HH24')
;
---- END of Sample Script to obtain physical writes---------



2. Average Time Per Read (ATPR) in every hour during weekdays


In the Statspack report, there is a section called "Tablespace I/O Statistics", along with the "File I/O section" they can be used to identify whether I/O is especially slow or there are an exceptional number of I/Os on any specific data file or tablespaces. There is a field called "Av Rd (ms)", representing average time per read in milli-second. Generally speaking, 20ms - 40ms reads may be considered slow for single block reads. In this post, I used a script to count the number of tablespaces that have ATPR greater than 100ms.


----- Sample script to obtain the number of tablespaces
------ that have ATPR greater than 100ms
select mydate, sum(case when atpr_ms > 100 then 1 else 0 end)
from (
select
to_char(snap_time,'yyyy-mm-dd HH24') mydate,
old.tsname,
sum(new.phyrds-old.phyrds) phy_rds,
sum(new.phywrts-old.phywrts) phy_wrts,
sum(new.readtim-old.readtim) read_tim,
sum(new.writetim-old.writetim) write_tim,
decode(sum(new.phyrds-old.phyrds), 0,0,
sum(new.readtim-old.readtim) *10 / sum(new.phyrds-old.phyrds) ) atpr_ms
from
perfstat.stats$filestatxs old,
perfstat.stats$filestatxs new,
perfstat.stats$snapshot sn
where
old.snap_id = sn.snap_id
and
old.filename = new.filename
and
new.snap_id = sn.snap_id + 1
and
(new.phyrds-old.phyrds) > 0
and snap_time >= to_date('2009-01-26', 'YYYY-MM-DD')
and snap_time < to_date('2009-01-31', 'YYYY-MM-DD')
group by
to_char(snap_time,'yyyy-mm-dd HH24'),
old.tsname
)
group by mydate
;


3. Grapth the I/O pattern:

Sample Graph - Total IO changes with every hour
Sample Graph - Average Time Per Read Count

Friday, February 27, 2009

Using Case When

Considering the following table, suppose I want to count how many values associated with the same key are greater than 10, what can I do?


SQL> select * from t;

KEY VAL
---------- ----------
1 1
1 2
1 10
1 11
2 1
2 2
2 3
3 13
3 33
3 3

10 rows selected.


If I use the following statement, the problem is it does not show key=2, in which case there are no values greater than 10:


SQL> select key, count(*) from t where val > 10 group by key;

KEY COUNT(*)
---------- ----------
1 1
3 2


 

Case when probably is what I want:


SQL> select key, sum(case when val > 10 then 1 else 0 end) from t group by key;

KEY SUM(CASEWHENVAL>10THEN1ELSE0END)
---------- --------------------------------
1 1
2 0
3 2

Table mode import - same table name scenario

I have tested the folloiwng scenario in a 9i database:
- we have two tables with same name under different schemas
- we want to imp these two tables.


exp.par looks like:
-----
file=test.dmp
tables=
system.t,
denis.t
-----

imp.par looks like:
-----
file=test.dmp
fromuser=system,denis
touser=system,denis
tables=
t
-------
Note: I only specified one table name 't' in the par file

The following is the screen output during imp:


$ imp denis/oracle parfile=imp.par

Import: Release 9.2.0.8.0 - Production on Fri Feb 27 09:22:39 2009

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.


Connected to: Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning option
JServer Release 9.2.0.8.0 - Production

Export file created by EXPORT:V09.02.00 via conventional path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses WE8ISO8859P1 character set (possible charset conversion)
. importing SYSTEM's objects into SYSTEM
. . importing table "T" 1 rows imported
. importing DENIS's objects into DENIS
. . importing table "T" 1 rows imported
Import terminated successfully without warnings.



I verified that different table T created under desired schemas after imp.

Monday, February 16, 2009

Passing in a shell variable to AWK program used as a match pattern

I have a need to get the backlog message of a shareplex replicaton queue based on the queue name, I am not aware that there is a simple command in SharePlex to do it. So I developed a script. I have learned how to pass in a shell variable to the AWK program. This variable will thus be used as a match pattern in the AWK program.

The outoput of the SharePlex command I know to show the backlog messages looks like:

$ sp_ctrl qstatus on myhostname

Queues Statistics for myhostname
Name: QueueD (o.prddb-o.rptdb) (MTPost queue)
Number of messages: 11825 (Age 7 min; Size 21547 mb)
Backlog (messages): 126 (Age 0 min)

Name: QueueA (o.prddb-o.rptdb) (MTPost queue)
Number of messages: 809345 (Age 48 min; Size 414840 mb)
Backlog (messages): 806421 (Age 44 min)

Name: QueueB (o.prddb-o.rptdb) (MTPost queue)
Number of messages: 49512 (Age 158 min; Size 71059 mb)
Backlog (messages): 14425 (Age 4 min)

Name: QueueC (o.prddb-o.rptdb) (MTPost queue)
Number of messages: 9678 (Age 8 min; Size 129632 mb)
Backlog (messages): 155 (Age 0 min)

Name: QueueE (o.prddb-o.rptdb) (MTPost queue)
Number of messages: 2 (Age 0 min; Size 28042 mb)
Backlog (messages): 0 (Age 0 min)




My script:



----- script backlog_q.sh -------
#!/bin/ksh
# script bakclog_q.sh
# display backlog of a queuue

qname="QueueB"
sp_ctrl qstatus on myhostname /usr/xpg4/bin/awk -v pat="$qname" '
begin
{
if ( $0 ~ pat ) {
getline
getline
print $3
}
}' read backlog

echo $backlog

---- end of the script --------

Saturday, February 14, 2009

Set up read-only materialized view replication

I was assigned a task to evaluate the feasibility of seting up materialized view replication for another team. Currently this team obtains the data from one of our reporting databases through export and import. So I started to understand materialized view replication. As the first step, I build a testing materialized view replication environment on my pc. Below are some notes as the result of this effort.

 

=========================================
== Set up read-only MVIEW replication ==
=========================================

Environment:
Master site (DBT920) : 9.2.0.4
Materialized view site (DBT10G) : 10.2.0.2

Reference:
1. Metalink Note 256235.1 Scripts to create Trusted / Untrusted ReadOnly MVIEW Replication Sites
2. http://www.hpfuchs.com/2008/02/06/materialized-view-replication/

Overview
~~~~~~~~~~
1. create users at master site
2. create users at mview site
3. create database link at mview site
4. create mview logs at master site
5. create mview at mview site
6. create mview groups
7. Some basic operations


Detailed Steps
~~~~~~~~~~~~~~~

1. create users at master site

-- run as system

create user mviewproxy identified by mviewproxy;

grant create session to mviewproxy;
grant create any table to mviewproxy;
grant comment any table to mviewproxy;
grant select any table to mviewproxy;

BEGIN
dbms_repcat_admin.register_user_repgroup(
username => 'mviewproxy',
privilege_type => 'proxy_mviewadmin',
list_of_gnames => NULL);
END;
/

Note: Not sure if this register_user_repgroup is necessary


2. create users at mview site

-- run as system

create user mvowner identified by oracle
default tablespace users temporary tablespace temp;
grant connect, resource to mvowner;
grant create materialized view to mvowner;
grant create database link to mvowner;

-- run as system

CREATE USER mviewadmin IDENTIFIED BY mviewadmin;
ALTER USER mviewadmin DEFAULT TABLESPACE users;
ALTER USER mviewadmin TEMPORARY TABLESPACE temp;

EXECUTE dbms_repcat_admin.grant_admin_any_schema('mviewadmin');
GRANT comment any table TO mviewadmin;
GRANT lock any table TO mviewadmin;

GRANT create any materialized view TO mviewadmin;
GRANT alter any materialized view TO mviewadmin;


3. create db link at mview site

-- run as system
CREATE PUBLIC DATABASE LINK DBT92.US.ORACLE.COM USING 'DBT92';

Note: not sure why this is necessary.

-- run as mvowner
create database link DBT92.US.ORACLE.COM connect to mviewproxy identified by mviewproxy
using 'DBT92';

4. create mview logs at master site

-- run as mviewproxy
create materialized view log on mstowner.big_table tablespace mviewlog;


Note: 1. mview logs residing in their own tablespace, i.e.
create tablespace mviewlog datafile '/u03/oracle/oradata/DBT92/mviewlog01.dbf' size 50M;
2. mviewproxy does not have the privs to alter/drop materialized view log

5. create mview at mview site
login as mvowner

-- run as mvowner
CREATE MATERIALIZED VIEW mvowner.t_mv REFRESH FAST AS SELECT * FROM mstowner.t@DBT92.US.ORACLE.COM;
CREATE MATERIALIZED VIEW mvowner.big_table_mv REFRESH FAST AS SELECT * FROM mstowner.big_table@DBT92.US.ORACLE.COM;

Note:
Oracle will create three objects when executing the following command:
CREATE MATERIALIZED VIEW mvowner.big_table_mv REFRESH FAST AS SELECT * FROM mstowner.big_table@DBT92.US.ORACLE.COM;

OBJECT_NAME OBJECT_TYPE
------------------------------ -------------------
BIG_TABLE_MV TABLE
BIG_TABLE_PK INDEX
BIG_TABLE_MV MATERIALIZED VIEW



6. create mview groups at mview site

-- create the refresh group for the mview to ensure transactional
-- consistency when refreshing nore than one mview in the group.

-- run as mviewadmin on the mview side

begin
dbms_refresh.make(
name => 'RG_BIG_TABLE',
list => 'mvowner.big_table_mv',
next_date => sysdate,
interval => 'sysdate + 1/24',
implicit_destroy => true,
lax => true);
end;
/


7. Some basic operations

(1) Refresh the complete group
execute dbms_refresh.refresh('RG_BIG_TABLE');
(2) Refresh a single snapshot
execute dbms_snapshot.refresh('mvowner.big_table_mv');


(3) Check materialized veiw refresh status


set linesize 120
set pagesize 100
col owner format a20
col table_name format a20
col name format a20
col master_owner format a20
col master_link format a20
col next format a20

select owner
,name
-- ,table_name
-- ,master_owner
-- ,master
-- ,master_link
,to_char(last_refresh, 'YYYY-MON-DD HH24:MI:SS') lst_rfrsh
,next
,status
from dba_snapshots
/


(4) views:
dba_snapshots
dba_mviews