Monday, January 27, 2014

Move ASM database files from one diskgroup to another

In the past week, I have been involved in a migration of a standalone database to RAC with ASM through datab pump export and import.  After the migration was done, one of the ASM diskgroup is more than 98% percent used, which triggerred alerts from BMC Patrol, which is a standard monitoring tool in my working environment. Searching My Oracle Support website, I found the following note:

How to move ASM database files from one diskgroup to another ? (Doc ID 330103.1)


The note is pretty straight forword, however, there was a one suprise when I tried to offline a datafile :

ORA-01145: offline immediate disallowed unless media recovery enabled

That was because I put the db in noarchivelog mode in order to speed up the data pump import jobs. After I configured the archivelog mode, thing went fine.

Here are the steps I followed for future reference

1) Identify the data file to be moved

+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_01.dbf            NON_TTQ_CLB_MSG
2)  Identify the diskgroup on to which the file has to be moved.

+DATA_09/roeprdsc/datafile

3) Take the file offline.

 ALTER DATABASE DATAFILE '+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_01.dbf' OFFLINE;

4. Copy the data file

I used DBMS_FILE_TRANSFER package. We can also use RMAN, detail see the note.
     
          
create or replace directory orcl1 as '+DATA_01/roeprdsc/datafile';
        
create or replace directory orcl2 as '+DATA_09/roeprdsc/datafile';
      
 Note, if need to create a directory in a diskgroup, syntax looks like


 ASM> Alter disgroup asmdsk2 add directory  '+asmdsk2/test';        
       
BEGIN
 DBMS_FILE_TRANSFER.COPY_FILE(
 source_directory_object => 'ORCL1',
 source_file_name => 'non_ttq_clb_msg_01.dbf',
 destination_directory_object => 'ORCL2',
 destination_file_name => 'non_ttq_clb_msg_01.dbf');
END;                          
/



5) Rename the file to point to new location.

ALTER DATABASE RENAME FILE '+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_01.dbf' TO
 '+DATA_09/roeprdsc/datafile/non_ttq_clb_msg_01.dbf' ;


6) Recover the file.

  RECOVER DATAFILE '+DATA_09/roeprdsc/datafile/non_ttq_clb_msg_01.dbf'

7) Bring the file online.


 ALTER DATABASE DATAFILE '+DATA_09/roeprdsc/datafile/non_ttq_clb_msg_01.dbf' online;


8) Verify the new file location.
         
SQL> SELECT FILE_NAME FROM DBA_DATA_FILES where file_name like '%clb_msg%';

FILE_NAME
--------------------------------------------------------------------------------
...
+DATA_09/roeprdsc/datafile/non_ttq_clb_msg_01.dbf
+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_02.dbf
+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_03.dbf
+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_04.dbf
+DATA_01/roeprdsc/datafile/non_ttq_clb_msg_05.dbf

..



9) Delete the file from its original location either per SQLPLUS or per ASMCMD:

Syntax examples:

SQL:ASM> ALTER DISKGROUP ASMDSK2 DROP FILE users.256.565313879;

or:   ASMCMD> rm -rf 

Note:
Most Automatic Storage Management files do not need to be manually deleted because, as Oracle managed files, they are removed automatically when they are no longer needed. ( To not to have the alert come again, I have to do this)



Tuesday, January 14, 2014

Reference: Clean up SharePlex Queues

We use SharePlex replication as our DR solution for couple of applicatins. Last Saturday night, we did a DR test for one of the very important applications, but we could not make replication working from DR site to other target. As we ran out of time in the window, we just brought up application back to the production site without fixing issues. Today I involved vendor support to test DR configuration (with a dummy configuration) again to make sure that if real DR situation happens replication will work. I have learned that the key is that we should clean up orphan or corrupted queues before activating a configuration in DR. ( note: when we issue deactivate config , suppose all associated queues with this particular conifg will be gone, if not, those left queues need to be cleaned up).


Below are the steps to clean up SharePlex queues for future reference:


1. shutdown or shutdown force at source and target
2. qview -i
3. qview> qsetup
4. qview> qstatus
5. qview> deleteq p   -- for post
   qview> deleteq x   -- for export
6. On target:  truncate splex.shareplex_trans;

Thursday, October 10, 2013

Can Oracle GoldenGate DDL replication support interval partition?

We have a fast-growing table that requires keeping only two months' worth of data. We plan to take advantage of the interval partition feature in Oracle 11g, in which Oracle automatically creates an interval partition as data for that partition is inserted. Therefore, we will not worry about data load failure situation due to the possibility of forgetting to add new partitions by DBA.

In our environment, this table will also be in Oracle GoldenGate replication configuration.   So I try to do tests to confirm whether the GG DDL replication supports the following two operations:

1. Create interval partition

2. Drop partition

I used a test GG environment, in which  a two-way GG replication with DDL replication enabled  is set up between two databases: WESTDB and EASTDB; and the tables are in different schemas called west and east respectively.

The following are the testing steps:

1. create a table named interval_tab in the source db
 denis@WESTDB>> CREATE TABLE west.interval_tab
   2  ( prod_id        NUMBER(6)
   3   , cust_id        NUMBER
   4   , time_id        DATE
   5  )
   6  PARTITION BY RANGE (time_id)
   7  INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))
   8   ( PARTITION p0 VALUES LESS THAN (TO_DATE('1-1-2013', 'MM-DD-YYYY')),
   9     PARTITION p1 VALUES LESS THAN (TO_DATE('2-1-2013', 'MM-DD-YYYY')),
  10     PARTITION p2 VALUES LESS THAN (TO_DATE('3-1-2013', 'MM-DD-YYYY')),
  11     PARTITION p3 VALUES LESS THAN (TO_DATE('4-1-2013', 'MM-DD-YYYY')) );

 Table created.

Verify it is replicated in target:
 denis@EASTDB>> desc east.interval_tab;
  Name                                                                          Null?    Type
  ----------------------------------------------------------------------------- -------- ---------------------
  PROD_ID                                                                                NUMBER(6)
  CUST_ID                                                                                NUMBER
  TIME_ID                                                                                DATE

2. create indexes and contraint At source:
 denis@WESTDB>> create unique index west.interval_tab_pk on west.interval_tab(prod_id, time_id) local;

 Index created.

 denis@WESTDB>> create index west.interval_tab_ix1 on west.interval_tab(cust_id) local;

 Index created.

 denis@WESTDB>> alter table west.interval_tab add constraint interval_tab_pk primary key (prod_id, time_id) using index;

 Table altered.

Verify index and constraints creation are replicated at target:
 denis@EASTDB>> @tabix
 Enter value for tabowner: east
 Enter value for tabname: interval_tab

 TABLE_NAME           INDEX_NAME           COLUMN_NAME             COL_POS UNIQUENES
 -------------------- -------------------- -------------------- ---------- ---------
 INTERVAL_TAB         INTERVAL_TAB_IX1     CUST_ID                       1 NONUNIQUE
 INTERVAL_TAB         INTERVAL_TAB_PK      PROD_ID                       1 UNIQUE
 INTERVAL_TAB                              TIME_ID                       2 UNIQUE


 denis@EASTDB>> select owner,table_name,  constraint_name from dba_constraints where table_name='INTERVAL_TAB';

 OWNER           TABLE_NAME           CONSTRAINT_NAME
 --------------- -------------------- --------------------
 EAST            INTERVAL_TAB         INTERVAL_TAB_PK

3. Insert data at source that will cause new interval partition created automatically

Before the insert, check the current partitions:
 denis@WESTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER      TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ---------------- -------------------- -------------------- ---------------------------------------------
 WEST             INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
                                                                         HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')
Perform the following insert at source and check the partitions:

insert into interval_tab values (1004,1, sysdate);
 denis@WESTDB>> insert into west.interval_tab values (1004,1, sysdate);

 1 row created.

 denis@WESTDB>> commit;

 Commit complete.

 denis@WESTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER                    TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ------------------------------ -------------------- -------------------- ---------------------------------------------
 WEST                           INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         SYS_P81              TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 5 rows selected.

4. Check partitions at target database
 denis@EASTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER            TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ---------------------- -------------------- -------------------- ---------------------------------------------
 EAST                   INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         SYS_P123             TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 5 rows selected.


 denis@EASTDB>> select index_owner,index_name,partition_name from dba_ind_partitions where index_name='INTERVAL_TAB_IX1';

 INDEX_OWNER                    INDEX_NAME           PARTITION_NAME
 ------------------------------ -------------------- --------------------
 EAST                           INTERVAL_TAB_IX1     P0
 EAST                                                P1
 EAST                                                P2
 EAST                                                P3
 EAST                                                SYS_P123

 5 rows selected.

5. Drop a partition at source alter table west.interval_table drop partition P0;
 denis@WESTDB>> alter table west.interval_tab drop partition P0;

 Table altered.


 Verified, it is replicable


 denis@EASTDB>> /

 TABLE_OWNER                    TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ------------------------------ -------------------- -------------------- ---------------------------------------------
 EAST                           INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         SYS_P123             TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 4 rows selected.


In conclusion, the tests confirmed that GG DDL replication support the interval partition automatic creation and drop partition operations. It is worth noting that when we use DBMS_METADATA package to get the definitions of interval partitioned table and its associated local indexes, we won't see the interval partitions that are created automatically. To verify the existance of the partitions, we shall use dba_tab_partitions and dba_ind_partitions views.

P.S.

Note from Oracle docs:

You cannot explicitly add a partition to an interval-partitioned table unless you first lock the partition, which triggers the creation of the partition. The database automatically creates a partition for an interval when data for that interval is inserted. In general, you only need to explicitly create interval partitions for a partition exchange load scenario.

update Apr 9, 2014 -

I may have misunderstood the relationship of DDL replication and interval partition creation.  DDL replication may have no relationship with interval partition creation at all. Even without DDL replication enabled. Interval partition will be created due to the data insertion DML statement. 

Wednesday, September 11, 2013

Avoid Merge Join Cartesian in a SQL Tunning Exercise

Encountered a query that caused CPU utilization high. In a 15 min AWR, this query executes 78 times with total Buffer Gets 949M, which contributes 78.8% of the total. I filled in some bind variable values by checking v$sql_bind_capture view. And I executed the sql from sqlplus with gather_plan_statistics hint. Below is the execution plan witn E-Rows and A-Rows info :
Plan hash value: 4161037915

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                  | Name                   | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  | Writes |  OMem |  1Mem | Used-Mem | Used-Tmp|

-------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------ 
|   1 |  SORT ORDER BY                             |                        |      1 |      1 |    113K|00:02:27.54 |    4242K|  12873 |   8580 |    76M|  3013K|   67M (0)|         | 
|   2 |   HASH UNIQUE                              |                        |      1 |      1 |    113K|00:02:23.74 |    4242K|  12873 |   8580 |    72M|  7323K| 9538K (1)|   73728 | 
|*  3 |    FILTER                                  |                        |      1 |        |    114K|00:03:25.41 |    4242K|   4293 |      0 |       |       |          |         | 
|   4 |     NESTED LOOPS                           |                        |      1 |      1 |    114K|00:00:15.77 |    4239K|   4291 |      0 |       |       |          |         | 
|   5 |      MERGE JOIN CARTESIAN                  |                        |      1 |      1 |    199K|00:00:01.71 |    6001 |    123 |      0 |       |       |          |         | 
|   6 |       MERGE JOIN CARTESIAN                 |                        |      1 |      1 |   7112 |00:00:00.44 |    5998 |    123 |      0 |       |       |          |         | 
|*  7 |        HASH JOIN OUTER                     |                        |      1 |      1 |    889 |00:00:00.15 |    5995 |    123 |      0 |   985K|   927K| 1229K (0)|         |
|   8 |         NESTED LOOPS                       |                        |      1 |      1 |    889 |00:00:00.11 |    5979 |    123 |      0 |       |       |          |         | 
|   9 |          NESTED LOOPS                      |                        |      1 |      1 |    889 |00:00:00.10 |    5088 |    123 |      0 |       |       |          |         | 
|* 10 |           HASH JOIN OUTER                  |                        |      1 |      1 |    889 |00:00:00.09 |    4197 |    123 |      0 |   928K|   928K| 1265K (0)|         | 
|  11 |            NESTED LOOPS                    |                        |      1 |      1 |    865 |00:00:00.10 |    4193 |    123 |      0 |       |       |          |         |
|  12 |             NESTED LOOPS                   |                        |      1 |      1 |   1839 |00:00:00.07 |     504 |    123 |      0 |       |       |          |         | 
|  13 |              MERGE JOIN CARTESIAN          |                        |      1 |      1 |      1 |00:00:00.01 |      12 |      0 |      0 |       |       |          |         | 
|  14 |               NESTED LOOPS                 |                        |      1 |      1 |      1 |00:00:00.01 |       8 |      0 |      0 |       |       |          |         | 
|  15 |                NESTED LOOPS                |                        |      1 |      1 |      1 |00:00:00.01 |       5 |      0 |      0 |       |       |          |         | 
|  16 |                 TABLE ACCESS BY INDEX ROWID| OZBJZFDS               |      1 |      1 |      1 |00:00:00.01 |       3 |      0 |      0 |       |       |          |         | 
|* 17 |                  INDEX UNIQUE SCAN         | PK_OZBJZFDS            |      1 |      1 |      1 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         | 
|  18 |                 TABLE ACCESS BY INDEX ROWID| OZBJZFD_CATEGORY       |      1 |     26 |      1 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         | 
|* 19 |                  INDEX UNIQUE SCAN         | PK_OZBJZFD_CATEGORY    |      1 |      1 |      1 |00:00:00.01 |       1 |      0 |      0 |       |       |          |         | 
|* 20 |                TABLE ACCESS BY INDEX ROWID | OZBJZFD_MARKETS        |      1 |      1 |      1 |00:00:00.01 |       3 |      0 |      0 |       |       |          |         | 
|* 21 |                 INDEX RANGE SCAN           | PK_OZBJZFD_MARKETS     |      1 |      1 |      1 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         | 
|  22 |               BUFFER SORT                  |                        |      1 |      1 |      1 |00:00:00.01 |       4 |      0 |      0 |  2048 |  2048 | 2048  (0)|         | 
|* 23 |                TABLE ACCESS BY INDEX ROWID | MARKETS                |      1 |      1 |      1 |00:00:00.01 |       4 |      0 |      0 |       |       |          |         | 
|* 24 |                 INDEX RANGE SCAN           | PK_MARKETS             |      1 |      1 |      1 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         | 
|* 25 |              TABLE ACCESS BY INDEX ROWID   | OZBJZFD_OQNCVDSS       |      1 |      1 |   1839 |00:00:00.07 |     492 |    123 |      0 |       |       |          |         | 
|* 26 |               INDEX RANGE SCAN             | PK_OZBJZFD_OQNCVDSS    |      1 |      1 |   1894 |00:00:00.01 |      16 |     13 |      0 |       |       |          |         |
|* 27 |             TABLE ACCESS BY INDEX ROWID    | OQNCVDSS               |   1839 |      1 |    865 |00:00:00.03 |    3689 |      0 |      0 |       |       |          |         |
|* 28 |              INDEX UNIQUE SCAN             | PK_OQNCVDSS            |   1839 |      1 |   1839 |00:00:00.01 |    1841 |      0 |      0 |       |       |          |         | 
|  29 |            VIEW                            |                        |      1 |     29 |     29 |00:00:00.01 |       4 |      0 |      0 |       |       |          |         | 
|  30 |             SORT UNIQUE                    |                        |      1 |     29 |     29 |00:00:00.01 |       4 |      0 |      0 |  4096 |  4096 | 4096  (0)|         | 
|  31 |              UNION-ALL                     |                        |      1 |        |     29 |00:00:00.01 |       4 |      0 |      0 |       |       |          |         | 
|  32 |               INDEX FULL SCAN              | PK_SOURCE_TARGET_RULES |      1 |     20 |     20 |00:00:00.01 |       1 |      0 |      0 |       |       |          |         | 
|  33 |               TABLE ACCESS FULL            | CARRYOVER_ISOC_MAPPING |      1 |      9 |      9 |00:00:00.01 |       3 |      0 |      0 |       |       |          |         |
|  34 |           TABLE ACCESS BY INDEX ROWID      | SPEED_CODES            |    889 |      1 |    889 |00:00:00.01 |     891 |      0 |      0 |       |       |          |         |
|* 35 |            INDEX UNIQUE SCAN               | PK_SPEED_CODES         |    889 |      1 |    889 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         | 
|  36 |          TABLE ACCESS BY INDEX ROWID       | OQNCVDS_TYPES          |    889 |      1 |    889 |00:00:00.01 |     891 |      0 |      0 |       |       |          |         | 
|* 37 |           INDEX UNIQUE SCAN                | PK_OQNCVDS_TYPES       |    889 |      1 |    889 |00:00:00.01 |       2 |      0 |      0 |       |       |          |         |
|  38 |         INDEX FAST FULL SCAN               | PK_OFFER_PROD          |      1 |   1255 |   1255 |00:00:00.01 |      16 |      0 |      0 |       |       |          |         | 
|  39 |        BUFFER SORT                         |                        |    889 |      8 |   7112 |00:00:00.01 |       3 |      0 |      0 |  2048 |  2048 | 2048  (0)|         | 
|  40 |         TABLE ACCESS FULL                  | BILLING_FREQ           |      1 |      8 |      8 |00:00:00.01 |       3 |      0 |      0 |       |       |          |         | 
|  41 |       BUFFER SORT                          |                        |   7112 |     28 |    199K|00:00:00.20 |       3 |      0 |      0 |  2048 |  2048 | 2048  (0)|         | 
|  42 |        TABLE ACCESS FULL                   | UNIT_TYPES             |      1 |     28 |     28 |00:00:00.01 |       3 |      0 |      0 |       |       |          |         | 
|* 43 |      TABLE ACCESS BY INDEX ROWID           | UHCDN_RATES            |    199K|      1 |    114K|00:02:16.61 |    4233K|   4168 |      0 |       |       |          |         | 
|* 44 |       INDEX RANGE SCAN                     | PK_UHCDN_RATES         |    199K|     36 |   4879K|00:01:48.28 |     727K|    911 |      0 |       |       |          |         | 
|* 45 |     INDEX RANGE SCAN                       | PK_OZBJZFD_OQNCVDSS    |    976 |      1 |    933 |00:00:00.02 |    2928 |      2 |      0 |       |       |          |         | 
|* 46 |      INDEX UNIQUE SCAN                     | PK_OQNCVDSS            |      9 |      1 |      4 |00:00:00.01 |      18 |      0 |      0 |       |       |          |         |

-------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------

The AUTOTRACE statistics is as follows:
Statistics
----------------------------------------------------------
         72  recursive calls
          0  db block gets
    4242325  consistent gets
      13434  physical reads
          0  redo size
    9673175  bytes sent via SQL*Net to client
      13494  bytes received via SQL*Net from client
       1141  SQL*Net roundtrips to/from client
          5  sorts (memory)
          0  sorts (disk)
     113976  rows processed

By comparing the E-rows and A-rows from the execution, it is easy to identify that the problem starts from the operation id 25 and 26, where E-rows=1 and A-rows=1839 and 1894.
|* 25 |              TABLE ACCESS BY INDEX ROWID   | OZBJZFD_OQNCVDSS       |      1 |      1 |   1839 |00:00:00.07 |        ... 
|* 26 |               INDEX RANGE SCAN             | PK_OZBJZFD_OQNCVDSS    |      1 |      1 |   1894 |00:00:00.01 |        ...                   
With E-rows=1, Oracle CBO decides to use "MERGE JOIN CARTESIAN". Notice at the end, CBO estimate only 1 row whereas actual number of rows is 113K. So the key to tune this query is to avoid the Cartesian join at operation id 5 and 6. I modified the query by adding the following hints and of course make sure the tables order is correct in the FROM clause:

/*+ ordered use_hash(E), use_hash(F) */

Here is the execution plan  of the modified query:
---------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                  | Name                   | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                           |                        |     1 |  2841 |    74   (9)| 00:00:01 |
|   1 |  SORT ORDER BY                             |                        |     1 |  2841 |    74   (9)| 00:00:01 |
|   2 |   HASH UNIQUE                              |                        |     1 |  2841 |    73   (7)| 00:00:01 |
|*  3 |    FILTER                                  |                        |       |       |         |     |
|*  4 |     HASH JOIN OUTER                        |                        |     1 |  2841 |    72   (6)| 00:00:01 |
|   5 |      NESTED LOOPS                          |                        |     1 |  2835 |    68   (5)| 00:00:01 |
|   6 |       NESTED LOOPS                         |                        |     1 |  2828 |    67   (5)| 00:00:01 |
|*  7 |        HASH JOIN                           |                        |     1 |  2813 |    66   (5)| 00:00:01 |
|   8 |         NESTED LOOPS                       |                        |     1 |  2801 |    64   (5)| 00:00:01 |
|   9 |          NESTED LOOPS                      |                        |     1 |  2781 |    63   (5)| 00:00:01 |
|* 10 |           HASH JOIN OUTER                  |                        |     1 |  2682 |    17  (18)| 00:00:01 |
|  11 |            NESTED LOOPS                    |                        |     1 |   578 |    11   (0)| 00:00:01 |
|  12 |             NESTED LOOPS                   |                        |     1 |   326 |    10   (0)| 00:00:01 |
|  13 |              MERGE JOIN CARTESIAN          |                        |     1 |   263 |     7   (0)| 00:00:01 |
|  14 |               NESTED LOOPS                 |                        |     1 |   153 |     5   (0)| 00:00:01 |
|  15 |                NESTED LOOPS                |                        |     1 |   125 |     3   (0)| 00:00:01 |
|  16 |                 TABLE ACCESS BY INDEX ROWID| OZBJZFDS               |     1 |   120 |     2   (0)| 00:00:01 |
|* 17 |                  INDEX UNIQUE SCAN         | PK_OZBJZFDS            |     1 |       |     1   (0)| 00:00:01 |
|  18 |                 TABLE ACCESS BY INDEX ROWID| OZBJZFD_CATEGORY       |     1 |     5 |     1   (0)| 00:00:01 |
|* 19 |                  INDEX UNIQUE SCAN         | PK_OZBJZFD_CATEGORY    |     1 |       |     0   (0)| 00:00:01 |
|* 20 |                TABLE ACCESS BY INDEX ROWID | OZBJZFD_MARKETS        |     1 |    28 |     2   (0)| 00:00:01 |
|* 21 |                 INDEX RANGE SCAN           | PK_OZBJZFD_MARKETS     |     1 |       |     1   (0)| 00:00:01 |
|  22 |               BUFFER SORT                  |                        |     1 |   110 |     5   (0)| 00:00:01 |
|* 23 |                TABLE ACCESS BY INDEX ROWID | MARKETS                |     1 |   110 |     2   (0)| 00:00:01 |
|* 24 |                 INDEX RANGE SCAN           | PK_MARKETS             |     1 |       |     1   (0)| 00:00:01 |
|* 25 |              TABLE ACCESS BY INDEX ROWID   | OZBJZFD_OQNCVDSS       |     1 |    63 |     3   (0)| 00:00:01 |
|* 26 |               INDEX RANGE SCAN             | PK_OZBJZFD_OQNCVDSS    |     1 |       |     2   (0)| 00:00:01 |
|* 27 |             TABLE ACCESS BY INDEX ROWID    | OQNCVDSS               |     1 |   252 |     1   (0)| 00:00:01 |
|* 28 |              INDEX UNIQUE SCAN             | PK_OQNCVDSS            |     1 |       |     0   (0)| 00:00:01 |
|  29 |            VIEW                            |                        |    29 | 61016 |     5  (40)| 00:00:01 |
|  30 |             SORT UNIQUE                    |                        |    29 |  1138 |     5  (80)| 00:00:01 |
|  31 |              UNION-ALL                     |                        |       |       |         |     |
|  32 |               INDEX FULL SCAN              | PK_SOURCE_TARGET_RULES |    20 |   760 |     1   (0)| 00:00:01 |
|  33 |               TABLE ACCESS FULL            | CARRYOVER_ISOC_MAPPING |     9 |   378 |     2   (0)| 00:00:01 |
|* 34 |           TABLE ACCESS BY INDEX ROWID      | UHCDN_RATES            |    21 |  2079 |    46   (0)| 00:00:01 |
|* 35 |            INDEX RANGE SCAN                | PK_UHCDN_RATES         |    72 |       |     2   (0)| 00:00:01 |
|  36 |          TABLE ACCESS BY INDEX ROWID       | OQNCVDS_TYPES          |     1 |    20 |     1   (0)| 00:00:01 |
|* 37 |           INDEX UNIQUE SCAN                | PK_OQNCVDS_TYPES       |     1 |       |     0   (0)| 00:00:01 |
|  38 |         TABLE ACCESS FULL                  | BILLING_FREQ           |     8 |    96 |     2   (0)| 00:00:01 |
|  39 |        TABLE ACCESS BY INDEX ROWID         | UNIT_TYPES             |     1 |    15 |     1   (0)| 00:00:01 |
|* 40 |         INDEX UNIQUE SCAN                  | PK_UNIT_TYPES          |     1 |       |     0   (0)| 00:00:01 |
|  41 |       TABLE ACCESS BY INDEX ROWID          | SPEED_CODES            |     1 |     7 |     1   (0)| 00:00:01 |
|* 42 |        INDEX UNIQUE SCAN                   | PK_SPEED_CODES         |     1 |       |     0   (0)| 00:00:01 |
|  43 |      INDEX FAST FULL SCAN                  | PK_OFFER_PROD          |  1255 |  7530 |     3   (0)| 00:00:01 |
|* 44 |     INDEX RANGE SCAN                       | PK_OZBJZFD_OQNCVDSS    |     1 |    12 |     3   (0)| 00:00:01 |
|* 45 |      INDEX UNIQUE SCAN                     | PK_OQNCVDSS            |     1 |     6 |     1   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------------------

AUTOTRACE statistics of the modified query is also shown below:
Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
     479315  consistent gets
       2097  physical reads
          0  redo size
    9673175  bytes sent via SQL*Net to client
      13535  bytes received via SQL*Net from client
       1141  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
     113976  rows processed

It can be seen that after tunning the "consistent gets" drop to 479,315 from 4,242,325. In the  production database, I created a SQL Profile to enforce the better plan.

Monday, September 09, 2013

My Experience of Data Pump Export and Import for Database Migration

Recently I have carried out a test of using Data Pump Export and Import utilities to migrate a 17 TB database from Solaris to Linux platform, also from single-instance, conventional file system to a two-node RAC on ASM. During the exercise I have learned and become more familiar with quite a few interesting features or options of Data Pump Export and Import utilites and encountered a few difficulties as well.

The first obstacle I met was that I was unable to do Data Pump Export to the NFS storage. We have the NFS storage that is shared between the Solaris and Linux server. At first I tried to perform Data Pump Export to the NFS storage directly. Unfortunately, the job was just stuck there for long time. Having researched this issue for a while and tried the options mentioned in a MOS note (See References [1]) with the help of a SA, I just could not get this worked out. Finally we decided to export to the local file system first then move the dump files to the NFS. I was able to move about 150 GB dump file per hour. Fortunately, the Data Pump Import from NFS storage did not pose any problems.

Table mode Data Pump Export and Import were used in the test. Large tables were typically exported or imported individually and smaller tables were grouped together into several batches. For each large table or a group of smaller tables, I prepared the corresponding parameter files with parallel option if appropriate. In the Import job, I always excluded the index, constraint and statistics. I used SQLFILE option of the impdp command to extract the DDL commands related to the tables, including create table, index and constraint statements etc. Those DDLs were executed after the Import with parallel option for index creation and "ENABLE NOVALIDATE" for constraints typically. Therefore the database migration actually consisted of multiple tables migration jobs. For each tables migration, I performed and recorded the timing of the following steps: export, move dump file to NFS, import, DDLs for index etc. Those tables migration jobs were overlapped as two or three of them were executed at the same time and could be in different steps.

Shortly after the onset of the test I found out that Data Pump Export was extremely slow on a big table with lob column. The  export job was unable to run in parallel regardless of the parallel setting. The segment sizes of this big table are 237 GB and 382 GB for table and lob respectively. The table uses a sequence-based PK column called "EMAIL_SEQ_ID". To speed up the export, I started 23 export jobs at the same time, each job exported a particular range of rows based on the PK. The key is to use the QUERY option for the expdp command. The QUERY option contains a WHERE clause that specifies the PK range. Finally I was able to export this table in 30 hours, move the 420 GB dump files in 2 hours, import in about 15 hours and execute DDL for indexes etc in 2 hours. It is worth noting that although we can perform the lob table export job in parallel by such an approach, we cannot do the same for import. The import job will hold a TM lock for the table, so only one import job can run at a time. I used this approach for several big lob tables. Appendix showed example Shell script to prepare expdp or impdp parfile. By the way, later I found that unable to export in parallel for lob table is a known issue in the Oracle community. (see Reference [3]). A blog post suggested using rowid based approach to separate the rows of lob tables. (see Reference [4])

Occasionally I made mistakes and needed to terminate an export or import job. I found it very convenient to use KILL_JOB command in the interactive mode. First, I need to know the job name either from log file or from dba_datapump_jobs view. Then I can enter the interactive mode using attach option:

expdp / attach=[job_name]

Then I issued "KILL_JOB" command. Note when using KILL_JOB to the expdb, all dump files will be wiped out.

During the test, I have also learned that Data Pump Import can be done through db link. When specifying a source database link for the NETWORK_LINK option of impdp command, import job will retrieve data from the db link, and write the data directly to the target database There are no dump files involved.

I have mentioned previously SQLFILE option can be used to extract DDLs. But this option only available for impdp. Suppose I have a table in a database and I want to extract DDLs related to this table including indexes, constraints, object grants, comments, triggers etc associated with it, how should I do? Previously I always use DBMS_METADATA package for such tasks. If we create a db link to the database itself, we can use impdp with NETWORK_LINK and SQLFILE options to accomplish it very neatly.

Using export/import to migrate the database has one advantage over Transportable Tablespace approach at least: it allows the re-organization of tables and indexes. i.e. it is easy to move tables and indexes to different tablespaces if desired.

I completed the 17 TB database migration with the size of about 12 TB in the target database in about a month. The test was interrupted by other tasks with higher priority. I did total 17 tables migrations to complete the whole database migration. Ordered by end-to-end time (including export, move to NFS, import, DDL steps), they were refered to as as batch 1 to 17 here:

batch   time (hours) 
-----   ----------- 
1        50
2        30
3        25
4        24
5        21
6        16
7        15
8        14
9        11
10       10
11       10
12       9.5
13       5.5
14       5
15       3.5
16       3
17       3
------------



Based on above timing, if I perform the migration task as continuely as possible, I may be able to compete it in 5-7 days. The ultimate goal is to do a zero-down time migration. We plan to use oracle Golden Gate to caputure the changes at source during the database migration. So my next task will be investiagating whether this is possible in practise in our environment.

References:
[1] 781349.1 - ORA-27054: NFS file system where the file is created or resides is not mounted with correct options [ID 781349.1])
[2] Sample Export and Import parameter files:
 -- expdp.par  ----


 DIRECTORY=dpump_dir3
 DUMPFILE=dpump_dir3:email%U_q2.dmp
 TABLES=TTQ_GMPX.TTQ_EMAIL_EVENT_DETAILS
 LOGFILE=dpump_dir3:TTQ_EMAIL_EVENT_DETAILS_q2.log
 #PARALLEL=20
 FILESIZE=5G
 JOB_NAME=exp_email_q2
 QUERY="WHERE email_seq_id >=2*200000000   and email_seq_id < 3*200000000"


  -- impdp.par ----

 DIRECTORY=dpump_dir
 DUMPFILE=dpump_dir:email%U_q2.dmp
 TABLES=TTQ_GMPX.TTQ_EMAIL_EVENT_DETAILS
 TABLE_EXISTS_ACTION=APPEND
 LOGFILE=dpump_dir:TTQ_EMAIL_EVENT_DETAILS_q2_imp.log
 PARALLEL=2
 JOB_NAME=imp_email_q2
 EXCLUDE=index,constraint,statistics
 #EXCLUDE=statistics
 transform=storage:n,oid:n
 CLUSTER=NO

[3] Bug 5599947 - Export Data Pump is slow when table has a LOB column - Defect: Bug:5599947 "DATAPUMP EXPORT VERY SLOW"

[4] http://jensenmo.blogspot.com/2012/10/optimising-data-pump-export-and-import.html

[5] Some other references:
Master Note for Data Pump [ID 1264715.1]
Checklist For Slow Performance Of DataPump Export (expdp) And Import (impdp) [ID 453895.1]
Parallel Capabilities of Oracle Data Pump [ID 365459.1]


Appendix - sample shell script to generate expdp/impdp par file for big lob table

#!/bin/ksh
i=21
while [[ $i -le 43 ]];do
 
  ((j=i+1))
  PARFILE="EMAIL_CONTENT_q$i.par"
  echo "DIRECTORY=dpump_dir3" > $PARFILE 
  echo "DUMPFILE=dpump_dir3:ecntnt_q${i}_%U.dmp" >> $PARFILE 
  echo "TABLES=TTQ_GMPX.EMAIL_CONTENT" >> $PARFILE
  echo "LOGFILE=dpump_dir3:EMAIL_CONTENT_q$i.log" >> $PARFILE 
  echo "FILESIZE=10G" >> $PARFILE
  echo "JOB_NAME=exp_ecntnt_q$i" >> $PARFILE
  echo "QUERY=\"WHERE email_seq_id > $i * 2000000  and email_seq_id <= $j * 2000000\"">> $PARFILE
 
 
  echo "i=$i"
  echo "j=$j"
  ((i=i+1))
done

--- sample script to generate impdp par file for big lob table -----------------

i=6
while [[ $i -le 43 ]];do
 
  ((j=i+1))
  PARFILE="EMAIL_CONTENT_imp_q$i.par"
  echo "DIRECTORY=dpump_dir" > $PARFILE 
  echo "DUMPFILE=ecntnt_q${i}_%U.dmp" >> $PARFILE 
  echo "TABLES=TTQ_GMPX.EMAIL_CONTENT" >> $PARFILE
  echo "TABLE_EXISTS_ACTION=APPEND" >> $PARFILE
  echo "LOGFILE=EMAIL_CONTENT_q${i}_imp.log" >> $PARFILE 
  echo "JOB_NAME=imp_ecntnt_q${i}" >> $PARFILE
  echo "#EXCLUDE=index,constraint,statistics" >> $PARFILE
  echo "EXCLUDE=statistics" >> $PARFILE 
  echo "transform=storage:n,oid:n" >> $PARFILE
  echo "CLUSTER=NO" >> $PARFILE
 
 
  echo "i=$i"
  echo "j=$j"
  ((i=i+1))
done