Showing posts with label GoldenGate. Show all posts
Showing posts with label GoldenGate. Show all posts

Tuesday, April 14, 2015

Monitoring Oracle GoldenGate Latency

Today I've implemented an approach to monitor OGG latency. Here I will describe what I've done.  

1. Create a table gg_latency in source and target databases:
create table gg_latency
(
  extr varchar2(10),
  pump varchar2(10),
  repl varchar2(10),
  update_time date
);



alter table gg_latency add constraint gg_latency_pk primary key(extr, pump, repl) using index;

2. Create a procedure that is used to update the latency table: 

create or replace procedure proc_update_gg_latency
is
begin
  for rec in ( select * from gg_latency)
  loop
     update gg_latency set update_time=sysdate where extr=rec.extr and pump=rec.pump and repl = rec.repl;
     commit;
  end loop;
end;
/
3. Populate the table with every possible combination of the processing group names:

For example, in my replication enviroment, at source I have three Extract groups, three Pump groups, at target I have 15 Replicat groups :
insert into gg_latency(extr, pump, repl) values('ECRUDR1', 'PCRUDR1', 'CRURDR1');
insert into gg_latency(extr, pump, repl) values('ECRUDR1', 'PCRUDR1', 'CRURDR1A');
insert into gg_latency(extr, pump, repl) values('ECRUDR1', 'PCRUDR1', 'CRURDR1B');
insert into gg_latency(extr, pump, repl) values('ECRUDR1', 'PCRUDR1', 'CRURDR1C');
insert into gg_latency(extr, pump, repl) values('ECRUDR1', 'PCRUDR1', 'CRURDR1D');
  
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2A');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2B');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2C');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2D');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2F');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2G');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2H');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2M');
insert into gg_latency(extr, pump, repl) values('ECRUDR2', 'PCRUDR2', 'CRURDR2N');

insert into gg_latency(extr, pump, repl) values('ECRUDR3', 'PCRUDR3', 'CRURDR3');

4. For each EXTRACT group parameter file at source, add the TABLE clause with WHERE option for the GG_LATENCY table , e.g.

    TABLE DB_ADMIN.GG_LATENCY WHERE ( EXTR="ECRUDR1");

Note: do this for all the EXTRACT groups


5. For each PUMP group parameter file at source, add the TABLE clause with WHERE option for the GG_LATENCY table , e.g.
    TABLE DB_ADMIN.GG_LATENCY, WHERE (PUMP="PCRUDR1");
Note: add the line before the PASSTHRU if exists, do this for all the PUMP groups


 6. For each REPLICAT group parameter file at target, add MAP clause with WHERE option for the GG_LATENCY table , e.g.

MAP DB_ADMIN.GG_LATENCY, TARGET DB_ADMIN.GG_LATENCY, WHERE (REPL='CRURDR1');
 
Note: do this for all the REPLICAT groups. In 12c OGG, single quotation mark should be used for literal string.

7. Bounce all processes as parameter files are modified 


8. Create a scheduler job to update the latency table every minute
begin
DBMS_SCHEDULER.create_job (
   job_name         => 'UPDATE_GG_LATENCY_TABLE',
    job_type        => 'PLSQL_BLOCK',
    job_action      => 'BEGIN db_admin.proc_update_gg_latency; END;',
    start_date      => trunc(sysdate, 'HH24'),
    repeat_interval => 'freq=minutely',
    end_date        => NULL,
    enabled         => TRUE
    );
end;
/

9. Check latency by the following query at target:
 
SQL> select extr, pump, repl, update_time, round((sysdate - update_time) *24*60) latency_mins from gg_latency;

EXTR       PUMP       REPL       UPDATE_TIME          LATENCY_MINS
---------- ---------- ---------- -------------------- ------------
ECRUDR1    PCRUDR1    CRURDR1D   14-Apr-2015 12:46:00            1
ECRUDR1    PCRUDR1    CRURDR1B   14-Apr-2015 12:46:00            1
ECRUDR1    PCRUDR1    CRURDR1A   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2D   14-Apr-2015 12:46:00            1
ECRUDR1    PCRUDR1    CRURDR1C   14-Apr-2015 12:46:00            1
ECRUDR1    PCRUDR1    CRURDR1    14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2H   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2C   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2N   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2B   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2A   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2M   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2G   14-Apr-2015 12:46:00            1
ECRUDR2    PCRUDR2    CRURDR2F   14-Apr-2015 12:46:00            1
ECRUDR3    PCRUDR3    CRURDR3    14-Apr-2015 12:46:00            1

15 rows selected.

Note: As we update every minute, the smallest unit for latency is a minute.

Thursday, October 10, 2013

Can Oracle GoldenGate DDL replication support interval partition?

We have a fast-growing table that requires keeping only two months' worth of data. We plan to take advantage of the interval partition feature in Oracle 11g, in which Oracle automatically creates an interval partition as data for that partition is inserted. Therefore, we will not worry about data load failure situation due to the possibility of forgetting to add new partitions by DBA.

In our environment, this table will also be in Oracle GoldenGate replication configuration.   So I try to do tests to confirm whether the GG DDL replication supports the following two operations:

1. Create interval partition

2. Drop partition

I used a test GG environment, in which  a two-way GG replication with DDL replication enabled  is set up between two databases: WESTDB and EASTDB; and the tables are in different schemas called west and east respectively.

The following are the testing steps:

1. create a table named interval_tab in the source db
 denis@WESTDB>> CREATE TABLE west.interval_tab
   2  ( prod_id        NUMBER(6)
   3   , cust_id        NUMBER
   4   , time_id        DATE
   5  )
   6  PARTITION BY RANGE (time_id)
   7  INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))
   8   ( PARTITION p0 VALUES LESS THAN (TO_DATE('1-1-2013', 'MM-DD-YYYY')),
   9     PARTITION p1 VALUES LESS THAN (TO_DATE('2-1-2013', 'MM-DD-YYYY')),
  10     PARTITION p2 VALUES LESS THAN (TO_DATE('3-1-2013', 'MM-DD-YYYY')),
  11     PARTITION p3 VALUES LESS THAN (TO_DATE('4-1-2013', 'MM-DD-YYYY')) );

 Table created.

Verify it is replicated in target:
 denis@EASTDB>> desc east.interval_tab;
  Name                                                                          Null?    Type
  ----------------------------------------------------------------------------- -------- ---------------------
  PROD_ID                                                                                NUMBER(6)
  CUST_ID                                                                                NUMBER
  TIME_ID                                                                                DATE

2. create indexes and contraint At source:
 denis@WESTDB>> create unique index west.interval_tab_pk on west.interval_tab(prod_id, time_id) local;

 Index created.

 denis@WESTDB>> create index west.interval_tab_ix1 on west.interval_tab(cust_id) local;

 Index created.

 denis@WESTDB>> alter table west.interval_tab add constraint interval_tab_pk primary key (prod_id, time_id) using index;

 Table altered.

Verify index and constraints creation are replicated at target:
 denis@EASTDB>> @tabix
 Enter value for tabowner: east
 Enter value for tabname: interval_tab

 TABLE_NAME           INDEX_NAME           COLUMN_NAME             COL_POS UNIQUENES
 -------------------- -------------------- -------------------- ---------- ---------
 INTERVAL_TAB         INTERVAL_TAB_IX1     CUST_ID                       1 NONUNIQUE
 INTERVAL_TAB         INTERVAL_TAB_PK      PROD_ID                       1 UNIQUE
 INTERVAL_TAB                              TIME_ID                       2 UNIQUE


 denis@EASTDB>> select owner,table_name,  constraint_name from dba_constraints where table_name='INTERVAL_TAB';

 OWNER           TABLE_NAME           CONSTRAINT_NAME
 --------------- -------------------- --------------------
 EAST            INTERVAL_TAB         INTERVAL_TAB_PK

3. Insert data at source that will cause new interval partition created automatically

Before the insert, check the current partitions:
 denis@WESTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER      TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ---------------- -------------------- -------------------- ---------------------------------------------
 WEST             INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
                  HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST             INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
                                                                         HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')
Perform the following insert at source and check the partitions:

insert into interval_tab values (1004,1, sysdate);
 denis@WESTDB>> insert into west.interval_tab values (1004,1, sysdate);

 1 row created.

 denis@WESTDB>> commit;

 Commit complete.

 denis@WESTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER                    TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ------------------------------ -------------------- -------------------- ---------------------------------------------
 WEST                           INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 WEST                           INTERVAL_TAB         SYS_P81              TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 5 rows selected.

4. Check partitions at target database
 denis@EASTDB>> select table_owner, table_name, partition_name, high_value from dba_tab_partitions where table_name='INTERVAL_TAB';

 TABLE_OWNER            TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ---------------------- -------------------- -------------------- ---------------------------------------------
 EAST                   INTERVAL_TAB         P0                   TO_DATE(' 2013-01-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
          HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                   INTERVAL_TAB         SYS_P123             TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 5 rows selected.


 denis@EASTDB>> select index_owner,index_name,partition_name from dba_ind_partitions where index_name='INTERVAL_TAB_IX1';

 INDEX_OWNER                    INDEX_NAME           PARTITION_NAME
 ------------------------------ -------------------- --------------------
 EAST                           INTERVAL_TAB_IX1     P0
 EAST                                                P1
 EAST                                                P2
 EAST                                                P3
 EAST                                                SYS_P123

 5 rows selected.

5. Drop a partition at source alter table west.interval_table drop partition P0;
 denis@WESTDB>> alter table west.interval_tab drop partition P0;

 Table altered.


 Verified, it is replicable


 denis@EASTDB>> /

 TABLE_OWNER                    TABLE_NAME           PARTITION_NAME       HIGH_VALUE
 ------------------------------ -------------------- -------------------- ---------------------------------------------
 EAST                           INTERVAL_TAB         P1                   TO_DATE(' 2013-02-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         P2                   TO_DATE(' 2013-03-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         P3                   TO_DATE(' 2013-04-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')

 EAST                           INTERVAL_TAB         SYS_P123             TO_DATE(' 2013-11-01 00:00:00', 'SYYYY-MM-DD
           HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')


 4 rows selected.


In conclusion, the tests confirmed that GG DDL replication support the interval partition automatic creation and drop partition operations. It is worth noting that when we use DBMS_METADATA package to get the definitions of interval partitioned table and its associated local indexes, we won't see the interval partitions that are created automatically. To verify the existance of the partitions, we shall use dba_tab_partitions and dba_ind_partitions views.

P.S.

Note from Oracle docs:

You cannot explicitly add a partition to an interval-partitioned table unless you first lock the partition, which triggers the creation of the partition. The database automatically creates a partition for an interval when data for that interval is inserted. In general, you only need to explicitly create interval partitions for a partition exchange load scenario.

update Apr 9, 2014 -

I may have misunderstood the relationship of DDL replication and interval partition creation.  DDL replication may have no relationship with interval partition creation at all. Even without DDL replication enabled. Interval partition will be created due to the data insertion DML statement. 

Monday, June 17, 2013

GoldenGate: Steps of Resynchronizing a Table

In the target, Replicat process was abended due to the following error:

2013-06-17 10:17:30  ERROR   OGG-01163  Oracle GoldenGate Delivery for Oracle, rtdnfrd.prm:  Bad column length (3) specified for column TAX_AUTHORITY_TYPE in table WAX_TDN.V_TAX_DETAILS, maximum allowable length is 1.

I checked TAX_AUTHORITY_TYPE colume, it is char(1) in both source and target databases, so not sure how this error comes up. I have to comment it out in the Replicat parameter file in order to re-start the process. As a result, the table is out-of-sync. This actually gives me a chance to pratice steps of resynchronizing a table in GoldenGate for the first time.

I took the following steps:

1. Comment out the table in the Replicat parameter file
Already did.


2. Stop Replicat and start it again so that it continues for unaffected tables
Already did.

3. Record the time stamp on the source system
2013-06-17 13:49

4 Start a copy of the source data for the affected tables
 
Note: before making the copy, try to resolve any long-running transactions

5. Import the copy to the target table

6. Create a new Replicat group for the out-of-sync table, using Begin to start at the source time stamp that you recorded earlier and using the existing train for ExtTrain

Add Replicat  rvtd, ExtTrail ./dirdat/rb, Begin 2013-06-17 13:49:00

7. Create the new parameter file so that it includes 
  HandleCollisions

replicat rvtd
SETENV(ORACLE_HOME="/apps/opt/oracle/product/11.2.0/db_1")
SETENV(ORACLE_SID = "tdnprdfd1")
SETENV(NLS_LANG=AMERICAN_AMERICA.AL32UTF8)
-- DBOPTIONS SUPPRESSTRIGGERS
userid
gg_owner@tdnprdfd1, password AADAAAAAAAAAAAHAJIKGFGKGRJTBJBCIZGUERJNHBFPBLCAEOBUADFWASJMJCDWEICWGBEGHOIRESCPA, encryptkey securekey1
discardfile ./dirrpt/RTDNFRD.dsc, Append, megabytes 1
handlecollisions
assumetargetdefs
MAP WAX_TDN.V_TAX_DETAILS,             TARGET WAX_TDN.V_TAX_DETAILS;


8. Start the new Replicat

9. View the new Replicat's lag until it shows "At EOF, no more records to process"
GGSCI> Send replicat rvtd, GetLag

10. Turn off HandleCollisions in the new replicate with
GGSCI> send replicat rvtd NoHandleCollisions

11. Edit the parameter file to comment out or remove HandleCollisions if you ever bounce the process later
(The next steps merge the table back with the others so that only one Replicat group is needed

12. Stop Extract at source

13. View both Replicats's lag until you see "EOF" again

GGSCI> Send replicat rtdnfrd, GetLag
GGSCI> Send replicat rvtd, GetLag

14 stop both Replicats

15. Uncomment the resynced table in the original Replicat parameter file

16. Start Extract

17. Start the original Replicat:
GGSC> start rtdnfrd

18. Delete the new Replicat that you created:
GGSCI> delete replicat rvtd

All done!

GoldenGate Replicat Process Abending due to Tablespace Full and Discard File Exceeding Max Bytes

We have a cron job set up to monitor errors in the GoldenGate ggserr.log. This monrining, in a target database, we recieved:

< 2013-06-17 02:58:17  ERROR   OGG-01172  Oracle GoldenGate Delivery for Oracle, rvasip.prm:  Discard file (./dirrpt/RVASIP.dsc) exceeded max bytes (1000000).
< 2013-06-17 02:58:17  ERROR   OGG-01668  Oracle GoldenGate Delivery for Oracle, rvasip.prm:  PROCESS ABENDING.


It appeared that Replicat process abended due to  Discard file exceeded max bytes.
Discard file is used by GoldenGate to log records it cannot proccess. The maximum size of the discard file can be specified by MAXBYTES or MEGABYSTS options,
the defaults are 1000000 or 1MB.  If the specified size is exceeded, the process will abend.

Further troubleshooting showed the reason for the discard file filled up was due to a tablespace filled up in this case. In the RVASIP.dsc files we can found:

OCI Error ORA-01653: unable to extend table PPOWNER.VZ_JNR_FEED_TRX_LOG by 8192 in tablespace PPOWNER_DATA_1 (status = 1653). INSERT INTO "PPOWNER"."VZ_JNR_FEED_TRX_LOG" ("ID","JOURNAL_ID"
,"TRX_EXT_ID","BILLED","DB_MODIFICATION_DATE","DB_CREATION_DATE","SUB_TRX_ID","TRX_ID","CHG_ATTR","ACCESS_TYPE") VALUES (:a0,:a1,:a2,:a3,:a4,:a5,:a6,:a7,:a8,:a9)
Aborting transaction on ./dirdat/rt beginning at seqno 21275 rba 3795007
                         error at seqno 21275 rba 5169654


To fix the problem, I renamed RAVSIP.dsc, changed the max bytes of discard file to be 10MB in the parameter file:

discardfile ./dirrpt/RVASIP.dsc, Append,megabytes 10

Then I stopped and started the Replicat process. I have verified that those discarded dmls recorded in the discard file have been applied after Replicat process re-started. no manual intervene is ndeed.