"Redirection" after "dsp:include" wont work in ATG.

Normally in J2EE, we can redirect to another page after  any "include" in the page.

For example this will work

<@include page="test2.jsp"/>
  <jsp:forward page="test3.jsp"/>

or

<jsp:include page="test2.jsp"/>
<jsp:forward page="test3.jsp"/>

But once you have dsp:include in page, you will not be able to redirect to any other page using any redirection operation like "Redirect" droplet or "jsp:forward" or request dispatcher etc.

For example:
test1.jsp

<%@ taglib uri="dsp" prefix="dsp"%>
<dsp:page>
   I am in test1.jsp <br>

  <%-- including test2.jsp --%>
  <dsp:include page="test2.jsp"/>
 
  <%-- redirect to page test3.jsp --%>
  <dsp:droplet name="/atg/dynamo/droplet/Redirect">
    <dsp:param name="url" value="test3.jsp"/>
  </dsp:droplet>

  end of test1.jsp <br>
</dsp:page>

test2.jsp

<%@ taglib uri="dsp" prefix="dsp"%>
<dsp:page>
 I am in test2.jsp <br>
</dsp:page>

test3.jsp

<%@ taglib uri="dsp" prefix="dsp"%>
<dsp:page>
  I am in test3.jsp <br>
</dsp:page>

if you access http://localhost:8080/test1.jsp
output -

I am in test1.jsp
I am in test2.jsp
end of test1.jsp

 If you remove " <dsp:include page="test2.jsp"/>" in test1.jsp and
access http://localhost:8080/test1.jsp
you will be redirected to test3.jsp

I am in test3.jsp

If you use jsp:include or @include instead of dsp:include like below

  <jsp:include page="test2.jsp"/>
  <dsp:droplet name="/atg/dynamo/droplet/Redirect">
    <dsp:param name="url" value="test3.jsp"/>
  </dsp:droplet>

You will be redirected to test3.jsp. 

Using ATG development operation tags.

Generating DDL's of any repository
If you have created a new repository or new item-descriptor in reposiotry in your module.
To generate ddl,just start the instance (ignore table missing errors).
Once instance is up, go to that particular repository in dynamo component browser (/dyn/admin/)
In "Run XML Operation Tags on the Repository"
use

<print-ddl/>

Which will give you DDL for that repository.

output -

-- drop table www_instock_wic;
CREATE TABLE www_instock_wic (
    primary_wic         INTEGER    NOT NULL,
    primary_sku_id         varchar2(254)    NOT NULL,
    instock_wic         INTEGER    NOT NULL,
    ship_sub_wic         INTEGER    NOT NULL,
    ship_sub_sku_id     varchar2(254)    NOT NULL,
    update_dttm         DATE    NULL,
    sub_wic_desc         varchar2(250)    NULL,
    sub_start_date         DATE    NULL,
    PRIMARY KEY(primary_wic)
);

If your repository is version repository since $class for that repository is pointing to
 atg.adapter.version.VersionRepository

DDL generated will have all columns which are needed for versioning. 

-- drop table dcs_catalog;
CREATE TABLE dcs_catalog (
    catalog_id         varchar2(254)    NOT NULL,
    asset_version         INTEGER    NOT NULL,
    version         INTEGER    NULL,
    creation_date         DATE    NULL,
    last_mod_date         DATE    NULL,
    migration_status     INTEGER    NULL,
    migration_index     INTEGER    NULL,
    item_acl         CLOB    NULL,
    display_name         varchar2(254)    NOT NULL,
    version_deleted     number(1)    NULL,
    version_editable     number(1)    NULL,
    pred_version         INTEGER    NULL,
    workspace_id         varchar2(254)    NULL,
    branch_id         varchar2(254)    NULL,
    is_head         number(1)    NULL,
    checkin_date         DATE    NULL,
    CHECK (version_deleted IN (0, 1)),
    CHECK (version_editable IN (0, 1)),
    CHECK (is_head IN (0, 1)),
    PRIMARY KEY(catalog_id, asset_version)
); 

You can also generate ddl's using startSQLRepository.

In <ATG-HOME>/bin/

bin\startSQLRepository -m <module-name> -repository /atg/commerce/catalog/ProductCatalog -outputSQLFile product_catalog.sql

Testing RQL's - query-items

<query-items item-descriptor="inventory"> catalogRefId="sku386558" </query-items>

output -

Query: catalogRefId="sku386558" returns 1 items:
<add-item item-descriptor="inventory" id="inv1000">
  <!-- export is false   <set-property name="version"><![CDATA[4]]></set-property>  -->
  <set-property name="catalogRefId"><![CDATA[sku386558]]></set-property>
  <set-property name="creationDate"><![CDATA[__NULL__]]></set-property>
  <set-property name="startDate"><![CDATA[1/14/2005 05:10:17]]></set-property>
  <set-property name="availabilityStatus"><![CDATA[OUTOFSTOCK]]></set-property>
</add-item>

--- Querying using date or timestamp by using RQL
With date

<query-items item-descriptor="order"> creationDate>date("2011-10-10") and state="SUBMITTED" </query-items>

With timestamp

<query-items item-descriptor="order"> creationDate>date("2011-10-10 10:00:00 EST") and state="SUBMITTED" </query-items>

To check what items are in cache

<dump-caches item-descriptors="skuSpecialPriceRules" dump-type="both"/>

output -  

Desc: skuSpecialPriceRules mItemCache: 2 items, 0, weak items
LRU: [skuSpecialPriceRules:9400059, skuSpecialPriceRules:5100045]
mItems: {5100045=skuSpecialPriceRules:5100045,
9400059=skuSpecialPriceRules:9400059}
mWeakItems: {}
OnDeckItemCache: null
NullItemCache: {}

*** begin pre-cache XML output
<load-items item-descriptor="skuSpecialPriceRules">5100045,9400059</load-items>
*** end pre-cache XML output

Other operation tags
<add-item>, <update-item>, <remove-item> -to add or update or remove items
<export-items>, <import-items> - to export and import items
<print-item> - to print item
<transaction> - to maintain transactions while adding or removing items.

Reference - http://download.oracle.com/docs/cd/E23095_01/Platform.93/RepositoryGuide/html/s1201sqlrepositoryreference01.html

How to bypass dynamo logs from log4J in JBOSS?

By default,  when using JBOSS server ATG overrides
$class with  $class=atg.nucleus.logging.commons.CommonsLoggingLogListener
in \DafEar\Tomcat\config\tomcatconfig.jar/atg/dynamo/service/logging/ScreenLog.properties

So logs with colorizer will be like below (click to enlarge)

If you observe all debugs and infos will be in green and error log is split into multiple lines.
To get back normal and clean dynamo logs with ATG colorizer

If EAR is non-standalone
    In <ATG-HOME>/<LOCALCONFIG>/atg/dynamo/service/logging/ScreenLog.properties
If EAR is standalone
    In <ATG-DATA>/<LOCALCONFIG>/atg/dynamo/service/logging/ScreenLog.properties
have below lines

$class=atg.nucleus.logging.PrintStreamLogger
useNucleusPathForClassName=true
useFullPaths=true
suppressTimestamp=false

and comment log4j section in <JBOSS-HOME>/server/<server-name>/conf/jboss-service.xml

   <!-- Log4j Initialization --> 
   <!--
   <mbean code="org.jboss.logging.Log4jService"
      name="jboss.system:type=Log4jService,service=Logging"
      xmbean-dd="resource:xmdesc/Log4jService-xmbean.xml">
      <attribute name="ConfigurationURL">resource:jboss-log4j.xml</attribute>
      <attribute name="Log4jQuietMode">true</attribute>
      <attribute name="RefreshPeriod">60</attribute>
   </mbean>
   -->

Now you will get plain old dynamo logs - neater and simple. (click to enlarge)

 You can observe the difference between screenshots with same log statements. I feel the later one is better :)

How to take thread dumps in JBOSS?

If you see any request is taking a lot of time, in order to know what might be the issue, you can go through thread dumps.  Thread may be waiting for some resource (it can be database call or any third party call), you can know exact cause by looking at thread dumps.

What is a thread dump?
   Current snapshot of threads which are running in JVM.

By using jboss jmx-console

http://localhost:8080/jmx-console

and search "serverInfo" and click on that link
Click invoke under listThreaddump
Which will give you current snapshot of threads which are running in JVM.

By using twiddle
go to <JBOSS-HOME>/bin
In windows - 

twiddle.bat invoke "jboss.system:type=ServerInfo" listThreadDump > dump.html

dump.html will have your thread dump.

In unix-based machines

/jboss/bin/twiddle.sh invoke "jboss.system:type=ServerInfo" listThreadDump > dump.html 2>&1

Using "Interrupt" signal - SIGQUIT
Use kill -3 <process-id> to generate thread dump. You will find the thread dumps in server logs.

Typically to know exact cause you need to take at least 2 to 3 thread dumps with 30 seconds gap, so that you will know what is the common thread which is waiting for long time.

How to correct projects in unknown state in BCC/CA ?

Often deployments will fail due to some reason and even if we use cancel button sometimes ATG fails to revert the project to previous state. And because of JVM crash or normal bounce of BCC instance, currently deploying projects will go to unknown state, follow below steps to correct such type of projects.


If BCC instance is bounced while deploying a project to staging
we can get back the project previous state so that we can start from Author state

-- remove asset locks on the project. if any
delete from avm_asset_lock where workspace_id =
(select id from avm_devline where name =
(select workspace from epub_project where project_id = 'prj49001'));

-- change states. Locked to false and editable to true
--   locked to 0 and editable to 1
update epub_project
set locked=0
where project_id = 'prj49001';

update epub_project
set editable=1
where project_id = 'prj49001';

-- change states. Locked to false and editable to true
--   locked to 0 and editable to 1
update avm_workspace
set locked=0
where ws_id =
(select id from avm_devline where name =
(select workspace from epub_project where project_id = 'prj49001'));

update avm_workspace
set editable=1
where ws_id =
(select id from avm_devline where name =
(select workspace from epub_project where project_id = 'prj49001'));

-- finally change state to author which is 3 for all workflows
update epub_ind_workflow
set state=3
where process_id =
(select process_id from epub_process where project='prj49001');
commit;

Invalidate below repository caches
/atg/epub/PublishingRepository/
/atg/epub/version/VersionManagerRepository/

If BCC instance is bounced while deploying a project to production
we can get back the project previous state so that we can redeploy to
production.

--If project with workflow editCommerceAssets.wdl
--stuck while deploying to production

update epub_ind_workflow
set state=11
where process_id =
(select process_id from epub_process where project = 'prj49001');
commit;

-- For projects which are created by auto import workflow

-- If project with /Commerce/autoImportAssets.wdl workflow
-- stuck while production

update epub_ind_workflow
set state=10
where process_id =
(select process_id from epub_process where project = 'prj49001');
commit;

Invalidate only /atg/epub/PublishingRepository/

Completely delete a project

-- Removing locks of the project if any
delete from avm_asset_lock where workspace_id in
(select id from avm_devline where name in
(select workspace from epub_project where project_id = 'prj243002'))

-- delete history of the project
delete from EPUB_PR_HISTORY where project_id in
(select project_id from epub_project where project_id = 'prj243002');

-- delete the project
delete from epub_project where project_id = 'prj243002';

-- delete history of the process
delete from EPUB_PROC_HISTORY where process_id in
(select process_id from epub_process where project = 'prj49001');

-- delete task information of process
delete from EPUB_PROC_TASKINFO where id in
(select process_id from epub_process where project = 'prj49001');

-- delete states of project (if any)
delete  from EPUB_IND_WORKFLOW where process_id in
(select process_id from epub_process where project = 'prj49001');

-- finally delete the process
delete from epub_process where where project = 'prj49001';

commit;

---- Extra Sqls that may give more info --

To give all conflicting projects

select * from epub_project where workspace in
(select name from avm_devline where id in
(select workspace_id from avm_asset_lock))

To give all conflicting projects for a particular asset.

select * from epub_project where workspace in
(select name from avm_devline where id in
(select workspace_id from dcs_category where category_id = 'cat3484800002' and category_id in
(select repository_id from avm_asset_lock where descriptor_name='category')))

Understanding dead locks in Inventory Repository

How locking happens in InventoryManager?
Whenever we call InventoryManager.purchase and InventoryManager.setAvailabilityStatus to update inventory stock level and availability status ATG acquires row-level locking on the inventory item which it is trying to update.

What is row-level locking?
Once a dml query is fired on a row, oracle OOTB will acquire lock on that row and lock on that row will be released once commit or rollback is done on that row. In between acquiring and releasing lock, no other session can acquire lock on same row to update (but still others can read that row). Other transactions (which wants to update the same row) will wait until the lock on that row is released. You can find all locks in v$ object tables like v$lock, v$session ..

Typically, while order is getting placed or once order is placed we need to update inventory.

$class of  InventoryManager OOTB points to atg.commerce.inventory.RepositoryInventoryManager

In atg.commerce.inventory.RepositoryInventoryManager
purchase() {
  //  transaction is required.
  // call getInventoryItemForUpdate
  // call updateItem()
  // end transaction
}

getInventoryItemForUpdate() {
   //  call lock()
   //  get inventory item to update
}

lock(){
    // ensure transaction
    // until InventoryManager.maximumRetriesPerRowLock (default 5)
    // open connection - ((atg.adapter.gsa.GSARepository)getRepository()).getDataSource().getConnection()
    // ps = c.prepareStatement(sql); (sql is InventoryManager.inventoryRowLockSQL - UPDATE dcs_inventory SET inventory_lock = ? WHERE catalog_ref_id = ?)

    // ps.setString(1, lock);  (lock - "inventoryManager")
    // ps.setString(2, pItemId); (pItemId - "skuId")
    // rowcount = ps.executeUpdate();
    // ps.close();
    //        close the connection
    // c.close();
}

Once lock() method is called for one InventoryItem, it will fire udpate sql on that particular row  (the value which it is trying to update is not important here. Which row it is trying to update is important), by default in one transaction if one dml query is fired oracle OOTB acquires lock on that row (you can find those locks in v$objects if you have dba access) and lock on that row is released only once the transaction is either roll backed or committed.


How inventory dead locks happen?
For example
person1 cart has 2 sku's - sku-A and sku-B
person 2 cart has 2 sku's - sku-B and sku-A

If both are trying to place orders at same time.
thread-x of person1 got lock on sku-A inventory item.
thread-y of person 2 got lock on sku-B inventory item.
thread-x waits to get lock on sku-B since thread-y locked it.
thread-y waits to get lock on sku-A since thread-x locked it.
There is dead-lock between thread-x and thread-y

How to avoid those dead locks?
ATG OOTB we have atg.commerce.inventory.RepositoryInventoryManager.acquireInventoryLocks(List pItemIds)
which will sorts the sku's before getting locks on them.
You need to pass list of sku id's which are in the cart to acquireInventoryLocks method.
Once you call acquireInventoryLocks and then call purchase method you can avoid dead locks.

so now

person1 cart has 2 sku's - sku-A and sku-B
person 2 cart has 2 sku's - sku-B and sku-A

If both are trying to place orders at same time.
// first call acquireInventoryLocks - sort and get locks
// second call purchase

thread-x of person1 got lock on sku-A inventory item.
thread-y of person2 waits to get lock on sku-A since thread-x  has lock on it.

Note: OOTB we have releaseInventoryLocks which does nothing, it wont release locks. Locks are only released once the transaction is committed or roll backed.

Note: for some reason, because of out-of-memory or normal bounce if the instance is bought down, then the row-locks which are acquired will be still be locked forever.  If this updates are being done in a processor. Move this processor to last in the pipeline (and use sql JMS to update inventory status/level).