ATG: November 2011

Friday, November 25, 2011

Regarding ConcurrentUpdateException and InvalidVersionException

There are 3 places where order version is stored
1) Database level - dcspp_order.version
2) Repository item cache level - orderItem.getPropertyValue("version");
3) Session level - order.getVersion();

// This check is done in classes where order is modified
if (order.getVersion() != orderItem.getPropertyValue("version"))
throw atg.commerce.order.InvalidVersionException

// This check is done while updating item
if (dcspp_order.version != orderItem.getPropertyValue("version"))
throw atg.repository.ConcurrentUpdateException

Note: concurrency can be maintained on any repository item, please go through below documentation
http://docs.oracle.com/cd/E26180_01/Platform.94/RepositoryGuide/html/s0709maintainingitemconcurrencywithth01.html

atg.repository.ConcurrentUpdateException - Is thrown if 2 separate sessions (belong to 2 different instances) of same user edit same order simultaneously or if 2 applications are editing same order. One way to reproduce in cluster environment (in production)

This exception is thrown in logs of instance (browser 1 session instance)

CONTAINER:atg.commerce.CommerceException; SOURCE:CONTAINER:atg.service.pipeline.RunProcessException: An exception was thrown from the context of the link named [updateCommerceItemObjects].;

SOURCE:CONTAINER:atg.commerce.CommerceException: Saving order 2288449002 failed because doing so would result in data being overwritten. This save attempt had an out of date repository item [commerceItem].;

SOURCE:atg.repository.ConcurrentUpdateException: no rows updated oldVersion=20 for item=commerceItem:ci55251446 in GSATransaction=atg.adapter.gsa.GSATransaction@6a77cca2

thread=ajp-10.238.160.131-30100-37 transaction=TransactionImple < ac, BasicAction: aeea083:6d44:4ecb6eb0:da1fd5 status: ActionStatus.RUNNING >

Note: Typically item-cache-timeout by default is -1 (when not set) which means item will be in cache until invalidated. Suppose the item-cache-timeout for order is set to 30 mins.

Explanation -

When the user xyz login in one browser 1

suppose his/her session is tied to one instance 1 and suppose his order version is incremented to 2.

logout

Note that user xyz order is cached in this instance 1 and its version is 2 and it will be there in the cache until item-cache-timeout which is 30 mins.

If the same user xyz tries to login using another browser 2

suppose his/her session now got tied to another instance 2 and his order version will be incremented to 3.

logout

Note that user xyz order is cached in this instance 2 and its version is 3 and it will be there in the cache until 30 mins

since orderRepository has simple cache, instance 1 has no idea of updated version.

Note: If the browser 1 cookies are not removed, request will always go to instance 1.

In this 30 mins time, when user xyz login using browser 1, at the time of loadorder since the version in cache (which is 2) doesnt match with version in database

(which is 3) ConcurrentUpdateException is thrown.

This can also happen if the user in application like CSC and dotcom user both working on same order, then above exception is thrown

atg.commerce.order.InvalidVersionException - is caused if 2 sessions (of same instance) of same user modifying same order simultaneously.

One way to reproduce in dev environment (only 1 instance) or you need to make sure that 2 sessions belongs to same instance in cluster environment.

If the user has 2 seperate session in 2 seperate browsers, where both sessions are tied to one instance and since both session are updating same order, InvalidVersionException exceptions is thrown.

Explanation -
Whenever the order is updated, both order (session) version and order repository version are updated.
When updating if both order version and order repository version are not equal, then invalid version exception is thrown.

**** Error Fri Nov 25 15:08:57 CST 2011 1322255337883 /atg/commerce/order/purchase/RepriceOrderDroplet ---

CONTAINER:atg.service.pipeline.RunProcessException:

An exception was thrown from the context of the link named [updateOrderAfterReprice].; SOURCE:atg.commerce.order.InvalidVersionException: This order (2293186179) is out of date.

Changes have been made to the order and the operation should be resubmitted. Order version 4, Repository item version 9.

at atg.service.pipeline.PipelineChain.runProcess(PipelineChain.java:371)

at atg.service.pipeline.PipelineChainContext.runProcess(PipelineChainContext.java:185)

at atg.service.pipeline.PipelineManager.runProcess(PipelineManager.java:453)

at atg.service.pipeline.servlet.PipelineChainInvocation.service(PipelineChainInvocation.java:267)

at atg.commerce.order.purchase.RepriceOrder.service(RepriceOrder.java:438)

at atg.servlet.DynamoServlet.service(DynamoServlet.java:123)

at atg.taglib.dspjsp.DropletTag.invokeServlet(Unknown Source)

at atg.taglib.dspjsp.DropletTag.doAfterBody(Unknown Source)

at org.apache.jsp.checkout.shippinginfo_jsp._jspx_meth_dsp_005fdroplet_005f8(shippinginfo_jsp.java:2128)

....stack trace CROPPED after 10 lines.

Caused by :atg.commerce.order.InvalidVersionException: This order (2293186179) is out of date. Changes have been made to the order and the operation should be resubmitted. Order version 4, Repository item version 9.

at atg.commerce.order.OrderManager.updateOrder(OrderManager.java:2557)

at atg.commerce.order.processor.ProcUpdateOrder.runProcess(ProcUpdateOrder.java:111)

at atg.service.pipeline.PipelineLink.runProcess(PipelineLink.java:233)

at atg.service.pipeline.PipelineChain.runProcess(PipelineChain.java:343)

at atg.service.pipeline.PipelineChainContext.runProcess(PipelineChainContext.java:185)

at atg.service.pipeline.PipelineManager.runProcess(PipelineManager.java:453)

at atg.service.pipeline.servlet.PipelineChainInvocation.service(PipelineChainInvocation.java:267)

at atg.commerce.order.purchase.RepriceOrder.service(RepriceOrder.java:438)

at atg.servlet.DynamoServlet.service(DynamoServlet.java:123)

....stack trace CROPPED after 10 lines.

Tuesday, November 15, 2011

To start Motorprise application with oracle database

Below are the steps to start Motorprise application with oracle database.
To create Motorprise schema
Install oracle XE or oracle SE and connect using system/sys and create motorprise user.

create user motorprise identified by password1 default tablespace users temporary tablespace temp;
grant connect, resource to motorprise;
grant select any table to motorprise;
grant create any table to motorprise;
grant update any table to motorprise;
grant delete any table to motorprise;
grant alter any table to motorprise;
grant create any view to motorprise;

Install Motorprise scripts
Connect using motorprise user

Connect motorprise/password1

Run following script

@<ATGDir>\MotopriseJSP\sql\install\oracle\motorpriseall_ddl.sql
For example:
@C:\ATG\ATG9.1\MotorpriseJSP\sql\install\oracle\motorpriseall_ddl.sql
Commit.;

Exporting test data from Solid to Oracle
1) Check license files. If not available copy license files to your <ATGDir>\home\localconfig
2) Copy solid.db file from

Copy <ATGDir>\DAS\solid\atgdb\solid.db to <ATGDir>\DAS\solid\i486-unknown-win32\
For example
C:\ATG\ATG9.1\DAS\solid\atgdb\solid.db to C:\ATG\ATG9.1\DAS\solid\i486-unknown-win32\

3) Connect to solid database
Add below lines in <ATGDir>\home\localconfig\atg\dynamo\service\jdbc\FakeXADataSource.properties
For example:

C:\ATG\ATG9.1\home\localconfig\atg\dynamo\service\jdbc\FakeXADataSource.properties
$class=atg.service.jdbc.FakeXADataSource
driver=solid.jdbc.SolidDriver
URL=jdbc:solid://localhost:1313
user=motorprise
password=motorprise

4) Start solid database –

Execute <ATGDir>\DAS\solid\ i486-unknown-win32\solfe.exe
C:\ATG\ATG9.1\DAS\solid\i486-unknown-win32\solfe.exe

5) Exporting Motorprise data from solid to xml file

cd C:\ATG\ATG9.1\home
bin\startSQLRepository -m MotorpriseJSP -export all users.xml -repository /atg/userprofiling/ProfileAdapterRepository
bin\startSQLRepository -m MotorpriseJSP -export all products.xml -repository /atg/commerce/catalog/ProductCatalog
bin\startSQLRepository -m MotorpriseJSP -export all priceLists.xml -repository /atg/commerce/pricing/priceLists/PriceLists
bin\startSQLRepository -m MotorpriseJSP -export all inventory.xml -repository /atg/commerce/inventory/InventoryRepository

6) Connect to Oracle database
Now point FakeXADataSource to Oracle.
Add below lines in <ATGDir>\home\localconfig\atg\dynamo\service\jdbc\FakeXADataSource.properties
In C:\ATG\ATG9.1\home\localconfig\atg\dynamo\service\jdbc\FakeXADataSource.properties

$class=atg.service.jdbc.FakeXADataSource
URL=jdbc:oracle:thin:@localhost:1521:xe
user=motorprise
password=password1
needsSeparateUserInfo=true
readOnly=false
driver=oracle.jdbc.xa.client.OracleXADataSource

Note: Make sure you have classes12.jar or ojdbc.jar in classpath
CLASSPATH=C:\oraclexe\app\oracle\product\10.2.0\server\jdbc\lib\ojdbc14.jar;
If you are using oracle XE, run below steps to increase processes, sessions and transactions.

sqlplus sys/oracle as sysdba
alter system set processes = 150 scope = spfile;
alter system set sessions = 300 scope = spfile;
alter system set transactions = 330 scope = spfile;
shutdown immediate;
startup;

7) Import Motorprise data from xml to oracle

cd C:\ATG\ATG9.1\home
bin\startSQLRepository -m MotorpriseJSP -import users.xml -repository /atg/userprofiling/ProfileAdapterRepository
bin\startSQLRepository -m MotorpriseJSP -import products.xml -repository /atg/commerce/catalog/ProductCatalog
bin\startSQLRepository -m MotorpriseJSP -import priceLists.xml -repository /atg/commerce/pricing/priceLists/PriceLists
bin\startSQLRepository -m MotorpriseJSP -import inventory.xml -repository /atg/commerce/inventory/InventoryRepository

Build Motorprise.ear
Create Motorprise.ear – go to <ATGDir>\home\bin

cd C:\ATG\ATG9.1\home\bin
runAssembler C:\ATG\ATG9.1\Motorprise.ear -m DCS.PublishingAgent DafEar.admin MotorPriseJSP

Note: DCS.PublishingAgent is optional. Keep it if you have CA/BCC installed.
Please make sure that your FakeXADataSource is pointing to your local oracle.
Create FakeXADataSource.properties and JTDataSource.properties in
<ATGDir>\ \MotorpriseJSP\config\atg\dynamo\service\jdbc
In FakeXADataSource.properties

In JTDataSource.properties

$class=atg.service.jdbc.MonitoredDataSource
dataSource=/atg/dynamo/service/jdbc/FakeXADataSource

Finally Deploy Motorprise.ear
Use http://localhost:<port>/Motorprise/ to bring up the login page of Motorprise application.

Friday, November 11, 2011

To Prevent Cross site scripting.

By default when submitting form using dsp:form or dsp:a requiresSessionConfirmation is set to true.
Set warnOnSessionConfirmationFailure set to true and enforceSessionConfirmation set to true in /atg/dynamo/Configuration
Once you have above settings, for every dsp:form and dsp:a there will be a hidden _dynSessConf number associated with it and will be same across all forms in that session.

At the time form submit,
if session is invalidated or timed-out or if the _dynSessConf number doesn't match with one in current session,
then
if requiresSessionConfirmation and enforceSessionConfirmation set to true,
response code is set 409 and form is not processed.

Source - http://download.oracle.com/docs/cd/E23095_01/Platform.93/PageDevGuide/html/s0606preventingcrosssiteattacks01.html

Issues with jsessionid in url

When application server will append jsessionid to url?

If the request is first from client and there are no cookies (related to that commerce site) at client side,

then application server will create new session and have that sessionid appended to url (as url agrument parameter seperated by ";" its not query parameter "?")

and will also set jsessionid cookie at the client side.

Why application server is appending to url?
To support browsers which do not allow cookies.
As mentioned in http://randomcoder.org/articles/jsessionid-considered-harmful

"Cookieless sessions are achieved in Java by appending a string of the format ;jsessionid=SESSION_IDENTIFIER to the end of a URL. To do this, all links emitted by your website need to be passed through either HttpServletResponse.encodeURL(), either directly or through mechanisms such as the JSTL <c:out /> tag. Failure to do this for even a single link can result in your users losing their session forever."

For example: in url

http://localhost:8080/store/store/product/view_product_details.jsp;jsessionid=EF8300224952E06E951475223380F9B0?_DARGS=/store/store/product/includes/vpd_product_selection_display_fg.jsp

And in the generated html page -

Issues having jsessionid in url

But having "jsessionid" in the url leads to below problems

1) If we are caching the common pages at AKAMAI level.

For example:

User 1 requested page product_details.jsp and Since it is the first request, appserver includes jsessionid in the urls in the generated page.

Akamai cached the page (product_details.jsp) which got generated for user 1.

When user 2 requested same page, Akamai serves that cached page, the request wont even go to web server or app server.

Now if the user 2 tries to add an item, the form action url contains a session id of user 1 and when it comes to webserver/load balancer

which redirects the request based on sessionid to appserver, the appserver will add the item to user 2 session.

2) Google bot problems as mentioned in http://randomcoder.org/articles/jsessionid-considered-harmful

"To prevent abuse, search engines such as Google associate web content with a single URL, and penalize sites which have identical content reachable from multiple, unique URLs. Because a URL-encoded session is unique per visit, multiple visits by the same search engine bot will return identical content with different URLs. This is not an uncommon problem; a test search for ;jsessionid in URLs returned around 79 million search results."

3) Security issue as mentioned in http://randomcoder.org/articles/jsessionid-considered-harmful

"Because the session identifier is included in the URL, an attacker could potentially impersonate a victim by getting the victim to follow a session-encoded URL to your site. If the victim logs in, the attacker is logged in as well - exposing any personal or confidential information the victim has access to. This can be mitigated somewhat by using short timeouts on sessions, but that tends to annoy legitimate users."

How to fix above issues?

Below are three ways to fix this issue.
1) Fix using filter is mentioned in http://randomcoder.org/articles/jsessionid-considered-harmful

2) Below way using Apache mod_rewrite is mentioned in http://www.digitalsanctuary.com/tech-blog/general/hiding-jsessionid-parameter-from-google.html

Assuming you have mod_rewrite enabled in your Apache instance, use this configuration in your apache config:

# This should strip out jsessionids from google

RewriteCond %{HTTP_USER_AGENT} (googlebot) [NC]

ReWriteRule ^(.*);jsessionid=[A-Za-z0-9]+(.*)$ $1$2 [L,R=301]

This rule says for request where the user agent contains “googlebot” (with case insensitive matching), rewrite the URL without the jsessionid. It seems to work nicely.
By using above way url rewriting is done at web server level.

3)Other way is to use http://www.tuckey.org/urlrewrite/
"Based on the popular and very useful mod_rewrite for apache, UrlRewriteFilter is a Java Web Filter for any J2EE compliant web application server (such as Resin, Orion or Tomcat), which allows you to rewrite URLs before they get to your code. It is a very powerful tool just like Apache's mod_rewrite."

Example -

in /WEB-INF/web.xml

<filter-name>UrlRewriteFilter</filter-name>

<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>

<init-param>

<param-name>logLevel</param-name>

<param-value>INFO</param-value>

</init-param>

<init-param>

<param-name>confPath</param-name>

<param-value>/WEB-INF/urlrewrite.xml</param-value>

</init-param>

</filter>

<filter-mapping>

<filter-name>UrlRewriteFilter</filter-name>

<url-pattern>/*</url-pattern>

</filter-mapping>

in urlrewrite.xml

<rule>
          <from>^(.*);jsessionid=[A-Za-z0-9\-\+\*\._]+(.*)$</from>
          <to type="redirect">$1$2</to>
     </rule>

Please note that this filter mapping should be before PageFilter, so that before handling request to PageFilter the jsessionid should be stripped from url.

Wednesday, November 9, 2011

ATG session management

Please go through below ATG Tech notes on how ATG manages session on third party servers.
https://docs.google.com/open?id=0B8rpgofYlaJnYTBiNGNmNjMtODU0MS00NDNmLThmOWItNGQzN2UwODBjODRi

Application server creates separate session for each web application (war) in EAR.
For example in your application (EAR) test.ear you have common.war (context is /) and store.war (context is store).

In jboss, you can see how many sessions are created for each web application.

For example go to http://localhost:8080/jmx-console/
and search for host=localhost,path=/<web-app-context>,type=Manager, once you click on that link you will find many settings related to session like activeSessions, maxInactiveInterval etc. You can also list all sessions created for this context by invoking listSessionIds()
host=localhost,path=/,type=Manager to see how many sessions are created with context "/"
host=localhost,path=/store,type=Manager to see how many session are created with context "/store"

As mentioned in above ATG Tech note -

By default, DafEar\base\j2ee-components\atg-bootstrap.war is the parent application with a context root of /dyn.
All web applications to define the atg.session.parentContextName and atg.dafear.bootstrapContextName parameters in their web.xml to point to the parent web application.

ATG Nuclues components live outside the application servers session
So a listener (atg.servlet.SessionBindingReporter) is added to each web application session as attribute.
When ever application server creates a new session
      SessionBindingReporter will increment SessionNameContext.mNumWrappingNameContexts
When ever application server invalidates a session
      SessionBindingReporter will decrement SessionNameContext.mNumWrappingNameContexts

Thus when SessionNameContext.mNumWrappingNameContexts reaches zero, which means all parent and child web-app sessions are expired.

ATG Nuclues session scoped components for that session are removed.

Even if you have different timeout set at each war level, highest amongst them is considered for session expiration because ATG will remove session scoped components only when all web app sessions are invalidated.

Handling session expiration during form submission.
To redirect the user to session expiration page when his/her session is invalidated during form submission.

set checkForValidSession to true in your formhandler properties file or set using dsp:input at the time of form submission.
Above way will work only if formhandler extends GenericFormHandler and using "checkFormRedirect" method to redirect to success or failure page.

If above property is set, in GenericFormHandler.checkFormRedirect a form exception is added with key "sessionExpired" and redirected to failure page.

For some OOTB formhandler's like ShoppingCartModifier and CartModifierFormHandler there is other way to do

In both ShoppingCartModifier and CartModifierFormHandler we have property "sessionExpirationURL"
Set this property (as hidden in form) to page where you want the user to redirect if his/her session got invalidated.
Example:

<dsp:input bean="ShoppingCartModifier.sessionExpirationURL" type="hidden" value="../../common/SessionExpired.jsp"/>

Regarding session timeout
Note that since sessions are created and managed by application servers like JBOSS, Weblogic or Websphere. Session timeout set at application server is used (/atg/dynamo/servlet/sessiontracking/SessionManager.sessionInvalidationTime is not considered).

ATG session timeout can be set at each web-app level (web.xml) or at global level
By default session timeout is set to 30 mins in JBOSS
In /cygdrive/c/Ecomm/jboss-eap-4.2/jboss-as/server/<server>/deploy/jboss-web.deployer/conf

   <session-config>
        <session-timeout>30</session-timeout>
    </session-config>

In weblogic - weblogic-application.xml

<?xml version="1.0" encoding="UTF-8"?>
<wls:weblogic-application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wls="http://www.bea.com/ns/weblogic/90" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/j2ee_1_4.xsd http://www.bea.com/ns/weblogic/90 http://www.bea.com/ns/weblogic/90/weblogic-application.xsd">
<wls:application-param>
     <wls:param-name>webapp.encoding.default</wls:param-name>
      <wls:param-value>UTF-8</wls:param-value>
    </wls:application-param>
<wls:session-descriptor>
    <wls:cookie-name>JSESSIONID</wls:cookie-name>
    <wls:timeout-secs>1200</wls:timeout-secs>
    <wls:id-length>14</wls:id-length>
    <wls:sharing-enabled>true</wls:sharing-enabled>
    <wls:url-rewriting-enabled>false</wls:url-rewriting-enabled>
</wls:session-descriptor>
</wls:weblogic-application>

Warn user for session expiration
To notify user for session expiration like a pop up we need to use javascript. Mostly this javascript should be included in header page so that it will be executed in all pages.

function sessionTimeout(){
        var millsec = 29 * 60 * 1000; // depending on your session timeout.
        setTimeout("sessionWarning()", millsec);
}

function sessionWarning(){
    warningURL = "/common/sessiontimeoutlogoutwarning.htm";
    var win = window.open(warningURL, "sessionTimeoutWarning", "width=450,height=175");
    win.focus();
}

// calling session timeout method.
sessionTimeout();