Tuesday, November 30, 2010

Camel Integration with WebSphere MQ

This post will describe my experience with implementing a Camel route that integrates with WebSphere MQ. The Camel route runs inside Servicemix as an OSGI bundle. I used the Enterprise version of Servicemix from FuseSource available here:
http://fusesource.com/downloads/

This use-case was developed to show how to achieve interoperability with WebSphere MQ.
You can download the example code here .

You can extract this file and, using Eclipse, import as an "existing Maven project". You will need to have the M2Eclipse plugin installed.

Overview
The following diagram shows what is happening in the use case created to demonstrate interoperability with WebSphere MQ









The use case consists of a Camel route (deployed inside Servicemix) that receives an HTTP message from an external client. The arrival of the HTTP message triggers the route. The route is pretty simple. It simply takes the payload of the HTTP message and drops it onto a Websphere MQ queue.

The HTTP message received (that triggers the route) is a two-way (request/response.) The message exchange pattern (MEP) for any route is generally determined by the first (consumer) endpoint in a route. Therefore the MEP for this particular route is request/response.

In this route, the JMS producer is responsible for placing the message on the pre-configured MQ queue. Because the route is request/response, the JMS producer will by default wait on a response to arrive on the response queue. This response queue is determined by whatever queue name is set in the outgoing JMS message’s JMSReplyTo header. The JMSReplyTo header is a standard-based JMS message header that indicates which address (queue) the JMS consumer that receives the message, should reply to.

In some cases the replyTo queue could be a temporary queue set up (per-message) by Camel on the fly to listen for reply (dynamic replyTo). In other cases, the replyTo queue will be a well-known location (static replyTo.) In the current use-case the replyTo queue is a well-known (static) location. This follows a pattern common to many JMS applications.

Implementation
The following shows the Camel code for this route.


from("jetty:http://0.0.0.0:8888/placeorder")

.to(targetUri + "?replyTo=SYSTEM.SOAP.RESPONSE.QUEUE");


The code simply listens for incoming HTTP messages (via the camel Jetty component) and forwards the payload to a Websphere MQ queue. The route sets the value of the “JMSreplyTo” header using an option on the outgoing targetUri. Thus the route, after sending to the targetUri queue, will wait on a response on the queue specified in the ‘replyTo’ option. In this case the route will listen for a response on “SYSTEM.SOAP.RESPONSE.QUEUE”. Camel will by default use a message selector to identify the response message, searching for the response message that has a corrrelationId that matching the messageId of the original message. In this way, Camel handles the message correlation for you without additional code. The response then gets converted and sent back to the original client via HTTP.

The Spring configuration for WebSphere MQ, used in above use case is shown below:

<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="
    http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans.xsd
      http://camel.apache.org/schema/spring
    http://camel.apache.org/schema/spring/camel-spring.xsd">

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <routeBuilder ref="mqRoute"/>
</camelContext>

<bean id="mqRoute" class="com.fusesource.camel.MqRoute">
    <!-- This is the "request" queue -->
     <property name="outgoingQueue" value="outgoingPayments"/>
     <!-- This is the "response" queue -->
    <property name="incomingQueue" value="incomingPayments"/>

</bean>

  <bean id="myTransform" class="com.fusesource.camel.MyTransform"/>


<bean id="wmq" class="org.apache.camel.component.jms.JmsComponent">
  <property name="connectionFactory">
    <bean class="com.ibm.mq.jms.MQConnectionFactory">
        <property name="transportType" value="1"/>
        <property name="hostName" value="localhost"/>
        <property name="port" value="1414"/>
        <property name="queueManager" value="QM_TEST"/>
    </bean>
  </property>
</bean>

</beans>
Building the example
First make sure you have built the example and installed into local maven repo:

mvn install -DskipTests=true


You can see the routing rules by looking at the java code in the src/main/java directory. The Spring XML configuration lives in src/main/resources/META-INF/spring/camel-context.xml

WebSphere MQ Dependencies
When implementing a Camel solution to interoperate with WebSphere MQ, there is a required set of IBM-specific libraries needed. For the purpose of this POC, these libraries were obtained by downloading a WebSphere MQ v. 7.0.1.3 trial for Windows.

Making Dependency Available to Runtime
A set of MQ libraries are required to be installed into FUSE ESB (Servicemix) making them available at run time. This set of dependent runtime JARs is provided as part of a MQ installation. The libraries, and the folder from and MQ installation where they can be found is shown below:

%MQ_INSTALL_DIR%\java\lib\OSGI
\com.ibm.mq.osgi.directip_7.0.1.3.jar
\com.ibm.msg.client.osgi.commonservices.j2se_7.0.1.3.jar
\com.ibm.msg.client.osgi.jms.prereq_7.0.1.3.jar
\com.ibm.msg.client.osgi.jms_7.0.1.3.jar
\com.ibm.msg.client.osgi.nls_7.0.1.3.jar
\com.ibm.msg.client.osgi.wmq.nls_7.0.1.3.jar
\com.ibm.msg.client.osgi.wmq.prereq_7.0.1.3.jar
\com.ibm.msg.client.osgi.wmq_7.0.1.3.jar
These JARs packaged as standard OSGI libraries by IBM and therefore recognizable and deployable into any standard OSGI container. These libraries can be deployed into Servicemix easily using the following commands:

osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.jms.prereq_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.jms_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.wmq.prereq_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.wmq_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.nls_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.commonservices.j2se_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.mq.osgi.directip_7.0.1.3.jar
osgi:install -s file:C:/Progra~1/IBM/WebSph~1/java/lib/OSGi/com.ibm.msg.client.osgi.wmq.nls_7.0.1.3.jar
All but the last bundle are deployed with the ‘-s’ option (meaning to auto-start the bundle). The last bundle above is a bundle “fragment” so no start is required.

As a convenience, an optional step is to install these IBM MQ OSGI bundles into your local maven repo (or to a corporate Nexus repository.) If you do this, you would be able to use the Servicemix "mvn:" prefix with osgi:install (rather than "file:") telling Servicemix to use the local Maven settings to find the bundle and install it into the container. First it tries to find "mvn:" prefixed artifacts in the local Maven repository, it then tries any external configured repository.

Other Dependencies
There are a few features you need to make sure are installed into the Servicemix container prior
to deploying the sample.

features:install camel-jetty
features:install camel-jms
features:install camel-spring

Deploying the sample into Servicemix

Once all the dependent runtime MQ bundles are installed into Servicemix, you can install
the example bundle into Servicemix using the following from the servicemix command console:

osgi:install -s mvn:com.fusesource.examples/camel-mq-osgi/1.1

To enable logging for this example type:
log:set DEBUG com.fusesource.camel
For an even higher level of logging (logging Camel specifics)
log:set DEBUG org.apache.camel

Running outside of Servicemix

As an alternative to running this route in Servicemix, you can also run the camel route
in standalone mode outside of Servicemix using the camel maven plugin (see pom). This allows you to start the route using the "mvn camel:run" command.
When you run example using 'mvn camel:run' the camel-maven-plugin will construct a classpath from any Maven dependencies listed in the POM with scope "compile." So in order for maven to find these runtime dependencies, you will need to install this minimal set of IBM runtime libraries in your local maven repository. Similar to the IBM MQ OSGI runtime libraries, these jars can also be found in a installation of IBM MQ. You can install these JARS into your local maven repo as follows (the following assumes you have installed Websphere MQ version 7.)

mvn install:install-file -Dfile=C:\PROGRA~1\IBM\WEBSPH~1\java\lib\com.ibm.mqjms.jar -DgroupId=com.ibm -DartifactId=com.ibm.mqjms -Dversion=7.0.1.3 -Dpackaging=jar
mvn install:install-file -Dfile=C:\PROGRA~1\IBM\WEBSPH~1\java\lib\dhbcore.jar -DgroupId=com.ibm.mq.dhbcore -DartifactId=dhbcore -Dversion=7.0.1.3 -Dpackaging=jar
mvn install:install-file -Dfile=C:\PROGRA~1\IBM\WEBSPH~1\java\lib\com.ibm.mq.jmqi.jar -DgroupId=com.ibm.mq -DartifactId=com.ibm.mq.jmqi -Dversion=7.0.1.3 -Dpackaging=jar

It was necessary to take special action to install these artifacts into a well-known location (local Maven Repo) and tag the artifacts with appropriate naming scheme. Note the values specified for groupId/artifactIds are very sensitive to typos (this was the cause of some time spent debugging during testing - so be careful.) I took the liberty to define groupId and artifactId for the Maven artifacts associated with these IBM libraries. These names were based on common Maven conventions. It is unknown whether IBM maintains an appropriate public Maven repository that contains such “official” artifacts but none could be found when implementing the POC. If it is determined that such a public repo exists, then this step might be avoided.

Once the libraries are added to the local Maven repository as shown above, then the ‘pom.xml’ file for the given project can include the dependencies, using the groupId, artifactId, and version in order to identify the dependency to Maven. Maven should then be able to find the dependency in the local repo.

As a convenience, an optional step would be to install these jars into your local maven repo (or to a corporate Nexus repository.) In which case you could retrieve the dependencies from there and manually installing the JARS to the local maven repo would not be necessary.
Testing the Route
Whether running inside Servicemix or in standlone mode, the technique for triggering the route will be the same. To trigger the route you can perform an HTTP POST at the URL "http://localhost:8888/placeorder", sending in any valid XML as the content of the message. (I use a simple tool like SOAPUI or CURL to send http requests) The route will take the content of the incoming HTTP post and put it on the appropriate queue (as specified in the "camel-context.xml".)

As a convenience for testing, a secondary "server simulator" route has been added (see "MqRoute.java") that will listen on the outgoing queue. The route that is listening on this queue will pick up the message and place a response on the designated response queue (copying messageId into correlationId) per JMS request/reply convention. The primary route is then able to receive the response on expected queue and correlate it with the request.





15 comments:

  1. Very instructive article. Thanks for sharing!

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. com.ibm.msg.client.osgi.wmq_7.0.1.3.jar deployment leads to an error on my servicemix 4.4.1 OSGi container:

    org.osgi.framework.BundleException: R3 imports cannot contain directives.

    ReplyDelete
    Replies
    1. Known issue. This is fixed in version 7.0.1.6.

      Delete
  4. I'm not sure about that error. haven't seen that one. Sorry.If i think of anything i'll let you know.

    ReplyDelete
  5. Thanks. This stuff was helpful. Instead of using the queue names can i look them up from the JNDI name space?

    ReplyDelete
  6. Hi Manglu, I'm not sure if there is direct support for JNDI lookup in Camel but a very quick search shows one possible solution using a "destination name resolver" mentioned by Claus in the following trail http://camel.465427.n5.nabble.com/JMS-queue-JNDI-instead-of-physical-name-td494620.html
    Haven't done it myself though. Good luck! and if i find anything else will let you know.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. Since sharing this post I've discovered there is some extra spring configuration you can add to solution to get more intelligent reuse of resources. We saw some unnecessary opening and closing of connections without the connection caching offered by the following configuration:





    <bean id="wmqTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="wmqCachingConnectionFactory" />
    </bean>

    <!-- A cached connection to wrap the WMQ connection -->
    <bean id="wmqCachingConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory" ref="wmqConnectionFactory" />
    <property name="sessionCacheSize" value="10" />
    </bean>

    <bean id="wmqConnectionFactory" class="com.ibm.mq.jms.MQQueueConnectionFactory">
    <property name="transportType" value="${wmq.transporttype}"/>
    <property name="hostName" value="${wmq.hostname}" />
    <property name="port" value="${wmq.port}" />
    <property name="queueManager" value="${wmq.queuemanager}" />
    <property name="channel" value="${wmq.channel}" />
    <property name="useConnectionPooling" value="${wmq.connectionpooled}" />
    </bean>

    <bean id="wmq" class="org.apache.camel.component.jms.JmsComponent">
    <property name="connectionFactory" ref="wmqCachingConnectionFactory"/>
    <property name="transactionManager" ref="wmqTransactionManager"/>
    <property name="transacted" value="${wmq.transacted}"/>
    <property name="cacheLevelName" value="CACHE_CONSUMER" />
    </bean>

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. hmm. Its a public link to dropbox so should be accessable without any special privs. Perhaps your corporate or firewall blocking?

      Delete
    2. Yep my bad :)
      Let's go to test !!
      Thx

      Delete
  10. I am using Camel 2.13 with SpringDSL. the "to" element takes a URI in which the literal name of the Queue is loaded. However, WebsphereMQ allows queue names to be composite using QMGRname + Qname like "myQM/mqQ". However if I use the syntax with Camel, it fails. It tries to open a queue using the entire string as the queuename. WMQ's JMS docs explicitly define the above syntax as being their standard URI format (which I assumed Camel was passing through to". Anyone have any idea how to get QMGR name accepted as part of sending a message?

    ReplyDelete
  11. Hi,

    I want to integrate the Apache camel jms with the IBM MQ without spring.
    could you please let me know how to achevie.

    ReplyDelete
  12. File Not Found...Source code...Not available.

    ReplyDelete