Message Set to DFDL Automatic Conversion

Daffodils

Recently we’ve started working on migrating stuffs from WMBv7 to ACEv12. The biggest challenge we’ve encountered is so far is, converting the message sets to DFDLs. Please note I’m talking about Custom Wire Formats not Cobol copy books.

I’d say it is one of the most difficult task I’ve encountered so far because the messages sets are all EDIFACT ( IBM 500 ) messages with different initiators, separators and terminators along with various padding characters and justification configuration.

So, thought of finding some automatic solution in Google but no luck. There are 100+ models need to be converted in next couple of months. All I can do so far is, write a java program to read the MXSD files and clean up the unnecessary stuffs. Also, tried to have the group indicators/terminators and delimiters converted into DFDL annotations.

This is not a 100% working solution but definitely helping a lot to quickly have the DFDL created. If you’ve any better solution, please feel free to add in the comments.

import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;

import javax.swing.JFileChooser;
import javax.swing.JOptionPane;
import javax.swing.filechooser.FileSystemView;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.lang3.StringUtils;


public class TDSCleaner {

	/**
	 * @param args
	 * @throws IOException 
	 */
	public static void main(String[] args) throws IOException {
		// TODO Auto-generated method stub
		JFileChooser jfc = new JFileChooser(FileSystemView.getFileSystemView().getHomeDirectory());

        int returnValue = jfc.showOpenDialog(null);

        if (returnValue == JFileChooser.APPROVE_OPTION) {
            File selectedFile = jfc.getSelectedFile();
            System.out.println(selectedFile.getAbsolutePath());
            
            Path path = Paths.get(selectedFile.getAbsolutePath());
            
            List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);
            List<String> newLine = new ArrayList<>();
            /*String rootElm = JOptionPane.showInputDialog("Enter the root element name");
            
            
            if (rootElm != null) {
				rootElm = "       01  "+FilenameUtils.getBaseName(selectedFile.getAbsolutePath())+".";
				newLine.add(rootElm);
			}
            
            String rootElm =  "       01  "+FilenameUtils.getBaseName(selectedFile.getAbsolutePath())+".";
            newLine.add(rootElm);*/
            String delimiter,groupTerminator,groupIndicator;
            delimiter = null;
    		groupTerminator = null;
    		groupIndicator = null;
            
            for (String line : lines) {
				
            	if (!StringUtils.containsAny(line, "annotation","appinfo",
            			"MRMessage","MRMessage","MRComplexType","tdsElemRep"))  {
					
            		if (line.contains("tdsStructRep")) {
            			
            			line = StringUtils.replace(line, "<U+001D>", "%GS;");
            			line = StringUtils.replace(line, "<U+001C>", "%FS;");
            			
            			if (line.contains("groupIndicator")) {
							groupIndicator = "dfdl:initiator="+StringUtils.substringBetween(line, "groupIndicator=", " ");
						}
            			
            			if (line.contains("groupTerminator")) {
            				groupTerminator = "dfdl:terminator="+StringUtils.substringBetween(line, "groupTerminator=", " ");
						}
            			
            			if (line.contains("delimiter")) {
            				delimiter = "dfdl:separator="+StringUtils.substringBetween(line, "delimiter=", " ");
						}
            			
            			continue;
            			
					}
            		
            		if (line.contains("<xsd:sequence")) {
            			
            			if (delimiter != null) {
            				line = StringUtils.replace(line, ">", " ")+delimiter+">";
						}
            			
            			if (groupIndicator != null) {
            				line = StringUtils.replace(line, ">", " ")+groupIndicator+">";
						}
            			
            			if (groupTerminator != null) {
            				line = StringUtils.replace(line, ">", " ")+groupTerminator+">";
						}
            			
            		}
            		
            		
            		delimiter = null;
            		groupTerminator = null;
            		groupIndicator = null;
            		
            		newLine.add(line);
            		System.out.println(line);
				}
            	
			}
            
            if (!lines.isEmpty()) {
            	
            	String cpyFileName = FilenameUtils.getFullPath(selectedFile.getAbsolutePath())
            			+"Modified_"+FilenameUtils.getBaseName(selectedFile.getName())+".xsd";
            	System.out.println(cpyFileName);
            	
            	FileUtils.writeLines(new File(cpyFileName), newLine);
            	
            	JOptionPane.showMessageDialog(null, "File created successfully");
				
			} else {
				JOptionPane.showMessageDialog(null, "Lines are empty check your source file");
			}
        }

	}

}

IIB to ACE

Some of the notes we’ve taken while working on a migration project from IIB to ACE. If you see anything has changed or provided incorrectly, feel free put those in the comments and we’d update this post with your comments.

  1. App Connect Enterprise v11 (first released 2018):
    1. Onpremise + Cloud
    1. IIB + App Connect Professional
    1. Local -> Toolkit ; Cloud -> Designer
  2. Integration Node is optional
  3. Integration Servers can run independently
  4. Configurable services are replaced with Policy files
  5. Local queue manager is required if Collector, Timer and Aggregate nodes are used.
  6. One Web UI can show any number of Integration Servers & Nodes
  7. Not all features available in v10 Web UI are available in ACE V11 but soon they will be available
  8. Flow and Resource Statistics are turned on by default
  9. Some mqsi commands are not available like mqsiformatlog, mqsireadlog, mqsimigratecomponents, mqsideleteconfigurableservice
  10. IIB End of Support is announced as Apr 2022
  11. ACE can run in on-premise and cloud
  12. Use TransformationAdvisor for the migration plan and impact
  13. App Connect Standard is effectively IIB standard with the addition of the Salesforce Request node whereas App Connect Enterprise is the official successor to IIB and includes the ACE V11 software, App Connect Professional V7.5.2 as well as the IBM App Connect on Cloud Service.
  14. A new administrative REST APIv2 for configuring App Connect Enterprise servers
  15. All Runtime node properties can be modified using server.conf.yml file
  16. IBM App Connect professional has 100+ connectors
  17. If there is any project created in older versions like WMB v6/v7 or anything as integration project, those would be deployed into a default Application project.
  18. Integration projects are no longer encouraged. Any attempt to subsequent redeploy independent resources will replace the entire content of the default application of an integration server with the new BAR file’s independent resources.
  19. New REST API testing utility in admin Web UI
  20. Migration
    1. Parallel Migration
      1. Install your ACEv11 environment,
      1. Configure the environment manually from scratch or reuse command scripts from IIBv10
      1. Take a branch in version control and then deploy your BAR files to the new environment
    1. Parallel Migration using extraction to help
      1. Install your ACEv11 environment
      1. Use mqsiextractcomponents to configure the v11 environment (this will help to create policies for you from config services for example) to help ensure you don’t forget configuration from your v10 system. Discard the run directory contents, because instead you are going to deploy items to the new system one by one as the next action
      1. Take a branch in version control and then deploy your BAR files to the new environment.
    1. Big Bang using extraction
      1. Install your ACEv11 environment
      1. Use mqsiextractcomponents to configure the v11 environment (this will help to create policies for you from config services for example) to help ensure you don’t forget configuration from your v10 system. Don’t discard the run directory contents
      1. Restart the integration node.
  21. There is no change in the project types. It is same as v10.
    1. Application (same as v10)
    1. Library (same as v10)
    1. Shared Library (same as v10)

How do I capture the messages on the fly in IIB?

IBM has provided lot of ways to capture the messages flowing through our message flows. In this post, we are going to discuss about one of them and it is also one of the most unnoticed.

IIB allows you to capture the messages by enabling monitoring profile or by writing custom subflows or using trace nodes.

The most frequently used one is subflows and now a days people use monitoring profile too a lot. There are companies which do not have logs at all ( don’t know how they survive).

Have you ever used Record & Replay? If not, just google to get some basic idea on how that works.

This post is also utilizing the same Record & Replay concept but it does on the fly. I do not have any database setup to store the messages permanently. Just going with built-in capacity of the Integration node.

Let me put here the sequence of steps to achieve this

  1. Enable recording
  2. Enable Injection
  3. Retrieve Recorded Message

Let me put here the Java APIs used here

public void enableRecording(ExecutionGroupProxy proxy, boolean status) throws ConfigManagerProxyLoggedException, ConfigManagerProxyPropertyNotInitializedException {
		
		if (status) {
			proxy.setTestRecordMode(AttributeConstants.MODE_ENABLED);
		}else {
			// clear recorded data and reset server to turn off recording and injection
			proxy.clearRecordedTestData();
			proxy.setTestRecordMode(AttributeConstants.MODE_DISABLED);
		}
		
		enableInjection(proxy, status);
	}
	
	public void enableInjection(ExecutionGroupProxy proxy, boolean status) throws ConfigManagerProxyLoggedException, ConfigManagerProxyPropertyNotInitializedException {
		
		if (status) {
			proxy.setInjectionMode(AttributeConstants.MODE_ENABLED);
		}else {
			// clear recorded data and reset server to turn off recording and injection
			proxy.setInjectionMode(AttributeConstants.MODE_DISABLED);
		}
	}
public List<RecordedTestData> retreiveRecordedMessage(MessageFlowProxy mf) throws ConfigManagerProxyPropertyNotInitializedException, ConfigManagerProxyLoggedException {
		
		Properties filterProps = new Properties();
		String name = getApplication(mf).getName();
//		System.out.println(name);
		if (name != null) {
			filterProps.setProperty(Checkpoint.PROPERTY_APPLICATION_NAME,name);
		}else {
			name = getLibraryProxy(mf).getName();
			if (name != null) {
				filterProps.setProperty(Checkpoint.PROPERTY_LIBRARY_NAME,name);
			}
		}
//		System.out.println(mf.getName());
		filterProps.setProperty(Checkpoint.PROPERTY_MESSAGE_FLOW_NAME,mf.getName());
		
//		ExecutionGroupProxy egProxy = mf.getExecutionGroup();
//		List<RecordedTestData> dataList = egProxy.getRecordedTestData(filterProps);
		
		return mf.getExecutionGroup().getRecordedTestData(filterProps);
	}

That’s it. Once you enable recording, the message flow automatically turns into like this

The recorded message is here (I’ve a UI to make this easier to read)

Here is the local environment message

<localEnvironment xmlns:iib="http://com.ibm.iib/lt/1.0" iib:parser="MQROOT" iib:injectable="true">
   <MQTT>
      <Input>
         <Retained iib:valueType="BOOLEAN">FALSE</Retained>
         <QualityOfService iib:valueType="INTEGER">0</QualityOfService>
         <Topic iib:valueType="CHARACTER">IBM/IntegrationBus/default/Monitoring/Hello</Topic>
         <Duplicate iib:valueType="BOOLEAN">FALSE</Duplicate>
      </Input>
   </MQTT>
</localEnvironment>

And here is the payload

<message xmlns:iib="http://com.ibm.iib/lt/1.0" iib:parser="GENERICROOT" iib:injectable="true">
   <Properties iib:parser="GENERICPROPERTYPARSER">
      <MessageSet iib:valueType="CHARACTER" iib:elementType="0x03000000"></MessageSet>
      <MessageType iib:valueType="CHARACTER" iib:elementType="0x03000000"></MessageType>
      <MessageFormat iib:valueType="CHARACTER" iib:elementType="0x03000000"></MessageFormat>
      <Encoding iib:valueType="INTEGER">546</Encoding>
      <CodedCharSetId iib:valueType="INTEGER">5348</CodedCharSetId>
      <Transactional iib:valueType="BOOLEAN">TRUE</Transactional>
      <Persistence iib:valueType="BOOLEAN">FALSE</Persistence>
      <CreationTime iib:valueType="GMTTIMESTAMP">2021-04-13 16:54:58.451</CreationTime>
      <ExpirationTime iib:valueType="INTEGER">-1</ExpirationTime>
      <Priority iib:valueType="INTEGER">0</Priority>
      <ReplyIdentifier iib:valueType="BLOB"></ReplyIdentifier>
      <ReplyProtocol iib:valueType="CHARACTER">UNKNOWN</ReplyProtocol>
      <Topic iib:elementType="0x03000000"/>
      <ContentType iib:valueType="CHARACTER" iib:elementType="0x03000000"></ContentType>
      <IdentitySourceType iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentitySourceType>
      <IdentitySourceToken iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentitySourceToken>
      <IdentitySourcePassword iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentitySourcePassword>
      <IdentitySourceIssuedBy iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentitySourceIssuedBy>
      <IdentityMappedType iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentityMappedType>
      <IdentityMappedToken iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentityMappedToken>
      <IdentityMappedPassword iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentityMappedPassword>
      <IdentityMappedIssuedBy iib:valueType="CHARACTER" iib:elementType="0x03000000"></IdentityMappedIssuedBy>
   </Properties>
   <XMLNSC iib:parser="xmlnsc">
      <wmb:event xmlns:wmb="http://www.ibm.com/xmlns/prod/websphere/messagebroker/6.1.0/monitoring/event">
         <wmb:eventPointData>
            <wmb:eventData>
               <wmb:productVersion iib:valueType="CHARACTER" iib:elementType="0x03000100">100011</wmb:productVersion>
               <wmb:eventSchemaVersion iib:valueType="CHARACTER" iib:elementType="0x03000100">6.1.0.3</wmb:eventSchemaVersion>
               <wmb:eventSourceAddress iib:valueType="CHARACTER" iib:elementType="0x03000100">MQ Input.transaction.Start</wmb:eventSourceAddress>
               <wmb:eventIdentity>
                  <wmb:eventName iib:valueType="CHARACTER" iib:elementType="0x03000100">OrderReceived</wmb:eventName>
               </wmb:eventIdentity>
               <wmb:eventSequence>
                  <wmb:creationTime iib:valueType="CHARACTER" iib:elementType="0x03000100">2020-06-01T02:02:32.402892Z</wmb:creationTime>
                  <wmb:counter iib:valueType="CHARACTER" iib:elementType="0x03000100">1</wmb:counter>
               </wmb:eventSequence>
               <wmb:eventCorrelation>
                  <wmb:localTransactionId iib:valueType="CHARACTER" iib:elementType="0x03000100">f8a1af6d-013b-4420-9c1b-baf23d5e10a4-5</wmb:localTransactionId>
                  <wmb:parentTransactionId iib:valueType="CHARACTER" iib:elementType="0x03000100"></wmb:parentTransactionId>
                  <wmb:globalTransactionId iib:valueType="CHARACTER" iib:elementType="0x03000100"></wmb:globalTransactionId>
               </wmb:eventCorrelation>
            </wmb:eventData>
            <wmb:messageFlowData>
               <wmb:broker>
                  <wmb:name iib:valueType="CHARACTER" iib:elementType="0x03000100">IB10NODE</wmb:name>
                  <wmb:UUID iib:valueType="CHARACTER" iib:elementType="0x03000100">b7d2e6dd-0b36-4728-9015-818133dbeb14</wmb:UUID>
               </wmb:broker>
               <wmb:executionGroup>
                  <wmb:name iib:valueType="CHARACTER" iib:elementType="0x03000100">default</wmb:name>
                  <wmb:UUID iib:valueType="CHARACTER" iib:elementType="0x03000100">61704de3-b47c-4b23-b95f-0ffd934fdcb1</wmb:UUID>
               </wmb:executionGroup>
               <wmb:messageFlow>
                  <wmb:uniqueFlowName iib:valueType="CHARACTER" iib:elementType="0x03000100">IB10NODE.default.TestMon.TestMon</wmb:uniqueFlowName>
                  <wmb:name iib:valueType="CHARACTER" iib:elementType="0x03000100">TestMon</wmb:name>
                  <wmb:UUID iib:valueType="CHARACTER" iib:elementType="0x03000100">429ff9ef-1c8d-48c1-bf42-99ac3bd247fb</wmb:UUID>
                  <wmb:threadId iib:valueType="CHARACTER" iib:elementType="0x03000100">21980</wmb:threadId>
               </wmb:messageFlow>
               <wmb:node>
                  <wmb:nodeLabel iib:valueType="CHARACTER" iib:elementType="0x03000100">MQ Input</wmb:nodeLabel>
                  <wmb:nodeType iib:valueType="CHARACTER" iib:elementType="0x03000100">ComIbmMQInputNode</wmb:nodeType>
                  <wmb:detail iib:valueType="CHARACTER" iib:elementType="0x03000100">IN.Q</wmb:detail>
               </wmb:node>
            </wmb:messageFlowData>
         </wmb:eventPointData>
         <wmb:applicationData xmlns="">
            <wmb:complexContent>
               <wmb:elementName iib:valueType="CHARACTER" iib:elementType="0x03000100">MQMD</wmb:elementName>
               <MQMD>
                  <SourceQueue iib:valueType="CHARACTER">IN.Q</SourceQueue>
                  <Transactional iib:valueType="CHARACTER">true</Transactional>
                  <Encoding iib:valueType="CHARACTER">273</Encoding>
                  <CodedCharSetId iib:valueType="CHARACTER">1208</CodedCharSetId>
                  <Format iib:valueType="CHARACTER">MQSTR   </Format>
                  <Version iib:valueType="CHARACTER">2</Version>
                  <Report iib:valueType="CHARACTER">0</Report>
                  <MsgType iib:valueType="CHARACTER">8</MsgType>
                  <Expiry iib:valueType="CHARACTER">-1</Expiry>
                  <Feedback iib:valueType="CHARACTER">0</Feedback>
                  <Priority iib:valueType="CHARACTER">0</Priority>
                  <Persistence iib:valueType="CHARACTER">0</Persistence>
                  <MsgId iib:valueType="CHARACTER">414d5120494942763130514d475220204064d15e10000105</MsgId>
                  <CorrelId iib:valueType="CHARACTER">000000000000000000000000000000000000000000000000</CorrelId>
                  <BackoutCount iib:valueType="CHARACTER">0</BackoutCount>
                  <ReplyToQ iib:valueType="CHARACTER">                                                </ReplyToQ>
                  <ReplyToQMgr iib:valueType="CHARACTER">IIBv10QMGR                                      </ReplyToQMgr>
                  <UserIdentifier iib:valueType="CHARACTER">vaithu      </UserIdentifier>
                  <AccountingToken iib:valueType="CHARACTER">160105150000002d74cc5dfa721e73431607412b32030000000000000000000b</AccountingToken>
                  <ApplIdentityData iib:valueType="CHARACTER">                                </ApplIdentityData>
                  <PutApplType iib:valueType="CHARACTER">28</PutApplType>
                  <PutApplName iib:valueType="CHARACTER">vaithu</PutApplName>
                  <PutDate iib:valueType="CHARACTER">2020-06-01</PutDate>
                  <PutTime iib:valueType="CHARACTER">02:02:32.480</PutTime>
                  <ApplOriginData iib:valueType="CHARACTER">    </ApplOriginData>
                  <GroupId iib:valueType="CHARACTER">000000000000000000000000000000000000000000000000</GroupId>
                  <MsgSeqNumber iib:valueType="CHARACTER">1</MsgSeqNumber>
                  <Offset iib:valueType="CHARACTER">0</Offset>
                  <MsgFlags iib:valueType="CHARACTER">0</MsgFlags>
                  <OriginalLength iib:valueType="CHARACTER">-1</OriginalLength>
               </MQMD>
            </wmb:complexContent>
         </wmb:applicationData>
         <wmb:bitstreamData>
            <wmb:bitstream iib:valueType="CHARACTER">TUQgIAIAAAAAAAAACAAAAP////8AAAAAEQEAALgEAABNUVNUUiAgIAAAAAAAAAAAQU1RIElJQnYxMFFNR1IgIEBk0V4QAAEFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIElJQnYxMFFNR1IgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN4cDI2NyAgICAgIBYBBRUAAAAtdMxd+nIec0MWB0ErMgMAAAAAAAAAAAALICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAcAAAAZ21haWwuc3Jpbml2YXNhLkFwcGxpY2F0aW9uIDIwMjAwNjAxMDIwMjMyNDggICAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA/////zxPcmRlcnM+CiAgIDxPcmRlcj4KICAgICAgPEN1c3RvbWVySUQ+R1JFQUw8L0N1c3RvbWVySUQ+CiAgICAgIDxFbXBsb3llZUlEPjY8L0VtcGxveWVlSUQ+CiAgICAgIDxPcmRlckRhdGU+MTk5Ny0wNS0wNlQwMDowMDowMDwvT3JkZXJEYXRlPgogICAgICA8UmVxdWlyZWREYXRlPjE5OTctMDUtMjBUMDA6MDA6MDA8L1JlcXVpcmVkRGF0ZT4KICAgICAgPFNoaXBJbmZvIFNoaXBwZWREYXRlPSIxOTk3LTA1LTA5VDAwOjAwOjAwIj4KICAgICAgICAgPFNoaXBWaWE+MjwvU2hpcFZpYT4KICAgICAgICAgPEZyZWlnaHQ+My4zNTwvRnJlaWdodD4KICAgICAgICAgPFNoaXBOYW1lPkdyZWF0IExha2VzIEZvb2QgTWFya2V0PC9TaGlwTmFtZT4KICAgICAgICAgPFNoaXBBZGRyZXNzPjI3MzIgQmFrZXIgQmx2ZC48L1NoaXBBZGRyZXNzPgogICAgICAgICA8U2hpcENpdHk+RXVnZW5lPC9TaGlwQ2l0eT4KICAgICAgICAgPFNoaXBSZWdpb24+T1I8L1NoaXBSZWdpb24+CiAgICAgICAgIDxTaGlwUG9zdGFsQ29kZT45NzQwMzwvU2hpcFBvc3RhbENvZGU+CiAgICAgICAgIDxTaGlwQ291bnRyeT5VU0E8L1NoaXBDb3VudHJ5PgogICAgICA8L1NoaXBJbmZvPgogICA8L09yZGVyPgo8L09yZGVycz4=<wmb:encoding iib:valueType="CHARACTER" iib:elementType="0x03000100">base64Binary</wmb:encoding>
            </wmb:bitstream>
         </wmb:bitstreamData>
      </wmb:event>
   </XMLNSC>
</message>

By this way, if you’ve a need to immediately see what is flowing through your messages, simply implement this solution and see the messages on the fly. Once done, clear up the recordings.

If you’re interested to know more about this solution, feel free to contact us at support@vaithu.com. WhatsApp +1 6123058684.

Setup IIB Logging in 10 minutes

Are you using log4j or any custom subflow to capture the IN and OUT messages but looking for a simple solution which can

  • captures all messages, not just IN & OUT but also exception message
  • captures the header values, environment values etc
  • does not need you to add subflows or log4j node
  • works with simple configuration like enabling monitoring profile and that’s it
  • automatically creates the log based on Node/ServerName/FlowName

A simple flow like this does all the magic for you.

IIB Transaction Monitor Flow
MQTT properties
UDP to specify Log4j config file location
2021-04-11 19:05:14.889 [Thread-30] INFO  - ----------------------------------------Transaction Starts---------------------------------------------------------
2021-04-11 19:05:14.889 [Thread-30] INFO  - Event Number :1
2021-04-11 19:05:14.889 [Thread-30] INFO  - Data Element Name :MQMD
2021-04-11 19:05:14.889 [Thread-30] INFO  - ***********************************************************************
2021-04-11 19:05:14.889 [Thread-30] INFO  - SourceQueue : IN.Q
2021-04-11 19:05:14.889 [Thread-30] INFO  - Transactional : true
2021-04-11 19:05:14.889 [Thread-30] INFO  - Encoding : 273
2021-04-11 19:05:14.889 [Thread-30] INFO  - CodedCharSetId : 1208
2021-04-11 19:05:14.889 [Thread-30] INFO  - Format : MQSTR   
2021-04-11 19:05:14.889 [Thread-30] INFO  - Version : 2
2021-04-11 19:05:14.905 [Thread-30] INFO  - Report : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - MsgType : 8
2021-04-11 19:05:14.905 [Thread-30] INFO  - Expiry : -1
2021-04-11 19:05:14.905 [Thread-30] INFO  - Feedback : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - Priority : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - Persistence : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - MsgId : 414d5120494942763130514d475220204064d15e10000105
2021-04-11 19:05:14.905 [Thread-30] INFO  - CorrelId : 000000000000000000000000000000000000000000000000
2021-04-11 19:05:14.905 [Thread-30] INFO  - BackoutCount : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - ReplyToQ :                                                 
2021-04-11 19:05:14.905 [Thread-30] INFO  - ReplyToQMgr : IIBv10QMGR                                      
2021-04-11 19:05:14.905 [Thread-30] INFO  - UserIdentifier : vaithu      
2021-04-11 19:05:14.905 [Thread-30] INFO  - AccountingToken : 160105150000002d74cc5dfa721e73431607412b32030000000000000000000b
2021-04-11 19:05:14.905 [Thread-30] INFO  - ApplIdentityData :                                 
2021-04-11 19:05:14.905 [Thread-30] INFO  - PutApplType : 28
2021-04-11 19:05:14.905 [Thread-30] INFO  - PutApplName : vaithu
2021-04-11 19:05:14.905 [Thread-30] INFO  - PutDate : 2020-06-01
2021-04-11 19:05:14.905 [Thread-30] INFO  - PutTime : 02:02:32.480
2021-04-11 19:05:14.905 [Thread-30] INFO  - ApplOriginData :     
2021-04-11 19:05:14.905 [Thread-30] INFO  - GroupId : 000000000000000000000000000000000000000000000000
2021-04-11 19:05:14.905 [Thread-30] INFO  - MsgSeqNumber : 1
2021-04-11 19:05:14.905 [Thread-30] INFO  - Offset : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - MsgFlags : 0
2021-04-11 19:05:14.905 [Thread-30] INFO  - OriginalLength : -1
2021-04-11 19:05:14.905 [Thread-30] INFO  - ***********************************************************************
2021-04-11 19:05:14.905 [Thread-30] INFO  - OrderReceived Payload :
<Orders>
   <Order>
      <CustomerID>GREAL</CustomerID>
      <EmployeeID>6</EmployeeID>
      <OrderDate>1997-05-06T00:00:00</OrderDate>
      <RequiredDate>1997-05-20T00:00:00</RequiredDate>
      <ShipInfo ShippedDate="1997-05-09T00:00:00">
         <ShipVia>2</ShipVia>
         <Freight>3.35</Freight>
         <ShipName>Great Lakes Food Market</ShipName>
         <ShipAddress>2732 Baker Blvd.</ShipAddress>
         <ShipCity>Eugene</ShipCity>
         <ShipRegion>OR</ShipRegion>
         <ShipPostalCode>97403</ShipPostalCode>
         <ShipCountry>USA</ShipCountry>
      </ShipInfo>
   </Order>
</Orders>
2021-04-11 19:05:14.905 [Thread-30] INFO  - ----------------------------------------Transaction Ends---------------------------------------------------------

There is much more. For more info, contact us at support@vaithu.com/ WhatsApp +1 6123058684

How to read excel files using Java Compute Node

Sometimes, we need to read excel files and there is no native connector in IBM Integration Bus. So, the obvious solution is to read them using Java. The most commonly used library is Apache POI.

Here I’ve written a simple program which reads the sheets ( assuming it has a header and multiple rows) and attach them to Environment tree. The java code is

public class ReadExcel_JCN extends MbJavaComputeNode {

	public void evaluate(MbMessageAssembly inAssembly) throws MbException {
		MbOutputTerminal out = getOutputTerminal("out");

		MbMessage inMessage = inAssembly.getMessage();
		MbMessageAssembly outAssembly = null;
		try {
			// create new message as a copy of the input
			MbMessage outMessage = new MbMessage(inMessage);
			outAssembly = new MbMessageAssembly(inAssembly, outMessage);
			// ----------------------------------------------------------
			// Add user code below
			
			MbElement environment = inAssembly.getGlobalEnvironment().getRootElement();
			
			File excelFiles = new File((String) getUserDefinedAttribute("ExcelDir"));
			
			Workbook workbook = null;
			
			DataFormatter dataFormatter = new DataFormatter();
			
			List<String> headerNames = null;
			String cellValue = null;
			MbElement cellElm = null;
			
			for (File excel : excelFiles.listFiles()) {
				
				workbook = WorkbookFactory.create(excel); 
				
				for (Sheet sheet : workbook) {
					
						MbElement sheetElm = environment.createElementAsLastChild(MbElement.TYPE_NAME, sheet.getSheetName(), null);
						headerNames = new ArrayList<>();
						for (Row row : sheet) {
							
							if (row.getRowNum() != 0) {
								cellElm = sheetElm.createElementAsLastChild(MbElement.TYPE_NAME, "rows", null);
							}
							for (Cell cell : row) {
								
								cellValue = dataFormatter.formatCellValue(cell);
								if (row.getRowNum() == 0) {
									headerNames.add(cellValue);
								} else {
									
									cellElm.createElementAsLastChild(MbElement.TYPE_NAME_VALUE, headerNames.get(cell.getColumnIndex()), cellValue);
								}
							}
						}
						
						if (sheetElm.getLastChild() == null) {
							sheetElm.detach();
						}
						
				}
				
				workbook.close();
			}
			

			// End of user code
			// ----------------------------------------------------------
		} catch (MbException e) {
			// Re-throw to allow Broker handling of MbException
			throw e;
		} catch (RuntimeException e) {
			// Re-throw to allow Broker handling of RuntimeException
			throw e;
		} catch (Exception e) {
			// Consider replacing Exception with type(s) thrown by user code
			// Example handling ensures all exceptions are re-thrown to be handled in the flow
			throw new MbUserException(this, "evaluate()", "", "", e.toString(),
					null);
		}
		// The following should only be changed
		// if not propagating message to the 'out' terminal
		out.propagate(outAssembly);

	}

}

If you’re looking for any support in IIB, please contact us support@vaithu.com.

About the Author

He has worked for various clients and developed more than 800+ message flows. He is an expert in ESQL, Java Compute Node and DFDL. If you’ve any complex code to be written in ESQL, feel free to contact him at +1 6123058684.

XML/JSON Beautifier

There are lot of online tools available in the internet for this but they say, do not put sensitive information as they’re cached on the server side. So, obviously it is not a safe place to beautify. And not all companies allow admin access to developers to install plugins for this. What’s the solution for this? Let us develop something on our own and add what ever we want.

Here is a solution developed in-house keeping in mind all IIB/ACE developers.

You can not only beautify xml/json messages but also do the following

  • Blob to string
  • String to blob
  • Base64 Encode/Decode
  • XML to JSON
  • JSON to XML
  • XPath evaluation
  • XSLT Transformation
  • Validate XML against XSD
  • XML Decoding
  • Change themes

Above all this is developed as web-application and no external dependencies. Just deploy the war file to a web-server and start using it. By default it runs on port 8080 and so you can try http://hostname:8080

Here are some screenprints to please your eyes.

XML PrettyPrint
XML to JSON conversion
Blog to String

You can download this for free for personal use. For commercial use, please contact us at support@vaithu.com.

We also do full-stack development. If you’re looking for modernizing your UI, please contact us.

Vaadin themes

By default, Vaadin use Lumo theme but it also provides multiple themes which you can customize as you need. Vaadin provides an editor here https://demo.vaadin.com/lumo-editor/ which allows you to create your theme or customize the default ones. Let us see how to use them.

  • Go to https://demo.vaadin.com/lumo-editor/
  • In the right hand side, you will see the settings to modify. Modify them as you need and finally it will show you a HTML page like
  • In Vaadin v14, importing html is not supported so you’ve to copy the lines within <custom-style><style>...</style></custom-style> and copy it to a file under styles directory like
  • Now, import this file into your Java class like
@CssImport("./styles/my-lumo-theme.css")

Reload your project and you will the difference. I modified the theme to use Material theme and here is the page after importing them into my Java class

If you need DARK theme, just two lines would make it and they are

@Theme(value = Lumo.class,variant = Lumo.DARK)
public class MainView extends AppLayout {

Note these line should go into the Main/Parent class. Let us see how the dark theme looks

Let me put here my custom-theme HTML code for reference

html {
  --lumo-size-xl: 3rem;
  --lumo-size-l: 2.5rem;
  --lumo-size-m: 2rem;
  --lumo-size-s: 1.75rem;
  --lumo-size-xs: 1.5rem;
  --lumo-space-xl: 1.75rem;
  --lumo-space-l: 1.125rem;
  --lumo-space-m: 0.5rem;
  --lumo-space-s: 0.25rem;
  --lumo-space-xs: 0.125rem;
  --lumo-font-size: 1rem;
  --lumo-font-size-xxxl: 1.75rem;
  --lumo-font-size-xxl: 1.375rem;
  --lumo-font-size-xl: 1.125rem;
  --lumo-font-size-l: 1rem;
  --lumo-font-size-m: 0.875rem;
  --lumo-font-size-s: 0.8125rem;
  --lumo-font-size-xs: 0.75rem;
  --lumo-font-size-xxs: 0.6875rem;
  --lumo-shade-5pct: rgba(26, 26, 26, 0.05);
  --lumo-shade-10pct: rgba(26, 26, 26, 0.1);
  --lumo-shade-20pct: rgba(26, 26, 26, 0.2);
  --lumo-shade-30pct: rgba(26, 26, 26, 0.3);
  --lumo-shade-40pct: rgba(26, 26, 26, 0.4);
  --lumo-shade-50pct: rgba(26, 26, 26, 0.5);
  --lumo-shade-60pct: rgba(26, 26, 26, 0.6);
  --lumo-shade-70pct: rgba(26, 26, 26, 0.7);
  --lumo-shade-80pct: rgba(26, 26, 26, 0.8);
  --lumo-shade-90pct: rgba(26, 26, 26, 0.9);
  --lumo-primary-text-color: rgb(235, 89, 5);
  --lumo-primary-color-50pct: rgba(235, 89, 5, 0.5);
  --lumo-primary-color-10pct: rgba(235, 89, 5, 0.1);
  --lumo-error-text-color: rgb(231, 24, 24);
  --lumo-error-color-50pct: rgba(231, 24, 24, 0.5);
  --lumo-error-color-10pct: rgba(231, 24, 24, 0.1);
  --lumo-success-text-color: rgb(62, 229, 170);
  --lumo-success-color-50pct: rgba(62, 229, 170, 0.5);
  --lumo-success-color-10pct: rgba(62, 229, 170, 0.1);
  --lumo-shade: hsl(0, 0%, 10%);
  --lumo-primary-color: hsl(22, 96%, 47%);
  --lumo-error-color: hsl(0, 81%, 50%);
  --lumo-success-color: hsl(159, 76%, 57%);
  --lumo-success-contrast-color: hsl(159, 29%, 10%);
}

[theme~="dark"] {
}

We also started working on Full stack development using Vaadin. If you’ve any projects to develop in Vaadin, please contact us at support@vaithu.com

Cryptography in IIB

Not often but sometimes, there is a need to encrypt and decrypt messages flowing through IBM Integration Bus and there are some working solutions available in the market to achieve the same.

Since these algorithms are not available by default in ESQL, the next immediate solution is Java. So, if you are comfortable with Java Compute Node and know how to pick the Key from multiple locations like the below , then it is scalable and works like the same way as built-in nodes

  • User Defined Property
  • LocalEnvironment
  • Environment
  • Any other valid location in the Message Assembly

Here is a solution which uses AES algorithm to encrypt and decrypt the messages. All you need is 16bytes KEY.

Test flow

The KEY can be hard coded or its location can be $LocalEnvironment/Destination/Encrypt/Key or any other path which can be defined here

Now, this makes flexible that the KEY can be inside the incoming message or from LocalEnvironment or from Environment tree too.

Let us see how the encrypted message looks like after passing a simple XML file like below

Test message
Debugger
After Encryption Node
Encrypted Message
Decrypted Message in OUT Queue

If you are looking for the same solution or would like to know our capabilities in IIB, please contact us at support@vaithu.com/WhatsApp +1 6123058684.

We’ve rich experience in IIB and executed multiple projects since 2013. So, we assure that you get your solution on time in low cost without compromising on quality and efficiency.

IBM Integration API – What shall I get from this?

  • Are you a serious IIB developer/admin?
  • Are you switching with multiple applications/windows back & forth?
  • Are you not a fan of commands ?

Yes. Yes. Yes. We were also looking for some sort of way to get most of our stuff done with some simple clicks/buttons. So, we started looking the ways to achieve it and finally found this IBM Integration API which contains everything you & we needed.

Let us put here some high level things that you can get from this API

  • List all
    • messages flows/applications/static/shared libraries
    • UDPs
    • Queues used
    • Security Identities used
    • DSNs
    • Bar files
    • Integration Server properties
    • Configurable Services
    • Activity Log
    • Administrative Log
    • SubFlows
  • Stop/Start
    • Message Flow
    • Application
    • Integration Server
  • Deploy/Undeploy bar files
  • Update/Modify/Create Configurable service
  • Update UDPs
  • Set Workload Management Properties

Most of the above features might be available from Web UI but it does contain have a way to search. Also, it does not allow to copy one bar file from one environment to another. But when you know the above stuffs to do programmatically, you can do what ever you need.

import com.ibm.broker.config.proxy.*;
public class DeployBAR {

  public static void main(String[] args) {
    BrokerConnectionParameters bcp =
       new MQBrokerConnectionParameters("localhost", 2414, "IB9QMGR");
    try {
      BrokerProxy b = BrokerProxy.getInstance(bcp);
      ExecutionGroupProxy eg = b.getExecutionGroupByName("default");
      DeployResult dr = eg.deploy("MyBAR.bar", true, 30000);
      System.out.println("Result = "+dr.getCompletionCode());
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

You can copy everything from one server and put it in another server. You can cross verify what is in multiple environments and much more can be done easily.

If you want to know more about this, we take training on these periodically. Join one of our training and control the Node as you wish. For more details, contact us at support@vaithu.com/WhatsApp +1 6123058684.

MQEdit-Web-based RFHUtil

Are you using RFHUtil but looking for some alternatives for

  • Editing the message on the fly
  • PrettyPrint XML/JSON messages
  • List All queue details with human understandable format
  • Purge selective messages
  • Move messages from one queue manager to another
  • Move selective messages
  • Create Queue
  • Queue Properties
  • Search Queues
  • Read message from two different queue from the same window
  • Get description of the MQ Return Code
  • Trigger N messages at a time for load testing
  • And many more

Yes. We were also looking for the same and so built a tool to help you improve your productivity and lessen the development/testing time. This tool has most of the sought features available in RFHUtil. Here is the video demonstrating the features and how to use this tool.

This is tool is a web-based tool and so works on all platforms. All you need is to run the below command and open a browser,type localhost:8080

java -jar mqedit-1.0-SNAPSHOT.jar
Main Page

You can put all your queue manager configuration details like in a text file and put them in your user’s directory

IIBv10QMGR;LOCALHOST;1414;SYSTEM.DEF.SVRCONN

To purchase this software, please contact us at support@vaithu.com.

For immediate response, you can WhatsApp us at +1 6123058684.