Hello, my dear readers and welcome back. Today we release number 5 of the series and let me tell you something: I like the number 5. Why?

 

Maybe because I was born on a 5.

Or maybe because in some school grading systems 5 is the maximum grade. Also in the ones where 10 is the maximum grade, 5 means you passed.

Or maybe it is just because I used to listen to Lou Bega and his Mambo No. 5 in my teenage years.

 

Whatever the reason might be, let’s make this post a great one.

 

Last time we talked about performance improvements with special emphasis on caching, audit logging and pipeline management.

 

Today we will finish the performance topics because, well, nobody likes stuff half-done.

 

 

Performance considerations (Cont.)

 

 

Using transformers

 

I think we have established in the last post a strongpoint of transformers: they are useful when doing service caching.

If you want to catch up with this topic go ahead and read the Service Caching section from part 4, available here. (hint: do not skip the comments)

 

Pros for using transformers:

• • easier pipeline management: only the data from the service signature will be passed to the service (not all the pipeline data at the invocation point)
  • service caching: as explained above; however, there are other ways to achieve service caching
  • multiple transformers / MAP step: if you want to call multiple services (or the same service multiple times) in a single step then using transformers is the way to go; in such cases transformers actually improve readability
  • better performance: transformers scope down the data sent to the service; because fewer data gets passed around, the performance is better.

 

Cons for using transformers:

• • diminished readability*: you have to drill in a MAP step to see what service is invoked and open the transformer to see the signature; not all information is available at first glance
  • no implicit mapping: you have to map the inputs and the outputs explicitly

 

*The easy solution is to add a relevant comment to the MAP step so it is clear what the transformer does.

 

 

In regards to performance, I would question everything (as the blog tagline says: Question. Learn. Explore!).

If you have performance problems do some profiling and identify the bottleneck.

Do not replace direct invocations with transformers as a default. Do load tests in both cases. Review and analyze the results. Adjust accordingly.

 

 

As a personal opinion: Would I use transformers every time? Probably not!

I like to use them for what I feel is their purpose: transformation, i.e, invoking mapping services that take some input data and map it (with transformation if necessary) to some output data.

 

 

 

Using the Service Scope property

 

The Service Scope property is used to limit the pipeline data sent to invoked services.

The scope of the pipeline does not have to be reduced to the exact service signature, but it usually is.

 —-> 

 

I am personally rarely using this property. I use it when I cannot use a transformer.

Why I do not like it:

• • diminished readability: it is easy to overlook the Scope property; afterward you find yourself with a scoped down pipeline data without knowing immediately why.
  • development error-prone: do you feel that the Scope property is too close to the Comments. Cuz I do. I have written on several occasions (usually when in a hurry) the comments in the Scope field. Viceversa is possible as well.

 

How to choose between direct service invocations, transformers and using service scope?

Well, review your use case and see which style fits best at that time based on your requirements.

There is no absolute right, nor absolute wrong.

 

 

 

Usage of pub.storage services

 

The services from the folder pub.storage of the WmPublic package should not be used.

These services are used internally by IS to persist data in a short-term store in case of server restarts.

 

Actually, there is a recommendation from SAG in the webMethods Integration Server Built-In Services Reference that goes like this:

SAG does not recommend the usage of this functionality for the following cases:

• • large volumes of data
  • large data records
  • to permanently store data

 

People tend to use this as an all-purpose database that can fit multiple use cases as it is rather flexible.

I know it might be tempting to use this functionality because it is well, right there, but I would advise not to.

You will probably find better options that fit your use case.

 

If you want more details on the locking mechanism that IS uses for this functionality, you can find them in the corresponding chapter of the above mention document.

 

 

 

Debug Log

 

The default debug log service (pub.flow:debugLog) should not be used in order to avoid performance problems.

And performance is just a side of the problem. You cannot develop a professional application and rely on the default logging provided.

 

Logging is an important aspect of any application and has to be designed in such a way as to provide maximum value with a minimal footprint.

You will probably want your logs separated per package, configurable and easily manageable.

Consulting these logs should be easy and should give you fast the information you want.

 

Let’s face it, you are not checking the logs for fun. You probably have a bug to investigate and navigating the logs should not add extra complexity or take too much time.

pub.flow:debugLog will write to the server log and I guess that is the last thing you want.

 

 

 

Timeouts

 

I personally like timeouts. Probably in the same way I like the number 5 🙂

They are always so helpful, letting you grab something from the fridge….

 

All jokes aside, if a service offers a timeout option, I say you use it!

Core reason: if the service is trying to connect to another system and has to wait for the default timeout, it will consume IS resources during this time.

 

So the next time you are invoking a service that has a timeout option (pub.client:ftp, pub.client.ftp:login, pub.client:http, etc.), put a value there!

Pay attention especially to the FTP cases as there the default is to wait forever and forever is a really long time.

 

But what value to set there? Well, it certainly depends on the application landscape and the number of systems you are integrating.

If you are integrating multiple systems in a chained fashion, the timeout settings should look like a funnel.

This means that the systems that are upstream of you have a bigger timeout and the ones downstream have a smaller timeout.

If this is not respected, then, in border scenarios, the upstream system will timeout before you are able to respond.

 

 

 

Combine map steps

 

Having extra or unnecessary MAP steps can affect performance.

I have mentioned some guidelines on how to structure MAP steps here (especially the Bring MAP steps together chapter).

Go with those guidelines and you should be OK.

You will want to find the best compromise between performance and readability.

 

 

 

Loop operations

 

For performance reasons, it is not recommended that you use the services pub.list:appendToDocumentList and pub.list:appendToStringList.

The performance degradation is noticeable when you are trying to append a large number of records (1000+) and, of course, it depends on the resources you allocate to the JVM.

You will fill no pain in case of small lists.

 

The problem comes from the way these services are implemented. There is a lot of array allocation going on there.

To not duplicate information, I will point you to some very good TechCommunity posts on this topic: this one and this one and, also, this one. 🙂

 

So, what to do in this case?

Well, if the size of the list will not increase very much you can continue to use the appendTo… services.

 

If not, you can contemplate the following options:

• • check the PSUtilities package if there is a service that matches your needs
  • do not add records to a list in a loop; use the Output array property of the LOOP construct; use this if the output array has the same size as the input array
  • implement your own Java service that stores the records in the collection of your choice (ArrayList, LinkedList, etc..)

 

In the first link from the TechCommunity forum, Percio describes other options as well and their performance comparison.

Of course, nothing is more relevant in these cases than rigorously testing (both functional and performance).

Create and run your tests and they will speak the truth.

 

 

 

Copy by value vs. Copy by reference

 

So much is written (at least in the Java world) on this topic. The StackOverflow site is filled with questions on when variables are passed by value and when by reference.

So how does this hot topic look for the Integration Server?

 

Definition

 

When the source or target variable is a string, IS will copy the value of the source variable in the target variable => copy by value.

For other variable types (including document types), IS will create a reference to the source variable and set the value of the target variable to this reference => copy by reference.

 

The copy by value definition has an exception: When the source is a string and the target is an object a copy by reference will be performed.

Although not mentioned in the docs (I did not find it), a simple test shows that primitive values (i.e: objects with a defined Java wrapper type) are copied by value as well.

 

Other things to know

 

Copying by reference offers a superior performance compared to the copy by value in terms of memory usage and execution time. That is the main reason this topic appears in the performance guidelines.

 

Copying by reference has a logical side effect but nonetheless a side effect that you should take into consideration. Because the target variable holds a reference to the source variable, subsequent changes to the source variable will affect the target.

 

This can be easily tested with a 3 MAP steps Flow service:

Prerequisite: have 2 documents with the same structure in the pipeline (for easily testing, the document might contain just a string)

 

1. 1. Set the value of the source document content to “value1”
   2. Link the source document to the target document
   3. Modify the value of the source document content to “value2”

 

Aftermath: if you run the service you will see that the target document content is now also “value2”

 

What can you do with this info

 

Well, you cannot change the way the copying is done, that’s for sure.

But you can change the way you define your service signatures.

 

Let’s take an example. Let’s assume you have a service that takes the address details of a client.

How would one define the input of the service? Well, there are (at least) 2 options:

Option 1: Define every field of the address (street, number, flat number, district, etc..) as a String and add it to the service input

Option 2: Define a document type, put the address fields in it and add the document type to the service input

 

The number of the address fields might be big (maybe 10, maybe more). Copying all of these fields by value is less performant than copying a document type by reference.

Therefore, the best option would be Option 2 (for both performance and readability).

 

Now, where do you draw the line between using just strings and having a wrapper document type for them? What is the maximum number of strings that a service should expose?

Other people might have other answers, but mine is: 3.  (You thought it will be 5, didn’t you?)

 

In my opinion, having up to 3 string fields in the service input is acceptable. Anything more and it is better to have a wrapper document type for them.

 

 

 

Unused services

 

Delete the services that are not used.

You will gain 2 benefits from this:

1. 1. No memory will be unnecessarily allocated in the JVM => performance improvement
   2. Other developers will not lose time analyzing why these services still exist => maintenance improvement

 

 

 

Well, there you go. Number 5 is done. We are close to wrapping up this wonderful journey that we had so far.

 

I was planning to include the security part in this post. But after talking and talking and talking some more about performance I found the post growing too big.

Therefore, the security guidelines will be part of the next (and last) post.

 

 

Meet me next time for the Coding Guidelines parting post.

Until then, as always,

 

Happy exploring,
Tury

 

PS: Are you tired of not knowing when my next post is going to come out?

No worries! Hit the Sign Me Up! button and you will receive my posts as soon as they are published.

Do you think my ideas would benefit someone you know? Share it! Spread the word! Sharing is caring 🙂