Friday, August 12, 2016

Writing integration tests for Apache CXF REST API that supports XML and JSON

Hey folks,

This comes up as a part of my ongoing GSoC project. The task as described in an earlier post was simple. To convert a SOAP service to REST. Yet the individual issues that comes up along the way sums up to writing those separate blog posts about them.

So this is the last phase of the project that I'm currently carrying out, that is to write integration tests for the now available REST web service. Now have the following assumptions before we go through the procedure

1) The service is implemented with Apache CXF and JAX-RS
2) It can consume and produce both XML and JSON based requests and responses
3) Service is up and running and accessible at https://localhost:5000/rest
4) Service has a one service method called getResponse that accepts XML or JSON requests (in a predefined way) and outputs according to the headers given.


Issue now is that although we can easily write unit tests for any given piece of code, writing unit tests for the web service api is just pointless. We need to be able to test the service online it self whenever we make a change to see that everything works good (the whole point of testing). So what we need are integration tests.
Moreover since my request/response architecture supports XML and JSON I should be able to send corresponding requests and check if I am getting the desired responses. So let's dig into the steps.


1) Add all dependencies

We need 2 things for our code to work
i) Apache CXF
ii) TestNG (Or if you preper you can use JUnit as well)

Since my project is all maven based, I will show how to add the required dependencies using maven. If you use any other dependency management software, feel free to find the libraries appropriately.

2) Prepare the directory structure

Before going into code we need to structure our codes so that the test classes doesn't mess up with our source code. Also keep in mind that since we have to test XML/JSON requests/responses we need to store these code files somewhere and read them as well. Hardcoding them inside the test classes would be an ugly option. So we have 3 things to keep apart from each other
i) Source code
ii) Test classes
iii) XML/JSON codes

Here's how I managed to seperate those

There are 2 main directories under the src directory
i) main : contains source code and it's resources
ii) test : contains test classes and it's resources

Since XML/JSON codes are part of the test classes, they are put within the resources directory under test directory.

3) So now we are ready to start with the code. First make a single class named TestService which extend Assert class under the test directory. (Chose the package names appropriately)

4) First we write a simple method to check if the service is online, if it's offline we cannot do any integration test on it.

If your service is secure (https), then probably the java WebClient will need to verify the certificates before any request can be made. If such verification cannot be done, an exception would occur. The easiest way to overcome is to make up your own keystore and specify the path to the keystore or truststore from the code. You can easily google that step, anyhow I added the following to the constructor of TestService class to mention the location of the trustStore to the WebClient. (This corresponds only according to my own configuration)

5) Everything's ready to go but we need to make some mock XML/JSON requests and responses first.



Keep those files in the xml directory under the resources.
Assume that when we send the contents of request-getResponse-1.xml over to the getResponse service method, we receive the response with the content in response-getResponse-1.xml.

6) Next we need a method to read the content from those 2 files. Here's a neat method that does the job, note that the string replacement at the end makes sure that things work correctly when we have to do assertEqual at the desired and actual responses at the end.

7) Let's write our first test method now
You can change / add / remove the headers as necessary, and keep in mind that this suits to a service method that reads the body of the request and processes it. If you play around a bit you can find the way to send any type of request to a service.

8) Here's the full code when all put together, and now you test the code. And TA-DAA
All test cases passed!

If you get any test case failures check the outputs of response and webResponse to see what's going wrong. So happy testing!


Friday, August 5, 2016

[GSoC 2016] REST implementation for WSO2 IS Entitlement Service

Hi all,

This post is to document the process followed, decisions taken and implementations done in implementing the REST service for WSO2 Identity Server's Entitlement Service. This was done as a GSoC project. If you are nto familiar with the WSO2 Identity Server, please first refer to my previous post which would answer most your questions.


IS                      -  Identity Server
XACML           -  eXtensible Access Control Markup Language
REST                -  Representational state transfer
JSON                -  JavaScript Object Notation

Requirements of the service

  1. Allow RESTful access to 5 service methods (Already implemented in SOAP service)
  2. Allow XML XACML requests to be processed and responses to be given
  3. Allow XML web requests to be processed and responses to be given
  4. Allow JSON XACML requests to be processed and responses to be given
  5. Allow JSON web requests to be processed and responses to be given
  6. Exceptions should be handled globally and necessary responses should be given in a centralized manner
  7. Comply fully with the REST profile for XACML 3.0
  8. Provide swagger definitions for extra methods that are not specified in the REST profile

Implementation Decisions

  1. REST Framework - Apache CXF
       Apache CXF is a framework that allows easy deploying of REST Services. The key reasons for using Apache CXF for this implementation are

  1. Apache CXF already used with other REST Services (ex: SCIM inbound provisioning service). So libraries are already bundled with IS
  2. JAXB provides inherent support for marshaling and unmarshaling Java Beans to and from XML
  3. Additional libraries can be used to easily support JSON
  4. Exception mappers allow centralized exception handling

  1. Jackson for JSON support

Jackson is a JAX-RS provider for JSON, and it's already used within IS.

  1. Implementing JSON profile for XACML 3.0 defined here
  1. Using exception mapper to centralize the exception handling of the service and providing customized error messages from the service.

Final Implementation Design

Figure 1

Rough Class Diagram on critical components

Figure 2


1) Branching from SCIM inbound provisioning service implementation

Since the the above project is already implemented and functioning within IS, a copy of that was used as a starting point for the REST implementation of Entitlement Service. The following classes and packages were derived as it is from the SCIM implementation

i) auth
ii) impl

i) AuthenticationFilter
ii) AbstractResource
iii) ClearThreadLocalInterpretor

2) Removing SCIM related dependencies

All SCIM related dependencies were removed from the project. Since Authentication is not centralized within IS (and will be in future), the following authenticators were hardcoded.

i) BasicAuth
ii) OAuth

3) Restructure project paths and doing necessary changes to cxf-servlet.xml and web.xml

4) Adding JSON marshaling and unmarshaling to and from Java Bean support using Jackon

Currently Jackson 2.5.4 is added as an external library to the project. And the Jackson Json Provider was registered as a provider for the service. For which the following lines we added in cxf-servlet.xml

5) Define a new DecisionResource class which would handle the service endpoint requests

DecisionResource will extend AbstractResource. Accordingly the necessary service methods were then added (Refer Figure 2).

6) For methods getDecision and getDecisionByAttributes Request/Response converters were needed

By default Balana engine does not posses the capabilities to evaluate JSON XACML requests nor provide JSON XACML response. Hence 2 new classes were added to provide support.

i) JSONRequestParser : Converts JSON XACML request to Balana ResponseCtx object
ii) JSONResponseWriter : Converts Balanda ResponseCtx object to JSON XACML Response

7) For the rest of the web service requests, necessary Request/Reponse models were defined. Such defined models were

i) getHome : HomeResponseModel

ii) getDecisionByAttributtes / getBooleanDecision: DecisionRequestModel

iii) getEntitledAttributes :EntitledAttributesRequestModel, EntitledAttributesResponseModel

iv) getAllEntitlements : AllEntitlementRequestModel, AllEntitlementResponseModel

8) These models allow better Request/Response structures from the service

9) Custom Exception Mapper (EntitlementExceptionMapper) was added

This act as an intermediate to all exceptions occurred during the service method executions as well as JAX binding executions. Exception will not be handled inside the service methods, but rather will be thrown out to be caught be the exception mapper. Hence through which the exception handling procedure can be centralized. Custom error codes and messages were introduced. (Will be stated in a latter section)

10) Finally, the webapp is created to a war file and deployed to the IS webapps directory

It can be accessible by https://localhost:9443/wso2-entitlement/entitlement/Decision?_wadl

11) Implementing Swagger for populating service methods

Swagger is a framework that easily provide a globally understandable representation to the RESTful APIs. Since our REST service contained methods that were not defined by the XACML REST profile, swagger need to be used to populate the methods definitions of those additional methods and of the overall API itself. Swagger product a json/yaml file that nicely describes the service methods, input parameters and return values to any client that consumes it.

12) Writing integration tests for the service using testNg and Jetty 

Once all done, the integration tests were written. Used jetty module that comes with Apache CXF to fire up a web client and send and retrieve responses from the service. The full procedure is at


The complete code can be found at

Github Commits

Public JIRA Ticket


Thursday, July 14, 2016

phpmyadmin not found on LAMP install in Ubuntu 16.04+

Hey folks,

Just another quick one. Recently I installed Ubuntu 16.04 and Ubuntu 16.10 both uses the shinny new php7.0. And some newer version of apache as well. And both times when I installed phpmyadmin, and go to or
it gives me a simple, annoying 404!

Well there's a lot of solutions for this in Q&A threads, but for me the only working solution is that, you need to Include the phpmyadmin configuration at the end of the apache2 configuration, so that apache knows that phpmyadmin exists. Here's how

1) Open up apache configurations

sudo nano /etc/apache2/apache2.conf

2) Add the following to the very end

Include /etc/phpmyadmin/apache.conf

3) Save and exist and then restart the apache server

sudo service apache2 restart

4) Browse


Saturday, May 21, 2016

Tomcat error when deploying web app : addFilter, addServlet NoSuchMethodException

Hey peeps,

Just a quick update on something i found today. I was hacking things with a tomcat server (wso2 IS). My need was to duplicate a webapp and make some changes as I like. But when I changed those 'stuff' in the webapp source I had to specifically mention some dependency versions (build tool was maven) rather than obtaining them from the parent pom.

And when I generated the war and deployed it in the server I got the following error


It turns out that the reason for this error has nothing to do with the actual content of the web.xml
But a dependency mismatch
That is some libraries in your WEB-INF\lib directory of the war file is conflicting with some libraries already loaded. So the fix was for me to delete the lib directory completely and deploy which worked!

But according to some stackoverflow answers, deleting just tomcat-catalina.jar and/or all tomcat-*.jars will fix it

Happy Deploying! :D

Monday, April 11, 2016

Laravel 5 : Getting started for collaborative development

Hey folks,

Laravel, for those who aren't familiar is a php framework that is widely gaining popularity these days. This came into my attention when I had to implement an HR system in PHP. Usually I would have gone with Symfony, which I know has a larger support base. But the issue is that Symfony has a huge learning curve and I have to work with a team. Laravel on the hand has a less difficult learning procedure and it can be used to start implementation rapidly.

But Laravel it self is really confusing sometimes. Mostly if you are not familiar to tools like composer and git (for collaborative development). So here's a quick tutorial in which I mention steps I took to setup Laravel and a git repo for development.


1) Git installed 
You can download git for any platform here,
Make sure if in windows, that you add Git to shell commands when they ask to, so that you can access git from command prompt at any place.

2) LAMP / WAMP installed
Linux Apache MySQL PHP (LAMP)
Windows Apache MySQL PHP (WAMP)
You can do this per your liking

3) phpMyAdmin installed
If you are a windows user using popular WAMP like XAMPP, you will have phpMyAdmin already installed. LAMP users need to add it manually.

4) Composer installed
Composer is a dependency management tool for Laravel. Much like Maven  (if you are familiar) it will take a list of necessary dependencies and download these to the computer which removes the need of uploading all the 3rd party libraries everywhere.
Download and install composer according to the instructions given here

Oookay now we are officially ready to start off with the things.


1) Make an online repository to host your source code
For collaborative development, first thing you need it to host your code somewhere. Since we are using git for source control, we need a host that supports git.
Github : will let you have public repos for free
Bitbucket : will let you have private repos for free (plus team integration with limits)

I went with bitbucket since our code is SUPER SECRET!
So goto, make a new repo. let's say "AwesomeHR"
Now ask all your team members to sign up with bitbucket as well and in your repo settings, add all of the team members and give them write / read access.

2) Clone your repo to the local drive
It's time to get your repo to your hard drive. Goto your webroot directory (usually www, htdocs, etc depends according to your server) and clone the repo using this command,

git clone

3) Installing Laravel

There are 2 ways of installing Laravel,
One is the official guide in their installation page using composer and laravel installer

Second one is the hard code way in which we download the Laravel master branch from their github repository and extract it where we want. I had more success with this method in several platforms than using the installer (which has varying steps depending on the platform)

i. First download the Laravel master repo as a zip file from their Github repo
You can also use the following link for a direct download

ii. Next extract it to your newly cloned directory so that when you go into your AwesomeHR folder, it shown all the php files (not laravel-master folder, in that case move everything inside laravel-master to outside)

Now comes a very critical understanding. Laravel has 1000 different 3rd party libraries in use. Those libraries as well as the Laravel core source are NOT included in the zip file you just downloaded. why?

  • Because doing such would make it a very big file
  • We can't always have the latest version of the library if you pre bundle it with the Laravel source, any bug fixes won't reach once downloaded.

So here's how composer come into play. Composer's job is to download all those other libraries into the directory named "vendor" in your Laravel installation and setup them so that it works smoothly.
But keep in mind that you SHOULD NOT copy this vendor directory when you are sharing your code. In other words, the vendor directory be default has a .gitignore file which prevents ANY file inside that folder being uploaded to your remote repo.
Any collaborator using your code has to follow the steps of cloning your repo and then using composer to download the additional libraries.

iv. After that loong discussion, you use the following command while being inside of the extracted folder (AwesomeHR)

composer install

It will take some time for this to complete. Be patient my friend.

To check if all went good, goto http://localhost/AwesomeHR/public/ and you see this

Ta-Daaa you have a fresh Laravel installation now!

4) Configuring Laravel
Still the shit ain't working. Obviously you need to follow an installation procedure.
You need a database for Laravel to store all the data to start with.
Goto phpMyAdmin and make a new database called "laravel"
I will assume our database configuration is as follows

Database name : laravel
Database user : root
Database pass : root
Database host :

To make Laravel use these settings, open the .env.example file in your laravel root directory using your IDE or something like SublimeText. (Notepad won't work)
Edit the lines 5 - 8 to match your configuration


Officially, you can start development with Laravel. You can try a few simple migration codes and make sure the installation is okay. Before you forget, you need to upload the code to the remote repo in bitbucket. Use the following command,

git add .
git commit -m "initial commit"
git push origin master

5) Collaborative development
To use the above code in a team mate's disk, follow these steps

i) Clone the bitbucket repo

git clone

ii) Use composer to install the vendor files

composer install

iii) Do configurations as in step 4

Happy Laravel-ing folks!

Friday, March 25, 2016

WSO2 Identity Server : Understanding XACML 3.0 policies, implementing them in IS and testing with SoapUI

Hi folks,

This is a weird way to resume a blog that was silent for a long time. But I guess this will do. The reason for this post is mainly GSoC 16' which is currently happening. And in selecting a project for it, I came up with WSO2 which has middleware based products. And with time constraints, I was forced to study a system overnight and thought that documenting what I found would be beneficial for those who try to do the same later.

So the question here is what exactly WSO2 Identity Server does?

WSO2 Identity Server (IS) is an identity management system based on open standards. It's a middleware that would connect all dispersed authentication services you have into one manageable system. It has many features that makes authentication and authorization of users for your resources a simple and efficient task. Since the scope of this post is not explaining what IS is, I recommend you to follow the following links to have a better idea. (Dear future GSoC applicant, it is very important that you go through those seemingly boring but highly important links :) )

Get to know the high level artichecture

A very simple getting started guide

The blog that would contains answers to many questions on IS

Now since you have some idea about IS, let's dig into XACML specification, its uses and how IS has implemented it.
XACML is set of standards by OASIS that are related to access control. In general, it specify a way or a language through which applications should communicate to request authentication and authorization data from eachother. Since the specification in discuss is XACML 3.0, from the following link you can observe how a XACML document would look like.

So why XACML is important to Identity Server is:

  1. It is used as the request/response language standard
  2. It is used as the access control policy standard

With that we get the question what exactly is a policy? and how is it all related?

Okay, let's take some theory in.
A policy (in this context) is a set of rules that would define to IS who can access what. Basically its like a paper from a rule book, Whenever someone asks if a user "A" can perform action "WRITE" to the resource "R", it will go through those policy pages and see if there's any policy defining those requirements.
Let's see the above explanation in a more technical way.

Assume there's a resource link. Whoever accessing the resource now would meet a blocker before the actual resource end point. This blocker is known as the Policy Enforcement Point (PEP). The PEP is like the guardian of the front gate that would catch the request and it wants to know if the requests comes from a client that actually has access to the resource. For that it sends the request data to a place called Policy Decision Point (PDP). PDP is like the rule book reader that matches requests data against any rules in the rule book. The rule book is known as the Policy Administration Point (PAP), it is where all the policies are written.
Additionally there's a component known as Policy Information Point (PIP). This is a place where the PDP can ask for more information of a user that's requesting the authorization.

Now the three parts PIP, PDP, PAP are working together to figure out if the request can pass through the blocker (PEP) and reach the resource. It can do so only if they given the 'Permit' flag back as response to the PEP. If you look at the following diagram things may get more clear.

[Well just looking won't help, you have to really try to understand]

XACML architecture implementation in IS

I think that actually clears-up the theory part. So we can now get into real action :D


Suppose there's a resource located at
And this service should only be accessed by admins (Or any user that is given the user role as 'admin')

1) Configure IS to have a policy to support the above rule
2) Validate the policy using the IS inbuilt Try-it feature
3) Use SoapUI to simulate a PEP that is sending a SOAP request to check if a potential user is can access the given resource

1) WSO2 Identity Server downloaded / build from source
2) SoapUI (free version would do)


1) Adding a policy to IS
Download and Start the Identity Server (Please read above links again if you have idea how to do that)
Login as username ; admin, password: admin
Goto Policy Administration -> Add New Entitlement
Note that entitlement is adding a new policy or a rule to the rulebook
Select Write Policy in XML from the list

2) Writing the policy
Now that there's a windows where we can write our policy in XACML format. We can also use their inbuilt editors, but for the scope of this we'll stick to the text editor.
Use the following policy code. paste it there and click Save Policy

A above policy defines a set of rules,
i) The policy has the id 'auth_admin'
ii) It matches the attribute resource to
iii) It matches the action to exactly be 'read'
iv) It matches the group of the user accessing to exactly be 'admin'

3) Testing the policy
Once you add the policy and save you will see it in the Policy Administration list. From the list there select Try
A new UI would open and select 'Create Request using Editor'
What we going to do here is to create a sample request that might be coming from an actual PEP and test it against our PDP to see if it's working.
Use the folloing xml code, which is a XACML request.

Now click Test Evaluate and you would see that the XACML response appear with the decision tag saying 'permit'. ta-daa! We have successfully made a working XACML policy in IS. Now try changing things in the request, like group name, and resource location and observe the response.

Finally we have to actually put the policy to working mode. This is done by publishing it.
Go back to Policy Administration and publish auth_admin

4) Simulating the PEP using SoapUI
Although we evaluated the Policy using the internet tools, we need to verify that it works from outside too. On the other hand, it is important to observe the response from IS to a PEP using SOAP.
We first need to prepare the IS to show the WSDL definition for SoapUI to extract all endpoints.

Open <product-home>/repository/conf/carbon.xml

Restart IS and navigate to, https://localhost:9443/services/EntitlementService?wsdl
You can see all the endpoints

Open SoapUI and add new project, then add wsdl and put the above url. Once entered you would see that SoapUI automatically identify the endpoints and the service names.
Goto getDecision and right click -&gt; add request
Use the following XMLto have a request

Now we have to put a XACML request INSIDE the SOAP request. In order for the 2 xmls not to fight with each other, we put the XACML request inside CDATA tag, which would prevent it from considering as XML, but it would still be a part of the request text.

Before sending the request we need to add authentication of course.
So add auth from SoapUI and of course put username:password as admin:admin
And hit SEND button. [That green triangular button]

And you would see a SOAP response, in which a CDATA section has the actual XACML response saying 'permit'

So this has been a very long post, and i hope to upload and fix issues here and there in this. But hope this helps.


Saturday, August 1, 2015

Explained: Finding the Longest Increasing Subsequence using Segment Tree

Hey guys,

Well this is just random but i found out that for my most disappointment that the tutorials on the above topic arent really useful for the novice. Most tutorials just give the procedure in english or the code but not both. Hence I'm trying to write something human-understandable in the simplest english i know :D


Find the length of the longest increasing subsequence in a given array of integers
For the context of the tutorial we assume all integers are positive hence 0 will be our minimum.
Read more about in the wikipedia article

So the most obvious method is the complete search of the array which would take O(n^2) time. This is highly inefficient for a data set of very large n. Thus an alternative method should be used. Using the segment tree we can find the answer in just O(nlogn) time which is acceptable.

For those who aren't familiar with Segment Trees please refer to wikipedia or this Youtube tutorial. I actually found the Youtube tutorial quite informative and simple to understand.

So back to the solution, what we gonna do is to build segment tree and find the length of the longest subsequence. Please note that this won't give you the actual sequence but just the length of it. But you can implement the finding sequence part easily.


1. Sort the original array but keep the original index of each element.
 original array  - the input array with the elements in their original order
 sorted array  -  an array of objects that contain information on the value of the element as well as its position in the original array. This array is sorted according to the value of the element.

2. Initialize the segment tree with all 0

2. We loop each element of the sorted array and in each iteration
      i. Take the original index of the element
      ii. Find the value corresponding to (0,index) from the segment tree. This would be 0 in initial iterations.
     iii. This value would be the length of longest subsequence upto that point, then add 1 to this value since obviously the current element has a value greater than all previous elements (since the array is sorted) and this extend the previous biggest sequence by 1.
     iv. Use this value to build the segment tree further. (update the tree)


Let's look at the code and then I'll do some drawing and stuff to explain the process VISUALLY *_*

So it looks all okay till it comes to the loop. What eaxtly happend when the buildTree and findMax methods are called? How the tree is built?

Let's take our array, sorted array and the initial segment tree

Let's go through our loops

So finally the root of our tree gives the answer :D