Utilizing VO Standards and Protocols: Server Side

These are notes to accompany this part of the Summerschool program. This section is intended to be on Sept. 14 3:30 - 5:30 pm. This is a play along section - you are encouraged to look at the code and do the examples as we go along. Please ask questions at any time. Think more tutorial less lecture.

For this to work you need the Software listed on the the software page.

We shall look at :

All paths below are from the java level of the summerschool java directory that $VO_HOME on UNIX or %VO_HOME% on windows.

For examples where is used the equivalent on UNIX is $VAR.

Simple Cone Shell

Tomcat

First lets make sure our tomcat works. We should check that CATALINA_HOME is properly set.

> dir "%CATALINA_HOME%"
or on UNIX
>  ls $CATALINA_HOME

OK if that returns a listing we should be able to use our script to start tomcat:

> starttomcat

This may will fail if tomcat is already running on the port. The port is set in the server.xml file in the conf directory under CATALINA_HOME. The default is 8080 - but you may change it.

Java Server Pages (JSP)

JSP is a very quick way to develop highly functional web pages. Effectively each page becomes a java class (a servlet) which is compiled once by the server. When coding in JSP we may intermix java code with actual output such as XML or HTML to create our pages. The .NET framework also now provides a facility like this which are its ASPX pages.

Lets look at the cone service. Effectively it is a web application which takes 3 parameters: RA,DEC,SR. It then needs to do a lookup using this information and provide the answer in a VOTABLE. So a call to a cone service should look like

   conservice?RA=108&DEC=1.5&SR=0.5

So an easy way to do this would be to make a simple JSP page and have it pull these parameters from the request. Such a page is provided in the distribution package in dev/conserver/coneShell.jsp.

We will walk through the code before we deploy it. This is a nice easy JSP so for deployment all we need to do is copy it to a web directory. The default root of the tomcat installation is in the webapps/ROOT directory so we need to :

> copy coneShell.jsp "%WEB_DEPLOY%"\ROOT
or on UNIX
> cp coneShell.jsp $WEB_DEPLOY/ROOT

Then this will be available at

http://localhost:8080/coneShell.jsp?SR=1&DEC=180&RA=1

But there is no data there yet.

ConeSearch Server from VOTable- WAR Files, Junit, Ant

The simple JSP does not call any other java code. Nor is there any data pulled in to it. Still it is a good starting point and we know it works. A modified version of this is in the web subdirectory called cone.jsp. Normally we may connect this to a database and JDBC is good for that.

Here for simplification we will use an a VOTABLE for our data. This is not efficient nor probably a good way to do this but lets assume the table is small and we can load it all in memory. OK then we need some method which can do a ConeSearch on the loaded data. Thats fairly easy - we need a routine which calculates the angular distance between two points then (again not efficient) we may scan the table and see if the input point is closer than SR to the point in the catalog.

Still in the dev/conserver you will find a src\sumsch\VOTCone.java. This does all of the above. We shall take a look.

The code is in a src directory mainly through convention. You could put it in any directory you like. The readvotable example just has the code in the root directory.

Lets also have a look at how we might use that in the cone.jsp page.

Junit

Regression testing is essential for any code. Web applications such as we are building debugging is even more difficult. A good level of unit testing will make things go much more smoothly. Junit is a framework for building test cases and running them automatically. The notion of the unit test is to try each small part of the program individually. This may influence the way we write our code. Some testing philosophies suggest writing the tests before the code (eXtreme programming). Some tools however will build entire test suites for existing code (e.g. Jbuilder with Junit plugging - Axis ant also will attempt to build tests for generated stubs).

Whatever the philosophy having the test is good and with ant(more below) and Junit its easy. A simple set of test for VOT cone are in src\sumsch\test\TestVOTCone.java. It conventional to call the Test for a class Test and the individual method test test.

Web Archives (WAR files)

So now we have a JSP file a Java Class and a library (VOTCone uses ivoa-0.3.jar) which we need to deploy. We know we can copy the JSP file to the root and it will work. If we look at the ROOT dir in tomcat we will see a WEB-INF directory. In that there is a classes directory and a lib directory. This is where we can put individual classes (in classes) and jar files (in lib). However you can see how this would get messy with more than one application to deploy.

In fact the structure of ROOT is a web application structure defined for all java application servers. You may create any directory with this structure under webapps of any application server and your application should work. Normally you do not create a directory though you create a jar file which has this structure and contains your entire website. This is give the extension .war. When a war file is copied to the webapps folder the application server unpacks it creating a directory with the same name as the file. Hence if we create a cone.war and drop it in webapps we will have a new cone application. Furthermore app servers check for updates and manage hot deployment - so each time we put a new war file in webapps it will be automatically unpacked to replace the old one. Of course this also allows one to have multiple version of an app or website on the same server.

Building the war file would of course be tedious by hand but most IDEs have a method for doing this and Ant also has a target for it. Time to look at ant.

Ant

Ant is an very useful platform independent build system for java. Its not exactly make nor is it exactly a scripting language but it covers a little of both. Everything in ant is a task and tasks may depend on other tasks. All tasks are defined in xml files. The default name for the file is build.xml. When you type ant it looks for file called build.xml to execute tasks. The xml file is structured as a project which contains properties and targets. The targets generally contain the tasks. Any target may be specified on the the command line after ant is invoked so to get the cone classes compiled we could do

> ant compile 

NOTE for the tests you may need to chnage the dataPath in build.xml

    <target name="test" depends="compile">
   <junit fork="yes">
      <jvmarg value="-DdataPath=/nvoss/data/"/>

If we do not specify a target for ant then the default target is used. This is specified in the project declaration

   <project name="cone build file" default="dist" base dir=".">

In the properties we set up useful variables. One special variable to notice is web_deploy - this shows how to pull something from the environment :

   <property environment="env"/>
   <property name="web_deploy" value="${envy.WEB_DEPLOY}"/>

We shall look at the tasks in build.xml: javac,junit, war, copy.

Now we may run that test we saw earlier - or any other test which may exist or have been added to the project:

> ant test 
   :
test:
    [junit] Testsuite: nvo.test.TestVOTCone
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.625 sec

Of course we may now also deploy this simply with :

> ant deploy 

We have built cone.war and in it there is the cone.jsp. So this will appear as a new webapp and may be invoked with :

http://localhost:8080/cone/cone.jsp?RA=180&DEC=88&SR=10

System Properties

The correct way to pass setup parms to a WebApplicaiton? is using the Context. We can pass the path to this in the conf/web.xml these lines set the path to the abell.xml - you may need to modify it

<env-entry>
  <env-entry-name>dataPath</env-entry-name> 
  <env-entry-type>java.lang.String</env-entry-type> 
  <env-entry-value>/nvoss/data</env-entry-value> 
 </env-entry>

The fast and nasty way is to modify the catalina.bat or catalina.sh and pass the dataPath property do this by adding a line

set JAVA_OPTS=-DdataPath=/nvoss/data/

This also means your code may be put in a stand alone app.

Simple Image Access Server

The SIA is similar to cone it takes some parameters from the request and returns a VOTable. An SIA server in fact does not return any images only links to images so in a sense it is easier than a ConeSearch. This subdirectory of dev contains a simple image access server. Again it is in the form of a shell

UseBean

A Java Bean is an object of a class which adheres to the Bean interface standards in java. For our purposes this just means that it has 'get' and 'set' methods to access attributes and these are of the form getAttribute. In the ConeSearch we used the request directly to get data from the request and then managed these in the page. JSP provides a special construct where by we may specify a Java Bean and it will use the 'set' methods of the bean to determine the attributes it should look for in the request. Such beans are very useful as we can put them in the session and pass them around between pages on the server. Its also useful for refilling forms when there is an error or outputting the PARAM elements specified for the SIAP.

src\nvo\SIAbean.java is a bean which we could use for the SIA parameters. Notice it has setSIZE, setPOS and setFORMAT It also has an isOk method which we may use to check the validity of the input. This allows us to take the input validation away form the web page and put it in a neat class - this of course could be used in an app elsewhere.

Once we have the bean using it is easy we just need two lines in our jsp, the first declares the bean for use, the second tells JSP to grab all it can from the request:

<jsp:useBean id="sia" scope="page" class="SIAbean" />
<jsp:setProperty name="sia" property="*" />

jsp:include

Otherwise our JSP is not much different to the one for the cone search. We no longer need to build up the error string as that is now done by the bean so it s little tidier. One other small addition is tho show how part of a JSP file may be included from another file (Server side include basically) . The VOTABLE header is pulled in with :

 
<jsp:include page="top.xml"/>

As with the previous example there is a test class and a war is built by ant do it all with :

 
ant deploy

and we can see it on

http://localhost:8080/siap/siapShell.jsp

SOAP WebServices

At this point we have already heard a little about SOAP/XML and WebServices. The above are not WebServices since they do not contain a machine readable description which allows a set of client stubs to be constructed. Generally when we speak of WebServices we mean a services accepting and transmitting SOAP messages and defined with a WSDL.

AXIS

We could write JSP or servlet which behaved as WebServices but the general approach is to use a Framework like :

• SunOne
• The Mind Electric GLUE
• BEA WebLogic
• IBM WebSphere
• Microsoft .NET
• Apache AXIS.

These frameworks serialize and de-serialize objects from the native language to/from XML. They also contain some tool which converts from WSDL to native classes.

We have selected AXIS for today although some of the services you have seen are from .NET.

At this point we need to install axis. For some reason they don not provide a WAR file rather a directory still it is easy to install- we simply need to copy the contents (axis) of the webapps directory(e.g. %VO_HOME%\axis-1_2beta\webapps) from AXIS(in the java directory)into the webapps directory of our webserver (e.g. %CATALINA_HOME%\webapps).

We will also need to restart tomcat since it only picks up on new WAR files not necessarily new directories.

bouncetomcat

And check http://localhost:8080/axis/happyaxis.jsp

A few additional jars have been added to axis which you otherwise have to download separately.

The WebService Server

So this may seem complex but actually making a web service is in many ways easier than the type if pages we have already constructed. This is because the framework does all the hard work. Lets look at a WebService called Nums in simplewebservice\server\src\sumsch\Nums.java

In AXIS one may take a file like this and rename it .jws place it in the axis folder and it works. That however is a little like our trivial JSP example above - not very useful when we get even slightly more complex.

So we need to make a deployment descriptor - this is in the server directory called deploy.wsdd. To deploy this we need to put our classes into access. The easy way to do this is make a jar (ant can do that ) and put in in the WEB-INF/lib of axis. We also need to run the adminclient with the descriptor.

Axis comes with some special tasked for ant so we can have a deploy target such as :

    <target name="deploy" depends="dist">
   <move file="${dist}/nums.jar" todir="${axis_deploy}/WEB-INF/lib"/>
   <axis-admin
       failonerror="true"
       xmlfile="deploy.wsdd"
   />
    </target>

So in the server directory we may do

ant deploy

Axis may not find the jar when it is deploy in which case you may need to start and stop tomcat do this with :

bouncetomcat

The WebService Client

This is where the real beauty of SOAP Services lie. You have seen some already but lets look at this one too. Lets just look at NumsClient.java in the client directory.

The needed interface to the serer is all generated by AXIS (another ant task def):

    <target name="axis" depends="">
   <axis-wsdl2java output="${generated}" url="http://localhost:8080/axis/services/Nums?wsdl"/>
    </target>

Hence we may simply type

ant test

To show what you need to do to have a JSP page call such a service a deploy target has been added to the build.xml. This builds a war file with a webform to call the service. The jsps are in the web directory. View this at:

http://localhost:8080/numsc/numform.jsp

WebService Server from WSDL

We can also generate a server from a WSDL not just a client. The ConeSearch for SDSS for example is a WebService at http://skyserver.sdss.org/vo/dr2cone/SDSSConeSearch.asmx.

The conewebservice directory contains a build.xml which will generate the server side java classes for this WSDL. Just type

ant axis

Now look at generated\edu\jhu\pha\skyservice\ConeSearchWSSoapImpl.java. Notice now we are not dealing with XML or web pages just pure java objects.

Deployment is also built in the ant file so :

ant deploy

should work.

Basic SkyNode Server

(Many thanks to Ramon Williamson NCSA for the Java SkyNode)

The OpenSkyQuery web site contains a download of the .NET SkyNode. This requires SQLServer and a special set of database routines to do the XMATCH and spatial searches. This is a Full SkyNode.

It was considered useful to also define a Basic SkyNode which could accept queries in ADQL. In the dev directory is an implementation of this in Java.

The initial outline for this was generated from the WSDL contained in the specification of the SkyNode. In this case the WSDL provides a way to define an standard interface.

Client.

There is a client in the client sub dir of the basic skynode. It is possible to pass this a different end point to try other SkyNodes by passing the endpoint url on the command line.

Again most of this client is generated the only code that was written is in the SkyNodeClient? .java.

Again with Ant we generate all the stubs and compile everything with :

ant compile

You may try this against SDSS or ADIL to see it works with .NET and Java.

ant testADIL
or
ant testSDSS

Server

The SkyNode specification is generally to expose a relational database through a standard interface. So we need a database. We could use MySQL or SQL Server (which is now free). In the later case though we should then use the .NET SkyNode also - this is the fastest and best way to make a Full SkyNode. For now lets concentrate on MySql.

Create Database

For most DBs we will need to load data. To build a full skynode in SQLServer there is extensive documentation on SkyServer.org.

The first thing we will need is a CSV file. If you have one great if not we can make one from the Abell catalogue using OpenSkyQuery.

NOTE: You have Abell.csv in your nvoss/data directory so you may skip this.

Simply import it as Abell using the import page

Then ask for it back as a CSV with the following query on the query screen

select t.* from mydata:abell t

Click save results on the right panel lets put it in data called Abell.csv.

END NOTE

Now we need a table. MySql does not allow DEC as a column name ! Run MySql and create the table in the default test database (or equivalent schema if using another table)

 
mysql> use test  \g
mysql> create table Abell  ( unique_id int, name varchar(50), ra double, declination double , _count int, bmtype varchar(50), redshift real, rich varchar(50), dist varchar(50), vmag real, x real, y real, z real  ) \g

Now we import the data with :

mysqlimport  --verbose=TRUE --replace=TRUE  --fields-terminated-by=, test %VO_HOME%\data\Abell.csv 
or on UNIX 
mysqlimport --local --verbose=TRUE --replace=TRUE --fields-terminated-by=, test $VO_HOME/data/Abell.csv 

So We have a database, Normally we would tune this and create indexes etc. but we wont do that here.

To make this available to JDBC we have to grant some privs to our user:

>mysql  -u root

mysql> GRANT ALL PRIVILEGES ON test.* to skynode@127.0.0.1 identified by 'skynode';
:
:
mysql> FLUSH PRIVILEGES;

The Java SkyNode.

If we are using this MySql database then the build.xml is already setup for that. If not we need to modify build.xml - near the top the config file is named. A bunch of config files are provided by Ramon for different DBMS.

In the config file we ned to specify a full path to the Style Sheets this is important even if following this example especially if you are on unix look in the config file for a line like

    <XSLTPath>c:\nvoss\dev\basicskynode</XSLTPath> 

If you don't have the abell schema then the config file will need to be modified a little more. This config file also contains the username and password for conecting to the DBMS as well as the driver class name. The Driver for the dbms needs to be available in AXIS WEBINF/lib - mysql.jar has already been added if you used the standard setup. In any case lets look at Configure.xml_mysql_abell.

To compile and test this all we need to do is:

ant dist

To deploy it if the test worked all we need to do is:

ant deploy

We may check our deployment at http://localhost:8080/axis/servlet/AxisServlet. This may require a stop and start of tomcat to see it show up. Once we see it listed we should be able to run our client which we looked at earlier. In the Client directory ..

ant testLocal

Registration

Once you have a node you need to register it. The only registry where this is currently possible is the STScI/JHU registry at http://nvo.stsci.edu/voregistry/index.aspx . In the case of a basic node the Compliance field needs to have the word BASIC in it.

The OpenSkyQuery portal uses the registry to locate nodes. You will notice the ADIL node on the portal - this is a Java node. You may also notice it does not have the + and - symbols since it can not be included in a XMATCH.