HydroloGIS, 10 years of proud Italian Scientific Geographic Open Source and Environmental Engineering

I still remember it as if it was yesterday. We were sitting in the lab of Unitn (the Faculty of Evironmental Engineering of the University of Trento, where we were doing research for professor Rigon) and we were thinking about what to do. Money was running out in universities and the figure of external research staff was being obsoleted. Italy was going to face one big economical crisis.

I remember Silvia saying that she wanted to do a couple of years of research and then head off to do missionary work in Africa.

I remember not knowing exactly what I wanted to do. I had just found what I loved to do: developing open source GIS. It was not just developing anything. It had to be science and it had to be open. I remember dreaming about being able to develop any kind of tool for anyone and for free... right where big big companies were charging in an unfair way. I remember thinking that research should be for everyone out there. And I remember professor Rigon being of the exact same idea. 

At that time the slogan was: All in a aGIS!

We had figured that when it comes to environmental sciences, the glue to keep them all together could only be the GIS. With Riccardo we always joked about taking over ESRI at some point. Well, that one didn't work out that well. :-)

The first logo of HydroloGIS.

The second logo of HydroloGIS.

The current logo of HydroloGIS.

I told Silvia I thought she would be wasted for “normal” missionary work and that she should push her talents hard to do what many others would not be able to: create useful tools and then bring them developing countries together with training for it. And somehow I convinced her on something she didn't need to :-). 

I remember her telling me that most probably in about 5 years from that moment she would leave HydroloGIS anyways to follow her path in development countries.
I remember thinking: "It's ok even like that!". Being able to share a sparkling new experience with a pal that didn’t care a fig about making money exactly as I did, with the only wish to make science tools to release open source for the people… well, that was something that would not be easy to find again. We had to give it a try.

At that time she was following a master about project planning in Padova. One evening I was in the lab and she called me from Padova: “Are you serious about trying to create a company? There is this startup competition here... we could participate… everything needs to be delivered by midnight of today”. We chatted around for a short while, she gave me the link to the competition and then her cellphone battery died with her being spending the evening out (and not being easy to charge phones back then). I had no idea where to contact her again. I didn’t even know her home address at the time. I had to look up her parent’s phone in the paper phonebook (remember?) and call them to get Silvia's data for the contest. The lord meant it good to us, since during the following hours of struggle to write down the business plan idea and completing the competition bureaucracy also a name for the new company had to be chosen. Man, am I happy that HydroloGIS came that natural!

HydroloGIS at "Startcup"

Funny thing is that we wanted to try a startup competition to have confirmation about how good our idea was: Bring innovative open source technologies from the university to the professional world in order to make the world a better place. Sounds just about right, doesn't it?!?!?
Well, we didn’t even pass the first round!!! I remember project N.1 being a portable DNA analyzer device (I wonder where that company is today). We were so angry! How could they not understand!!!! So we decided to prove them wrong. And that is how HydroloGIS got born.

The really empty office of HydroloGIS at day 0. Those who now it now, would not recognize it.

We found shelter at the TIS Innovation Park Southtirol. At that time it was called BIC, the Business Innovation Center, which was an incubator and highly supportive for ideas both based on Environment and Open Source.

HydroloGIS, John Preston from the Jamaican reserach center ICENS and the University of Trento developed together JGrass, the open source GIS that wanted to be a userfriendly graphical interface for GRASS and specialize on hydro-geomorphological processing based on professor Rigon's past decade of research. 

The first JGrass logo, based on GRASS' logo

At that time the only development sponsorship to JGrass came from the MIGG course that we held with Rigon's team each year at the University of Trento.

And this was the first team preparing the course right after the MIGG2005 edition:

Andrea Cozzini, Erica Ghesla, Silvano Pisoni and HydroloGIS

Here the first JGrass versions have been developed.

One thing we have always been bad at was looking for work. At the very beginning we tried some commercial campaigns to sell HydroloGIS products around the region. Eventually we figured that it didn’t work. We have never been able to explain what we do properly and we simply didn’t have the skills to sell.

Luckily for us at the start there were some rather badly paid, but extremely interesting, projects with the university. Then later some jobs for government agencies came in. Interesting enough, we never looked for a job since then. The jobs just came to hunt us. And I mean it when I say "hunt". People started to come to us when they didn’t know where to head to solve their problems, usually after having tried for a while, hence burning both avaliable time and budget. And we took all those jobs, once again, bad paid and sometimes so demanding, that we had to work for free a long time to finish the job. Those were (hard but) great times, in which we grew incredibly from a professional point of view. But those were also hard times. And I exactly remember getting at the end of the year not having enough money to pay taxes. After a year spent working on average 15 hours a day!

But still we stuck to the plan: do everything the open source. It had to work!

Eventually one year passed and we were still in business:

 

and not in the worst shape (even if I was gaining quite som weight, eventually passing 100Kg :-) ):

In the years that followed (initially mainly on behalf of the University) we taught at several different courses and master courses, gave lessons at the university and to professionals, participated at many Conferences, EGUs and FOSS4G's, went to code sprints, developed GIS, made Engineering work, entered the OpenMI Steering Committee, the uDig Steering Committee, developed and coordinated JGrass, BeeGIS, the Nettools, the JGrasstools, Geopaparazzi and eventually won different innovation competitions.

Silvia winning the first price for innovative women entrepreneur.

GRASS code sprint in Prague.

JGrass went through many different periods. Being rejected from the GRASS community we had tried to develop it on our own, but it had overwhelmed us. we were not a pure software company.

A short history of JGrass until 2010, as presented at FOSS4G Sydney.

So in 2007 one big move had been done, the migration in what I was convinced was the best java open source desktop GIS (I still am convinced it has that potential): uDig

The first splashscreen of JGrass as uDig extension.

HydroloGIS, beyond others, teaching to high school teachers in Cape Town after the FOSS4G.

Talking about JGrass at Foss4G Sydney.

and having great fun at the same conference, always wearing the colors :-)

always ready to attend to important meetings to shout out our thoughts.

With the begin of my PhD at the University of Urbino a new open source door opened up. The mobile field mapping.

The BeeGIS extensions for uDig were developed in that context, which gained quite some interest. Being uDig based it worked only on full size pcs, or better tablets. Back then the big deal where Ultralight tablets, that had some complete-yet-downscaled operating system and were able to give enough juice to make uDig run on them. 

The splashscreen of the BeeGIS release for which ARPA Piemonte sponsored a nice part of development both for BeeGIS and uDig.

Digital field mapping as PhD student in Urbino.

Today I still have to laugh about that and I know some of you don't even remember. But tablets were heavy back then:

but everything is possible if you only want it:

In 2007 the first Geopaparazzi was born. And nothing has been as it was before :-)

We were finally able to have a tool that we would always have with us and with that we would be able to map in extreme situations:

Android made the BeeGIS project die out, because we simple stopped using it to develop the Geopaparazzi project, which, funny enough, turned out to be our most supported in the GFOSS world. There is even a facebook group for it now :-)

The years passed and many interesting things happened, as well as quite difficult things that hit hard on us. We tried to work tightly with other companies, which sometimes didn't work out well. We also tried to have employees, which did really work badly and we figured that would be something for when we are old :-). 

As I am writing this up now, overwhelmed by happiness and proud of what we achieved, all the bad things just vanish and I could go on telling you stories about our life, but I think it is really enough now. :-) It is really more material for a pub than a technical blog. But you got the point: I love the way HydroloGIS turned out, with its ups and downs.

The sole fact that few years ago I have been able to go back again to do music actively with my Alpentraum Orchestra, is a great indicator of having achieved what I wanted to: a job I love to work on every minute of the day and (some degree of) complete freedom in the organisation of time, which makes family and music possible.

Over time HydroloGIS made sense, even if they told us that it would not make sense to sell things for free and that we were crazy to do that in our niche of work... well, we were never able to properly explain it around here, we simply knew it was the right way. Eventually we figured that it was the somehow old Italian method that was wrong, not us. 

Our open source products and community involvement started to bring great jobs outside Italy and even overseas. While we still were not able to get work within a 100Km radius around our office (not one single job), we started to work in Germany, Finland, England, the United Arab Emirates, New York.

And obviously there were the projects in development countries, so important to Silvia :-). We were able to bring training sessions and our tools to Rwanda, Ethiopia and Kongo and work on water management systems. The sparkling new agreement with the GISMAP will help us to spread Geopaparazzi in the eastern side of the globe.

Silvia teaching Geopaparazzi in Arba Minch.

Teaching uDig and GIS at the University of Arba Minch.

Alright, I think I have kept you here for enough time now and my story is getting very confused. I know I missed many important things and people. Don't feel bad about it, this is a simple writedown, in a couple of hours, while browsing through old pictures and sensations.

One last thing I want to do is to thank those that believed in us even when there was no money and no future (well, that would be for about the first 6 years :-) ). Thanks to our families and partners, that supported us by any possible means. Thanks for being there, thanks for supporting without asking questions. Just thank you!

(In this picture, taken on HydroloGIS' 8th birthday, my mother and my sister Michela are missing. My mother was taking the picture and Michela lives too far away to be able to attend to every party we throw :-). I want them to know that they are part of it.)

Thanks also to Silvia, with who I had the luck to share these last 10 years. It has been sometimes difficult, she is soooooo hard-headed, but then I know I am too. :-) Thanks for letting me try all those things, even if no evident money would come out of it and we were starving :-) Thanks for being the other half of HydroloGIS.

HydroloGIS is Tony and Silli.

Last but not least, thanks to all of you that have sympathized with our concept of leading such a small company the meritocratic and open source way. We know you are out there and we have always been honoured when you told us how much you respect and like us.

Happy 10th birthday HydroloGIS, while my eyes are getting slightly wet...

How hard is it to add an OpenMI enabled module to JGrass?

Ask the hydrologic lady, she just made her first two of them :)

JGrass now ships with OpenMI 1.4

Since I will attend the next OpenMI meeting, I am in fact playing a lot with it in this time.

Today I migrated JGrass to the new 1.4 openMi SDK.

For those that didn't find the new java version of openmi, here it goes from svn.
This is the official OATC version:

http://openmi.svn.sourceforge.net/viewvc/openmi/branches/OpenMI-1.4.1-de
v/MyOpenSource/Alterra/

Just for those that do not know. For JGrass we use a modified version that we make support the OGC SimpleFeature model and currently the JGrass rasters. We are working to get the OGC standard also for rasters in it.


Find infos about OpenMI here.

OpenMI "exposed" in JGrass

I finally took the time to understand how an extension point is created. Well, now a first step is done. I wanted to be able to serve JGrass and GRASS command through rcp extention points and be able to configure in that one also its GUI and menu entries, but for now what works is the creation of the command and its registration to the console engine. The gui stuff will also be there at some point. But for now let's have a look at the new way to create OpenMI enabled JGrass modules.

1) assuming you created a plugin , open your plugin.xml with the Manifest-Editor


2) push the add button and in the upper filter part, type *openmi. Select the only entry left and push finish.


3) fill in some values for id, name, input and output items


4) Assuming that you know what OpenMI is and that you know what you are doing, you are now going to implement the algorythm part. Click on the class and you will get a new java class wizard to help you. It will already propose you the needed class to extend and the proper actions.
The only thing you need to fill in are the package and the class name, but we had that already in step 3.


Pushing ok will create package and class template for you.


Well :), I know what you are thinking... but hell, no one ever told that OpenMI is an easy thing. I just say it is STANDARD, and that is GOOD!

So fill out your class in a proper way with the input and output items and at the next run of JGrass you will be able to use it in the console, since at startup Jgrass queries all openmimodel extension points and registers them with the console.

if you feel really lost, have a look at the many existing modules. For example h.pitfiller couls be a good one, since it just takes a map in input and creates one in output. Also use the mailinglist, if we can help you, we will.

How to create a command OR Presenting the actions-commands-console-openmi-uibuilder taskforce

You want to implement a new model/algorithm/command/action/do-something (MACAD) in JGrass?

Here more or less the full list of things to do. More or less is due to the fact that I assume you already know the JGrass or at least the Udig or at very least the Eclipse RCP development environment. Also some sort of OpenMI knowledge should be here. All in all I'm not sure how many will find this useful. Hmmm... I'm sure at least one of the chapters could be.
Why I'm doing this? Because next week I have to present a whole bunch of the JGrass stuff at the EGU and I need some docu. Enjoy :)


UPDATE: please note that now the steps 1 and 2 are done the clean rcp way throught extension points. Read here for more info.

1) Create your MACAD

My example will treat an OpenMI based model, i.e. h.pitfiller.

The model extends ModelsBackbone, which implements OpenMI's interface ILinkableComponent.

Create such a MACAD and implement the needed methods.

2) Tell the console engine that the new model is there

In the list of available MACADs add your new model, so that the engine knows that your MACAD exists.
The line looks like the selected in the image below:



which basically tells us what the model's name and class are.
Also information is given about the arms the MACAD has. In this case 2 arm, which define the input elevation map and the output depitted map.

Note that at this point the model h.pitfiller can already be used in the JGrass console as
h.pitfiller --igrass-elevation elevation --ograss-pit pit


3) create a gui xml file definition for the MACAD (the UIBuilder of JGrass)

Create a file with the same name of the MACAD and xml extention and fill it with the following:





The first line will create a label of text descr, a textfield and a button that opens a map selection window.
The second creates a label of text descr and a textfield for the output map name.

When ok is pressed, the widget creates a command string build by the name of the command tag and the repr of the fileds tags, substituting the # with the user input:
h.pitfiller --igrass-elevation userinput --ograss-pit userinput


4) create an rcp action to launch the command

Once the gui definition is there, we need something to trigger the action.

Eclipse RCP's actionset comes to help us. I create a plugin inside which I keep all the gui for console commands and create an action. In the below image you can see the whole context with adifferent menus and action groups.


Please note the textfield class, inside which I put: eu.hydrologis.jgrass.ui.actions.h_pitfiller

The class MUST have the same name as the MACAD you want to create!
Let's create the class by clicking over the hyperlinked text class which will create the action class with all the needed implementation of interfaces and methods.


And now the same class with the needed changes:



Apply the same changes and that is all. If you didn't do so, put the xml gui definition file inside the same package as the action class. Take a look at the screenshot above to see how it should look inside of the actions package.

5) try it out!

Launch JGrass and search in the menu bar your new item. In my case there are a lot, as I already have a whole bunch of organised MACADs in my configuration. However, here I go with pitfiller:



and the autogenerated gui looks like:


6) what happens when I click ok?

The uibuilder creates the commandline representation of the command line execution.
After that the command is passed to the console engine and a backtrace console window is attached to the process in order to deal with output and errormessages.

So the running h.pitfiller command looks like the following:

---

The same could have been done with any GRASS command, apart of the fact that you will not need to create the MACAD yourself :)

Alright, so let's see how to do the same with r.in.gdal, the GRASS native high power raster data import tool.

What you have to do different from before, is that when you create the action class, you need to tell the execution facility class that it is a GRASS command:


Create the xml file with the options you need:



and fully enjoy your new command:

How to create OpenMI based JGrass models

As some of you probably know, the JGrass console engine has been designed to automagically link together OpenMI based models.

This post should really be a big big document and at some point it will be usefull also for first-time-openmi-developers, but as I start now, it is some growing documentation for the JGrass team members, so that we all follow same way of doing things.

So dear guest, if you feel this all sounds strange, don't worry, it has to.


Alright, let's start with some sparse thoughts:

1) for now I will assume you are extending the eu.hydrologis.jgrass.models.ModelsBackbone class
I don't have to tell you that you should not touch that class, right? :)

2) the out and err printstreams have been moved to the ModelsBackbone class. So do not re-declare them in your model's class.
You will be able to access out and err streams from your class, since they are declared protected in the ModelsBackbone.

Do not have such as the following lines in your model, else you will get troubles:

private PrintStream out;
private PrintStream err;

3) implement safeGetValues and safeInitialize instead of getValues and initialize

Two methods have been added to ModelsBackbone to trap exceptions and give the user a feedback on what is going on:
- safeGetValues
- safeInitialize

Those two methods are called from within the interface methods getValues and initialize and are wrapped inside a try-catch block that traps three types of Exceptions:

• OutOfMemory: this one is trapped and sends a standard message to the user on the console, suggesting to add memory to the JGrass process. This is an error that we know very well, how often the active region is too big or the resolution to high and some analysis eats all the memory? With this trap the user will finally be warned.
• ModelSematicException: this one is the most important for models developers. It takes a message for the user (passed directly as error to the console) and when thrown, it exits the safeGetValues or safeInitialize method and prints the message to console. That way semantic problems of a model can be told to the user. An example could be a wrong input flag in the models arguments or a non existing map warning. When processing the exception, also the variable isOkToGo is set to false. This is a protected boolean in the ModelsBackbone class, so it can again be used everywhere in the model to stop execution. If for example in the initialize method something goes wrong, the developer can throw a ModelSematicException and from that moment on the isOkToGo boolean is false and can be checked as first thing in the safeGetValues method.
• Exception: every other exception thrown is sent to console, prefixed by a standard message introducing the real problem. Users will not love the message, but they will at least have something to report :)
  

4) throw ModelSematicException if there are problems

Inside your model, if something goes wrong and you want to write something usefull to the user, throw a ModelSemanticException passing it your message. It will take care of it.

5) do not call safeGetValues and safeInitialize directly

Always call the OpenMI interface methods getValues and initialize! Else you will lose beeing OpenMI based!!

6) to finish for now, a small code example:

The following shows a model (like the one you are writing if you read until now) in which a check on the passed parameters is done. If in troubles, the exception with a nice message is thrown:

  public void safeInitialize( IArgument[] properties ) throws Exception {

      String grassDb = null;
      String location = null;
      String mapset = null;
      if (properties != null) {
          for( IArgument argument : properties ) {
              String key = argument.getKey();
              if (key.compareTo(ModelsConstants.GRASSDB) == 0) {
                  grassDb = argument.getValue();
              } else if (key.compareTo(ModelsConstants.LOCATION) == 0) {
                  location = argument.getValue();
              } else if (key.compareTo(ModelsConstants.MAPSET) == 0) {
                  mapset = argument.getValue();
              } else if (key.compareTo("activefield") == 0) { //$NON-NLS-1$
                  activeField = argument.getValue();
                  if (activeField == null) {
                      isOkToGo = false;
                      String pattern = "Parameter {0} supposed to be used but not supplied.";
                      Object[] args = new Object[]{key};
                      pattern = MessageFormat.format(pattern, args);
                      throw new ModelsSematicException(pattern);
                  }
              } else {
                  isOkToGo = false;
                  String pattern = "Parameter {0} not recognized for model h.netshape2flow.";
                  Object[] args = new Object[]{key};
                  pattern = MessageFormat.format(pattern, args);
                  throw new ModelsSematicException(pattern);
              }
          }
      }

Behind the coulisse, in the ModelsBackbone class, the exception is trapped and the isOkToGo variable is set to false:

       try {
           safeInitialize(properties);
       } catch (OutOfMemoryError e) {
           err.println(Messages.getString("ModelsBackbone.outofmemory")); //$NON-NLS-1$
           e.printStackTrace();
           isOkToGo = false;
           return null;
       } catch (ModelsSematicException e) {
           err.println(e.getLocalizedMessage());
           isOkToGo = false;
           return null;
       }
       //... etc. etc.

Again in the ModelsBackbone class, when it comes to do the job, which is in the getValues method, before launching the safeGetValues method, a check is done if before that moment everything was ok, in order to decide wheather to stop or go on:

   public IValueSet getValues( ITime time, String linkID ) {
       // if there were problems, stop and return null
       if (!isOkToGo) {
           return null;
       }
       // if everything ok, do your stuff
       try {
           return safeGetValues(time, linkID);
       }
       //... etc. etc.

The same can be done eveywhere inside your own code (i.e. not in ModelsBackbone).