Geopaparazzi 5.4.0 is out

We just released Geopaparazzi 5.4.0 into the wild!

It all started with a bug fix to have a 5.3.2 that would be able to read the latest mapsforge maps. For some unknown reason the backward compatibility had been lost and geopaparazzi was no longer able to visualize newer mapsforge maps. Being the most important map used, that had to be fixed.

Then we noticed that also GPS import was broken. In fact the new plugin system had a bug. That also had to be fixed.

Then we fixed some smaller usability issues that had been reported, but this also was ok for a 5.3.2 bugfix release.

What made us decide to go for a 5.4.0, was a fantastic project we are doing with other members of the gvSIG Association for INEGI. INEGI is the National Institute of Statistics and Geography of Mexico and recently we won a tender to supply a whole GIS stack for Desktop-Server-Mobile.

At HydroloGIS we are obviously taking care of the mobile part and it is thrilling to know that INEGI invested in Geopaparazzi/gvSIG Mobile and that hundreds of technicians will use it to do their digital field surveys.

That project already brought in the previous release an important change which is the plugin system.

Recently it also added to geopaparazzi two important features:

1) support for unique values map theming in spatialite databases

Geopaparazzi now supports in readonly mode unique value styled spatialite layers.

We are used to a simple style in geopaparazzi:

Now you can do things like these...

Polygons:

Lines:

Points:

Looks quite nifty :-)

But now the problem... how do we make such a styled database?

We created a tool inside the JGrasstools Spatialtoobox, which means it will also get to gvSIG. Right now there are compatibility issues between the spatialite drivers of JGrasstools and gvSIG. We will probably get there for the 2.4.0 release.

In the meanwhile people interested can pick the standalone version of the JGrasstools' Spatialtoolbox, which can be downloaded here.

Just run the script jgt-spatialtoolbox.sh and look for the right module:

Now pay attention. This will import a folder of shapefiles into a newly created (or existing) spatialite database. If the shapefile has a style defined in an SLD file, then that will be used. So use your favorite SLD editor and create a unique values theme, and it will be inserted in the new database, geopaparazzi ready!

2) pdf export

The second interesting feature is the PDF export of form based notes.

For those users that couldn't care less of GIS and complex stuff, but only want to have a nice ordered list of the surveyed data, the PROJECT FORMS TO PDF button is the right choice:

This will export the content of the form based notes to PDF, trying to group the information nicely into tables, adding any image contained in the note.
Just to have an idea, this is how 2 random pages would look like:

That's it, huge thanks to INEGI for pushing this project and making geopaparazzi better!!

First release of the JGrasstools and Geopaparazzi plugins for gvSIG

Ther are finally here. We were able to finalize, test and package the JGrasstools and Geopaparazzi plugins for gvSIG and make a first official release out of them.

It is a first shy 0.1.0, but it contains tons of functionalities ready to be tested by the community.

To download and install the plugins, please have a quick look at this document.

Not much more to say, everything is in the linked quickstart guide... enjoy!

Bye bye uDig, hello gvSIG world!

Bye bye uDig

This post should have been written ages ago. But I have never been ready to do so. And it still is very difficult for me to properly write something about my exit from the uDig community. The Open Source ecosystem is a particular one and people not involved in the same way understand things differently. But I love the Open Source ecosystem, it gives you always a new possibility even if it might seems there is none.

So I will try to stick with the facts, even if being my personal blog some passion could seep through.

We have loved uDig for many years. We have contributed about everything that means raster analysis in uDig and have been in the Project Steering Committee for several years. We fought the trend that was keeping uDig "only" an SDK for developers instead of a desktop GIS for the community. And well, it is time to say that we failed.

My guess is that uDig will never be a widely used desktop GIS, i.e. a standalone GIS. uDig will always be an amazing SDK. Something you use to build great location aware applications.

uDig has always suffered from the fact that there were almost no contributions in the last 5 years. No resources/interest to the fix bugs that were stopping people from using it. uDig didn't evolve much. Resources were really low and at some point I remember always the same 3 developers trying to get uDig up and running, adding stuff, making it work: Jody, Frank and myself.

Then, in this low resources situation we made a huge error. We started the migration into the Locationtech Fundation. Which is not an error per se, but in the conditions the community of developers was in, that has been an irresponsible move. uDig entered a kind of limbo from which I think it still didn't exit. There was no real well working version available and all resources were in the migration process. A process that lasted about 2 years. In the meanwhile we lost possible contributions (at least one big I know of), since it was not clear were to contribute: the old was old, the new was never finished.

In the middle of all this mess, in which Jody and Frank were doing a big big voluntary work and keeping the uDig ecosystem on their shoulders alone (kudos for all that to you guys), I started to retire from uDig. Several long discussions at the Nottingham Foss4G opened my eyes an made me understand that I was in the wrong project.
uDig was on the way to be an only-SDK project (if you disagree with that i will be happy to openly discuss about it) and this was no longer something I would be part of.
Also, the Locationtech environment was not exactly what I expected it to be (I am not saying good or bad, I am just saying not the right thing for me).

So we left... without knowing where to go. For a couple of years we were working without a desktop GIS. Something that had never happened before... and it felt really wrong!

But our clients needed our tools, so we created the standalone version of the spatial toolbox, S.T.A.G.E.
Funny thing is that once the tool was standalone, also QGIS users approached us, because it was handy. They didn't like it in uDig, but if they could use it with QGIS it was ok :-)

Hello gvSIG world!

Well, time passed and it was still unclear to us were we would land. uDig was still blocked somewhere in the migration process... when a company of friends of ours invited us to come to the gvSIG conference in 2014, offering to pay travel and accommodation. They were interested to see our Geopaparazzi digital field mapping tools inside gvSIG to use them in projects in development countries. While I was still emotionally too bound to uDIG to cheat on it, Silvia was very sensitive to the words "development countries", so we decided to accept the offer.

We had a private meeting with the gvSIG Association members and developers to discuss the possibilities to work together and migrate our JGrasstools and Geopaparazzi libraries into gvSIG.
I have to admit they were extremely helpful and we also made a session to quickly get me up and running with gvSIG development.

After the conference we came back to reality... with no funding we could never migrate... the process was simply too big. So time passed and nothing happened. Secretly I hoped the uDig migration nightmare would be over and we could go back to normal life.

A couple of months ago we decided to end the suffering and at least try something. So I contacted Alvaro Anguix from the gvSIG Association and discussed with him the blockers we had (mainly about the projection system) to start using gvSIG. After some discussions, we decided that it would be best for me to spend a week with them at the gvSIG headquarters in Valencia, hacking, coding and discussing the way thing should or should not be.

At the begin of this month I did exactly what that. I tried to prepare myself as much as possible in order to have tons of complex questions for their developers and flew to Valencia.

I had the luck to sit for a week near gvSIG's main architect and talk daily also to other developers, as well as discuss about user needs and features and Open Source philosophy. I found gvSIG to be a nice community, even if until that moment focused heavily on the latin world.

So let me show you what we were able to achieve in the last weeks.

I started with the aim to:

1. bring the water management module based un Epanet into gvSIG
2. bring support for geopaparazzi databases to gvSIG

The water management module

Porting that module has been fun. The biggest difficulty has been understanding the styling engine of gvSIG, which is way different from uDig.    

So for now we have a HydroloGIS menu inside which an Epanet menu appears. From there you can do the usual stuff: create new project files, link the layers properly together the Epanet way (filling in the attributes) and style layers:

When you run the module, the usual wizard appears. Now that wizard remembers exactly everything you entered before, which comes in handy when working a lot with simulations.

Results are stored directly into a sqlite database. One file can also contain more simulation runs.

The results can be visualized in two ways:

1. view the state of the network in one moment (in time) on the whole map. This now looks way better than in uDig, since gvSIG allows for a nice legend to be updated. So in the layerview you will see also the values for each colored pipe or pump or junction.
2. view the whole timeline of one of the pieces of the pipeline.

If you are interested to know more about it, come to this year's gvSIG conference, Silvia will give a talk about it:

GIS tools for water supply systems: an implementation using JGrassTools and gvSIG

Geopaparazzi support

gvSIG now has direct Geopaparazzi database support. That means that as you add any WMS, shapefile or tiff layer, now also an option for Geopaparazzi appears.

Once you select the database file, some information about the database will appear, as well as the layers it will create on loading:

You have the option to import it to temporary layer, but also to create shapefiles from the database.
The second option gives more features and is the suggested way to go.

Once imported, the layers will be generated with their own default style and labeling:

The media layer can now be queried with an own tool:

So if you select one or more images, they will be opened:

gvSIG now also has the tool to create a tileset (for Geopaparazzi basemaps) from the current view:

If you are interested in this, again, come to this year's gvSIG conference, I will be giving a talk about it. :-)

Digital field mapping with Geopaparazzi and gvSIG

The spatial toolbox: JGrasstools

Since things were going really fast (I was in Valencia and working around 15 hours a day, no partying for me this time :-) ), I decided to also give the spatial toolbox a try.

One thing that helped, was the fact that gvSIG is licensed under GPL license, which is the same one I love and JGrasstools are licensed under. That gave huge advantage during the integration.

This took a bit more work, also after the Valencia days, but it is already usable:

Obviously the layers are taken from the current selected map view and the coordinates (as in the above extract basin module) are set form clicks on the map.



If you are interested in this, again-again come to this year's gvSIG conference, I will be giving a talk about it. :-)

New tools for LiDAR, forestry, river management and hydro-geomorphology in gvSIG

Styling rasters

I am not sure why that happens, but every time I approach a GIS first, rasters is one of the last things to be considered. :-) Also the gvSIG raster system is not the best right now, but I know that there are resources to work on it next year, so I am quite happy about it.

For now, one thing that is really necessary, is the possibility to style rasters properly.

Right now, if I define a colortable for a map, I get 255 color rules.
One good example is the map of aspect. Such a map, that ranges between 0 and 360 degrees, is usually coloured from white to black between 0 and 180, and from black to white between 180 and 360. So all you need would be 3 rules, not 255 which make everything unreadable (apart of being wrong):

If I wanted to customize them, the only way I seemed to have, was selecting and editing each rule. That got odd very quickly.

So it has been better to invest that time to create a small styling tool, which right now is hosted in the HydroloGIS menu, but I really hope it will get into the raster style engine at some point.

An example: One of my favourite base maps is the elevation model with the aspect map overlayed with transparency, which gives that 3D sensation.

Just select the colortable and the transparency. Also the number format pattern in the legend and push apply. That is it:

Some legends are just colors, which will be adapted between the min and max of the raster map. Others, like the map of flowdirections instead, have defined unique values and colours:

One thing I have not been able to, is styling a map with a logarithmic scale. I have tried but failed. That one is for example important for a map like the one of total contributing areas, where the values range from 1 (many many cells) to huge numbers (but few few cells):

well, this should look more like this:

Coordinates Info tool

To exercise myself (and because Silvia forced me to :-) ) I developed a mini-plugin that allows the user to view the clicked coordinates and see them in another projection, but most of all allows to copy them quickly to use them:

Well known text tools

A last small tool that I added is the WKT toolbox. It is a very simple tool, but we find it very useful:

With it you can select a geometry in the layer and extract the WKT representation of the geometry.

The same way, in the lower box, you can write/paste some WKT geometry and it will be inserted as new feature in the currently selected layer, if it is of the same geometry type.

This makes it very easy, for example, to insert points in a layer.

Conclusions

Well, it is strange for me to finish a blog post with a conclusions section, but this one requires it.

I love the open source ecosystem. Once again, even if with much work and some investment, we have been able to find what we needed. HydroloGIS will from now on embrace the gvSIG community and bring its tools (as well as the ones from the universities of Trento and Bolzano) into gvSIG (time and resources permitting).

We have a desktop GIS again. We have a community that focuses on a community GIS again. One that focuses on user needs and tools, on cool selection features, even cooler editing features, on a working printing system (do we want to produce maps or not!??!?!), on 3D, on linear referencing, on table joins, and... and...

Hardhearted I bid farewell to the uDig project. With a small tear in my eyes I say ciao to those great developers with which I shared many nights, code sprints and fun talks. Jody and Frank, it has been a great journey with you and I will see you at the next Foss4g. I know you will bring uDig on top again at some point even without me ;-). Good luck in your quest!  

STAGE - Desktop Tools (also) for Geopaparazzi

With the new upcoming geopaparazzi 4 release a few issues had to be solved:

1. What should I do with all my old geopaparazzi 3 surveys? How do I integrate them with new data?
2. What do I use to export data from a geopaparazzi 4 project to my GIS environment?

Well, we solved both the above problems inside STAGE, a Spatial Toolbox And Geoscripting Ennvironment, which is the tool we develop and use for the JGrasstools library for environmental modelling.

For those asking if it is the same Spatial Toolbox that comes with uDig, the answer is yes and no. Yes, because it is the same source code. And no, because since uDig is in the process of (and a bit stuck in) migrating through the Locationtech incubation, we needed to step forward with the geotools versions. So see STAGE as the temporary solution to work with the Spatial Toolbox, until we can enhance the uDig version again.

So where do I get STAGE and how do I use it for Geopaparazzi? A video tutorial showing it all can be found here:

Enjoy!

ADDENDUM: since it has been asked, this is the link to the current download area of STAGE: http://git.io/stage_releases

GIS clipping benchmark: jgrasstools & uDig (on JTS & geotools)

I hate this. I hate it when I see a callout for a benchmark and am not able to not give it a try. And I hate it even more when I have tons of work to do. But yeah, I love the thrill :)

So here it goes. This morning Markus called out for the GIS Clipping benchmark contest revisited. Well, I somehow felt invited as part of the "other FOSS GIS", so I had to run for it. :)

On thing I noticed at once, is that the reported processing times were quite high. I had no proof, but in the last years I worked (also on editing) withmany 1.5 giga shapefiles in uDig. And to be honest I aslo found it was the only software (FOSS and non) that was able to handle the thing properly without making one get mad.

I wanted to have a look how the geotools/jgrasstools/uDig Spatial Toolbox would perform. I am not a real benchmark guy, but well, I can run some modules to have an idea about more or less what it takes. So I grabbed the testdata and gave them a run.

I have also to say that I ceated a bit because we didn't have a clipping tool inside the spatial toolbox up to now, since in uDig this would be done with the Axios tools. So I quickly wrote one, the VectorClipper.

The machine I tested this on is a 2 years old windows 7 64bit machine with 8 gigs of RAM and java 1.6.
It has 4 cores.

The first run was really simple, reading in the features, create an index to quickly get the features that might be involved in the clipping and clip one by one.

Result using one core: ~201 seconds

Interesting. GRASS takes about 5 minutes, but that is ok (or better, great), since it also does topology. QGIS seems to take about 5 minute, which looks a bit like bad performance to me, considered that our worst case scenario takes a bit over 3 minutes.

Anyways, let's focus on making things better.

The VectorClipper can run in multithreading mode. So I enabled it for my 4 cores.

Result using 4 cores: ~152 seconds

Hey, hey. Nice!
But We can do better than that.

Since the creation of the spatial index was taking quite some time, I decided to drop that one, and use instead JTS prepared geometries, which would make the intersects query faster.

So without an index but with fast intersect query I got:

Result using 1 core: ~130 seconds

Result using 4 core: ~110 seconds

So I though: what if I was wrong about the index thing?
So I added the index back...

Result using 4 core: ~102 seconds

Then I noticed that in the multithreading part I had some lists that got copied instead of just passed as indexes. Once that was fixed I got a nice:

Result using 4 core: ~82 seconds

Which is where I was very satified and stopped my tests.

So the above could be called the geotools/JTS/jgrasstools benchmark, since I was running the module from my development environment.

Running this inside uDig has some more overhead, so it made sense to test that, since that is what the user would get.

Here the result I got:

as you can see from the console output, the processing took about 117 seconds, which is anyways a good performance, considered the available benchmarks.


I can't take much credit on this, all the work is done by JTS and geotools, so the credit goes there.

I loaded the used jgrasstools libs on the site, so they could be used in uDig. If you need help in getting started with uDig's spatial toolbox, have a look at this video.

-------------------------------
UPDATE
Just for the interest/fun of doing it, I ran this on amazon AWS on an cc2.8xlarge instance with 32 cores. It took about 25 seconds, 2 of which for reading the data, 7 of which for writing the result, and most f the rest of the time for creating the index.