GIS clipping benchmark: jgrasstools & uDig (on JTS & geotools)

I hate this. I hate it when I see a callout for a benchmark and am not able to not give it a try. And I hate it even more when I have tons of work to do. But yeah, I love the thrill :)

So here it goes. This morning Markus called out for the GIS Clipping benchmark contest revisited. Well, I somehow felt invited as part of the "other FOSS GIS", so I had to run for it. :)

On thing I noticed at once, is that the reported processing times were quite high. I had no proof, but in the last years I worked (also on editing) withmany 1.5 giga shapefiles in uDig. And to be honest I aslo found it was the only software (FOSS and non) that was able to handle the thing properly without making one get mad.

I wanted to have a look how the geotools/jgrasstools/uDig Spatial Toolbox would perform. I am not a real benchmark guy, but well, I can run some modules to have an idea about more or less what it takes. So I grabbed the testdata and gave them a run.

I have also to say that I ceated a bit because we didn't have a clipping tool inside the spatial toolbox up to now, since in uDig this would be done with the Axios tools. So I quickly wrote one, the VectorClipper.

The machine I tested this on is a 2 years old windows 7 64bit machine with 8 gigs of RAM and java 1.6.
It has 4 cores.

The first run was really simple, reading in the features, create an index to quickly get the features that might be involved in the clipping and clip one by one.

Result using one core: ~201 seconds

Interesting. GRASS takes about 5 minutes, but that is ok (or better, great), since it also does topology. QGIS seems to take about 5 minute, which looks a bit like bad performance to me, considered that our worst case scenario takes a bit over 3 minutes.

Anyways, let's focus on making things better.

The VectorClipper can run in multithreading mode. So I enabled it for my 4 cores.

Result using 4 cores: ~152 seconds

Hey, hey. Nice!
But We can do better than that.

Since the creation of the spatial index was taking quite some time, I decided to drop that one, and use instead JTS prepared geometries, which would make the intersects query faster.

So without an index but with fast intersect query I got:

Result using 1 core: ~130 seconds

Result using 4 core: ~110 seconds

So I though: what if I was wrong about the index thing?
So I added the index back...

Result using 4 core: ~102 seconds

Then I noticed that in the multithreading part I had some lists that got copied instead of just passed as indexes. Once that was fixed I got a nice:

Result using 4 core: ~82 seconds

Which is where I was very satified and stopped my tests.

So the above could be called the geotools/JTS/jgrasstools benchmark, since I was running the module from my development environment.

Running this inside uDig has some more overhead, so it made sense to test that, since that is what the user would get.

Here the result I got:

as you can see from the console output, the processing took about 117 seconds, which is anyways a good performance, considered the available benchmarks.


I can't take much credit on this, all the work is done by JTS and geotools, so the credit goes there.

I loaded the used jgrasstools libs on the site, so they could be used in uDig. If you need help in getting started with uDig's spatial toolbox, have a look at this video.

-------------------------------
UPDATE
Just for the interest/fun of doing it, I ran this on amazon AWS on an cc2.8xlarge instance with 32 cores. It took about 25 seconds, 2 of which for reading the data, 7 of which for writing the result, and most f the rest of the time for creating the index.

The new uDig video tutorial series

The following comes from the udig-news blog, since I wrote it I will report it also here.

Today we activated a youtube channel for uDig: http://www.youtube.com/udiggis

Here we want to collect different video material about uDig.

Some members of the community have agreed on creating each week a new tutorial about any feature not yet described.

Need some examples?

Ever needed to change language of the gui?
Or maybe you are new to GIS and are wondering about how to load common gis data formats?
What about query data in uDig?

More advanced users might need to rasterize a shapefile or vectorize a raster?
Scientific users might want to get started with the Spatial toolbox to do some analyses, maybe start with the extraction of a watershed from a DEM?


Anyone can contribute to the channel and anyone can ask for a tutorial about a certain feature.
On this page information about ongoing tutorial production is kept.
If you need a tutorial about any uDig feature, check if it is not already listed there, and if it is not, join the mailinglist and ask kindly. :)

Geopaparazzi 3.2.5 is out - mostly fixes and localization

Well there have been a couple of bugfix release that didn't need any announcement, but for this one I think it is good to let you know.

One nice guy has added a French translation of the project to our crowdin translation platform. Thanks Fred whoever you are :) !

Please let me know if you are working on translations, I can cut a release for you to see how it results in the user interface.
Since we were there, we updated also the Italian translation.

On request of some users some infos were added to the notes and gps logs:

1) the notes now show also the lat/long of the point

2) the gps log properties now show the start and end timestamp

A major bug has been fixed when merging gps logs together. Now merging is safe:

Last but not least, the GPS indicator has been fixed. It has been reported to be a bit unstable on certain phones which resulted in being unable to record logs or notes.

Enoy and please report back any problem. Thanks!

Geopaparazzi 3.2.0 is out - tons of news

A new version of Geopaparazzi is out in the market: 3.2.0 (or at least it should be soon, lately google play takes some time to get in sync)

The list of news is quite nice:

• huge enhancement of forms with support for multiple tabs, fragments for tablets, images and map screenshots, date widgets...
• integration of forms with kmz export
• style enhancements on kmz export
• send simple notes and bookmarks info via sms
• geocoding via google services
• osm routing via http://openrouteservice.org
• measure tool is back
• opening of geoSMS by geopaparazzi in offline mode
• editing mode for geonotes with forms
• support for rendering themes through style maps for offline maps
• battery level visible in map mode
• gps connection status visible in map mode 

Again I will try to reference to the WIKI, so that you get used to check out the documentation we keep updated with much effort.

NEW ENHANCED FORMS

Forms have been enhanced quite a bit when we started to create survey forms for the earthquake issues and for post flooding surveys. There is now support for enhanced user experience on tablets via multiple tabs and fragments, multiple pictures can be saved and also map screenshots can be added to the forms.

A complete explanation can be found here.

ENHANCED KMZ EXPORT

KMZ export got enhanced in two ways:

• the new form is supported as table view
• the style from the device is kept

SEND DATA VIA SMS

For a quick exchange of basic data it is now possible to send bookmarks and notes (not forms) via sms. The same sms can be opened on an other device and imported. If there are too many data, they are split over multiple sms.

GEOCODING & ROUTING

Support has been added for geocoding and routing. Those features need a network connection.

The options are explained here.

MEASURE TOOL

Since our migration to mapsforge, the measure tool had been removed due to technical issues. Several users complained about that, so we had to get it back somehow.

OPEN GEOSMS OFFLINE

Geopaparazzi is now able to open geoSMS in offline mode directly from the sms. That is handy when you are not online (which in my case is almost always during surveys).

EDITING OF FORM BASED NOTES

Notes that were inserted with a form attached, can now be reopened ad edited. Even if that doesn't show bit screenshots, this is one of the most important features if you are doing complex surveys with forms.

IMPORT IN UDIG

uDig now directly supports the import of geopaparazzi data. A howto can be found here.



That's all folks! Enjoy!

Geopaparazzi 3 is out - explained through wiki-links

I learned through the years that jumping up major versions too often is not good. And I applied what I learned. Well, this time I was forced to, I swear. This had to be geopaparazzi 3. No 2.6.2, no 2.7.0. Just 3.

And  to show that to you all, I also upgraded the whole documentation wiki. And since most of the people out there do not even know that there is such an effort, I will make this release based on its entry.

So, let's walk through the changes that this release contains. The first two changes are definitely the most important ones and depend on the fact that we dropped osmand as mapping engine to go with mapsforge. The decision mostly came from the need to work with vector maps, but resulted in some other big advantage.

CUSTOM MAP TILES

It is now possible to load TMS tilesets into geopaparazzi. One can create tilesets from GIS data layers (also with jgrasstools) and load them in geopaparazzi. Tilesets can be in TMS or GOOGLE mode, and both remote or local.

Professionals will be particularly happy with this feature, since one can create maps from the overlay of complex GIS layers, as our small showcase shows.

VECTOR MAPS

Mapsforge works with vector maps. And so does Gaopaparazzi now (obviously). Vector maps are great, because they take much less space than tiles and one is able to carry along the whole map of his nation on the sdcard at each zoom level. That is an amazing feature. Never more missing tiles during your surveys!

Mapsforge generates and maintains a set of standard vector maps. Those guys are really amazing!

Recently the osmandguys started generating vector maps rendering after the opencycle maps.Those maps are great, because they also contain isolines.

OSM TOOLS

The OSM tools god some enhancement, mainly related to the ability to update the osm tags once they are extended on the main website. A tutorial is now available.

BOOKMARKS AND PROXIMITY ALERT

The bookmarks list has finally been enhanced to look a bit nicer. Note that it now gives the possibility to enable certain bookmarks with a proximity alert, which will ring when one get's near the point of interest.

FORMS AND TAGS

The forms part has been enhanced with some new tag, which can be usefull for more complex forms.

IMPORT AND EXPORT

New import and export features have been added. It is now possible to import and export from GPX, export to KMZ, export to some cloudbased service. An utility to import and export bookmarks has been added, which makes it easy to move bookmarks around projects.

CLOUD PROJECTS

Geopaparazzi exposes also a very simple API to load projects into the web and download online projects definitions as well as download actual projects. Currently we have no server to make this public, so you will have to believe in what we wrote here.

TRANSLATIONS

We didn't want to wait any longer to release this version, so we are behind with the translations. We have instructions for potential translators. Please help translating geopaparazzi in your language!

CONNECTIONS TO OTHER SOFTWARE

Until this release only BeeGIS was able to seamlessly import data into a real GIS software to allow further analysis.

Since the GRASS community sprint in Prague there is one new amazing software that is able to import geopaparazzi data. There is no info page available yet, but the module is already usable through the GRASS extensions engine. The module works with GRASS 7.


And that's it for this release. Enjoy|!

Geopaparazzi Web Map Tiles enhancements

Well, why should geopaparazzi hang behind uDig? :)

Ok, there are a number of reasons, but in this case, it doesn't have to be like that.

A small piece of history
Some of you might know that I am now part if the Italian TANTO geospatial (and not only) information/education blog.
In April I decided to go and visit the one guy that actually founded that blog in the most distant place in Italy from where I live: Andrea Borruso from Palermo.

We spent a couple of days daydreaming under the heavy effects of muffolette, cassatelle, cannoli and the grand finale of the sfince di San Giuseppe.

The recurring question was: how do we bring a custom raster into geopaparazzi? The potential is there, but we need to be able to load custom map!

Well, at the end we figured that the best thing would be to support local (as well as remote) TMS.
Andrea is black belt of TMS and was able to provide me with some tilesets.

So I started to implement the possibility to load the tiles on the phone and describe them with a small file, so that they would appear as tile sources.

We will throw out this new version as soon as the documentation is ready and some good tutorial on how to quickly handle data and stuff. as usual, if you wanna help, please contact us, we badly need support.

But in the meanwhile I really want to share some examples of the results we were able to achieve.

1) tiles of the ortofoto of sicily, zoomed on Palermo

2) an archeological area airbourne survey, done with drones. Imagery provided by the Consorzio TICONZERO. Interested how they are doing this? Have a look here.

3) technical map, overlayed with the results of a risk analysis (perfect for outdoor geologists and engineers)

Do you remember seeing this image somewhere already? Maybe loaded in uDig? :)

4) Last but not least, a scanned historical map (shiver down my spine)

Ok, need to stop here.
Basically soon it will be possible to load almost any kind of raster map into geopaparazzi, which is an amazing enhancement. Now all that's left, is to wait for the next release :)

This post is born as a parallel post (as in two different ways to tell the same story by the people that lived it) of this post.

uDig web map tiles enhancements

Many of you for sure know that uDig has the ability to show Web Map Tiles.

From the import dialog you select Web Map Tiles:

push the next button and find a long list of available web map tile servers (which we should probably check).

There is also the possibility to select a custom server. This is very handy, but had 2 problems:
- it could not load local folders of files
- it understood only the google tile schema numbering, which means no TMS

Well, this is the first enhancement. Those two problems were solved. Mind that you will have to check the right option:

The second enhancement will make uDig Spatial Toolbox users happy. With the next uDig release we will also release the module that creates TMS folders from a set of layers. I do not want to go into detail shere, but a new module will allow to export a set of styled layers as tiles folder. Those tiles come with the complete description of the zoom levels and so on.

In fact since that file is the only thing needed to load a folder of tiles, it made sense to add a small item to the list, the JGrasstools TMS folder:

And there we have it. Push the button, push, push the button and:

This will be extrememly helpful for those like us that use uDig with BeeGIS field extensions out in the field on netbooks. The base dataset gets extremely lightweight to load and rasters can be prepared to be fast and easy.