Geopaparazzi 4.2.0 is out: never out of data in the field

When I got that odd issue on the tracker about the GPS azimuth value not being properly handled in Geopaparazzi, I didn't think that this release would have any new feature. But well, once you sit down to fix things, you see things... and once you see thing, you want things :-)

First off, some bugs should be now fixed:

• notes halo is ignored 
• mapview doesn't render points and tracks when returning from standby
• while downloading remote tiles sometimes the dashboard freezes 
• azimuth should be only one: this is a big one, since now we the azimuth is always calculated properly and when a picture is taken it will be the right one.
• geopap sometimes crashes when taking pictures

And that is already nice, particularly the azimuth one.

But here comes the Boom:

OSM data extraction from mapsforge tiles in offline mode

The mapsforge tiles are generated on the device from a particular vector format. This means that there are information available in the database. Problem is that, very very simply put, the information contained is extracted differently at different zoom levels, because in fact the library and the format have been done that way to allow rendering performance.

But still it is possible to extract almost everything we see, which is nice.

Let's see how this works. Assume you have a job to do, are out in the field and want view information overlayed on the ortofoto pictures from the local WMS service.

Well, the map file you get from mapsforge looks like the following:

Now go in the mapvieW menu and you will find a new entry: Import mapsforge data

The import view will appear:

From the view you can see that 2 types of data can be imported: points and ways.

Points

Since the points are often visible on a different zoomlevel then the current, also 3 zoomlevels below the current are investigated to extract data and double points are not considered. So if you start this at zoomlevel 16, you will also get 17, 18, 19. Since the same are at a different zoomlevel will have many more tiles, about 10000 tiles are read to import the data.

You can add a filter text to import only tags containing a given text or exclude all those containing the text.

Points are imported in the current projectdatabase and saved as forms notes containing all the values Openstreetmap has. As such they can also be edited.

Ways

Many types of ways are stored in the mapsforge map files and many of them are actually related to areas.

The user can choose to import:

• ways: roads, railways, cableways and similar
• waterways: lines that represent water
• contours: contour lines if they are available

Since these data are heavy, the data are imported into a spatialite database.
This brings up a new issue: since this release a database for mapsforge extracted data is automatically created if there is none present. You will find a database named mapsforge_extracted.sqlite always present in your maps folder. And you will find 3 layers always present in the spatialite data layers: osm_waterlines, osm_roads and osm_contours.

But let's get back to import the data. Just select the data you want to import and push the start button. In the case you selected all data types, you should see first an import dialog like this:

and then something like this:

Nice ha? :-)
Depending on what has been imported first, the labels might not be coming from the right field. In that case you can simply change the label in the spatialite layer settings.

Before we look into it, let's see what happens in the case we use a map that also shows contour lines. To do so, we want to clear those layers. The fastest way to do so is to simply delete the mapsforge database and let geopap recreate it.
So after doing so and loading a map with contours I will have the same region as:

If you now import everything available exactly as before, you will get:

To have a better idea, change the background map to something different. I also change the contours color to white:

Label support is not advanced, so they get readable only once you zoom in:

That is quite cool ha? All from offline data!!

You now might wonder why all imported notes have a (MF) in their name. That is done so one can quickly select and remove them. Believe me, that is a feature you want to have. The region I am showing you has few points and already around 50 notes have been imported. In towns this is exponential.

This anyways brings us to the next point:

Enhancement of notes handling

While working on this, I noticed that when importing notes, all my notes got hidden in the mass of imported notes. Also, many of the imported notes are POIs that I do not care about (ex. benches, towers, etc).

The current notes list didn't have the tools to handle this. So how does it work now?
Enter the new notes list:

This is how it looks like after the previous import. Somewhere there is also a note I created manually "My tiny note".

As you can see, notes can now be selected (checkbox on left). You can zoom to the note as before and all other icons are now in the last one on the right side.

If you tap it, you get a context menu:

• Edit: as before it allows you to edit a note if it is a form note and view image notes
• Share: share the note in social networks
• Delete: delete the note
• Use as selection: this gives the possibility to select only those notes that have the same name as the one used to pop the menu
• All notes: opens a submenu for actions that apply to all notes

 Lets assume I want to remove all bus stops. I select the menu on any of the bus stops and tap on the Use as selection.

Only the bus stops will be visible and selected:

Now access the All notes menu:

well, it is quite clear what to do. Delete all selected.


What if I want to remove all mapsforge notes? Insert (MF) in the filter text and remove all selected.

Last but not least, with all this spatialite fiddling, a big help has been added for vector editing.

Import a template spatialite database

If you go in the import section, you will find a new entry: Default Database.

When hitting that, you are asked to name the new database to create, let's use testdb for the sake of this example.

Now you will have to restart geopap, sadly that is still required. One it is done, go in the spatialite database view. 3 new layer of the database testdb.sqlite will be visible:

While lines and points won't be of much use in geopap yet, you will find the polygon layer interesting, since it is editing ready. Enable editing and edit right away. Since it is a template db, the attributes table has been created as 20 fields with names from field1 to field10. As said, very simple, but in my opinion still of use when you have to quickly collect some polygon data with attributes.

That is all for this release... enjoy!!!

HydroloGIS, 10 years of proud Italian Scientific Geographic Open Source and Environmental Engineering

I still remember it as if it was yesterday. We were sitting in the lab of Unitn (the Faculty of Evironmental Engineering of the University of Trento, where we were doing research for professor Rigon) and we were thinking about what to do. Money was running out in universities and the figure of external research staff was being obsoleted. Italy was going to face one big economical crisis.

I remember Silvia saying that she wanted to do a couple of years of research and then head off to do missionary work in Africa.

I remember not knowing exactly what I wanted to do. I had just found what I loved to do: developing open source GIS. It was not just developing anything. It had to be science and it had to be open. I remember dreaming about being able to develop any kind of tool for anyone and for free... right where big big companies were charging in an unfair way. I remember thinking that research should be for everyone out there. And I remember professor Rigon being of the exact same idea. 

At that time the slogan was: All in a aGIS!

We had figured that when it comes to environmental sciences, the glue to keep them all together could only be the GIS. With Riccardo we always joked about taking over ESRI at some point. Well, that one didn't work out that well. :-)

The first logo of HydroloGIS.

The second logo of HydroloGIS.

The current logo of HydroloGIS.

I told Silvia I thought she would be wasted for “normal” missionary work and that she should push her talents hard to do what many others would not be able to: create useful tools and then bring them developing countries together with training for it. And somehow I convinced her on something she didn't need to :-). 

I remember her telling me that most probably in about 5 years from that moment she would leave HydroloGIS anyways to follow her path in development countries.
I remember thinking: "It's ok even like that!". Being able to share a sparkling new experience with a pal that didn’t care a fig about making money exactly as I did, with the only wish to make science tools to release open source for the people… well, that was something that would not be easy to find again. We had to give it a try.

At that time she was following a master about project planning in Padova. One evening I was in the lab and she called me from Padova: “Are you serious about trying to create a company? There is this startup competition here... we could participate… everything needs to be delivered by midnight of today”. We chatted around for a short while, she gave me the link to the competition and then her cellphone battery died with her being spending the evening out (and not being easy to charge phones back then). I had no idea where to contact her again. I didn’t even know her home address at the time. I had to look up her parent’s phone in the paper phonebook (remember?) and call them to get Silvia's data for the contest. The lord meant it good to us, since during the following hours of struggle to write down the business plan idea and completing the competition bureaucracy also a name for the new company had to be chosen. Man, am I happy that HydroloGIS came that natural!

HydroloGIS at "Startcup"

Funny thing is that we wanted to try a startup competition to have confirmation about how good our idea was: Bring innovative open source technologies from the university to the professional world in order to make the world a better place. Sounds just about right, doesn't it?!?!?
Well, we didn’t even pass the first round!!! I remember project N.1 being a portable DNA analyzer device (I wonder where that company is today). We were so angry! How could they not understand!!!! So we decided to prove them wrong. And that is how HydroloGIS got born.

The really empty office of HydroloGIS at day 0. Those who now it now, would not recognize it.

We found shelter at the TIS Innovation Park Southtirol. At that time it was called BIC, the Business Innovation Center, which was an incubator and highly supportive for ideas both based on Environment and Open Source.

HydroloGIS, John Preston from the Jamaican reserach center ICENS and the University of Trento developed together JGrass, the open source GIS that wanted to be a userfriendly graphical interface for GRASS and specialize on hydro-geomorphological processing based on professor Rigon's past decade of research. 

The first JGrass logo, based on GRASS' logo

At that time the only development sponsorship to JGrass came from the MIGG course that we held with Rigon's team each year at the University of Trento.

And this was the first team preparing the course right after the MIGG2005 edition:

Andrea Cozzini, Erica Ghesla, Silvano Pisoni and HydroloGIS

Here the first JGrass versions have been developed.

One thing we have always been bad at was looking for work. At the very beginning we tried some commercial campaigns to sell HydroloGIS products around the region. Eventually we figured that it didn’t work. We have never been able to explain what we do properly and we simply didn’t have the skills to sell.

Luckily for us at the start there were some rather badly paid, but extremely interesting, projects with the university. Then later some jobs for government agencies came in. Interesting enough, we never looked for a job since then. The jobs just came to hunt us. And I mean it when I say "hunt". People started to come to us when they didn’t know where to head to solve their problems, usually after having tried for a while, hence burning both avaliable time and budget. And we took all those jobs, once again, bad paid and sometimes so demanding, that we had to work for free a long time to finish the job. Those were (hard but) great times, in which we grew incredibly from a professional point of view. But those were also hard times. And I exactly remember getting at the end of the year not having enough money to pay taxes. After a year spent working on average 15 hours a day!

But still we stuck to the plan: do everything the open source. It had to work!

Eventually one year passed and we were still in business:

 

and not in the worst shape (even if I was gaining quite som weight, eventually passing 100Kg :-) ):

In the years that followed (initially mainly on behalf of the University) we taught at several different courses and master courses, gave lessons at the university and to professionals, participated at many Conferences, EGUs and FOSS4G's, went to code sprints, developed GIS, made Engineering work, entered the OpenMI Steering Committee, the uDig Steering Committee, developed and coordinated JGrass, BeeGIS, the Nettools, the JGrasstools, Geopaparazzi and eventually won different innovation competitions.

Silvia winning the first price for innovative women entrepreneur.

GRASS code sprint in Prague.

JGrass went through many different periods. Being rejected from the GRASS community we had tried to develop it on our own, but it had overwhelmed us. we were not a pure software company.

A short history of JGrass until 2010, as presented at FOSS4G Sydney.

So in 2007 one big move had been done, the migration in what I was convinced was the best java open source desktop GIS (I still am convinced it has that potential): uDig

The first splashscreen of JGrass as uDig extension.

HydroloGIS, beyond others, teaching to high school teachers in Cape Town after the FOSS4G.

Talking about JGrass at Foss4G Sydney.

and having great fun at the same conference, always wearing the colors :-)

always ready to attend to important meetings to shout out our thoughts.

With the begin of my PhD at the University of Urbino a new open source door opened up. The mobile field mapping.

The BeeGIS extensions for uDig were developed in that context, which gained quite some interest. Being uDig based it worked only on full size pcs, or better tablets. Back then the big deal where Ultralight tablets, that had some complete-yet-downscaled operating system and were able to give enough juice to make uDig run on them. 

The splashscreen of the BeeGIS release for which ARPA Piemonte sponsored a nice part of development both for BeeGIS and uDig.

Digital field mapping as PhD student in Urbino.

Today I still have to laugh about that and I know some of you don't even remember. But tablets were heavy back then:

but everything is possible if you only want it:

In 2007 the first Geopaparazzi was born. And nothing has been as it was before :-)

We were finally able to have a tool that we would always have with us and with that we would be able to map in extreme situations:

Android made the BeeGIS project die out, because we simple stopped using it to develop the Geopaparazzi project, which, funny enough, turned out to be our most supported in the GFOSS world. There is even a facebook group for it now :-)

The years passed and many interesting things happened, as well as quite difficult things that hit hard on us. We tried to work tightly with other companies, which sometimes didn't work out well. We also tried to have employees, which did really work badly and we figured that would be something for when we are old :-). 

As I am writing this up now, overwhelmed by happiness and proud of what we achieved, all the bad things just vanish and I could go on telling you stories about our life, but I think it is really enough now. :-) It is really more material for a pub than a technical blog. But you got the point: I love the way HydroloGIS turned out, with its ups and downs.

The sole fact that few years ago I have been able to go back again to do music actively with my Alpentraum Orchestra, is a great indicator of having achieved what I wanted to: a job I love to work on every minute of the day and (some degree of) complete freedom in the organisation of time, which makes family and music possible.

Over time HydroloGIS made sense, even if they told us that it would not make sense to sell things for free and that we were crazy to do that in our niche of work... well, we were never able to properly explain it around here, we simply knew it was the right way. Eventually we figured that it was the somehow old Italian method that was wrong, not us. 

Our open source products and community involvement started to bring great jobs outside Italy and even overseas. While we still were not able to get work within a 100Km radius around our office (not one single job), we started to work in Germany, Finland, England, the United Arab Emirates, New York.

And obviously there were the projects in development countries, so important to Silvia :-). We were able to bring training sessions and our tools to Rwanda, Ethiopia and Kongo and work on water management systems. The sparkling new agreement with the GISMAP will help us to spread Geopaparazzi in the eastern side of the globe.

Silvia teaching Geopaparazzi in Arba Minch.

Teaching uDig and GIS at the University of Arba Minch.

Alright, I think I have kept you here for enough time now and my story is getting very confused. I know I missed many important things and people. Don't feel bad about it, this is a simple writedown, in a couple of hours, while browsing through old pictures and sensations.

One last thing I want to do is to thank those that believed in us even when there was no money and no future (well, that would be for about the first 6 years :-) ). Thanks to our families and partners, that supported us by any possible means. Thanks for being there, thanks for supporting without asking questions. Just thank you!

(In this picture, taken on HydroloGIS' 8th birthday, my mother and my sister Michela are missing. My mother was taking the picture and Michela lives too far away to be able to attend to every party we throw :-). I want them to know that they are part of it.)

Thanks also to Silvia, with who I had the luck to share these last 10 years. It has been sometimes difficult, she is soooooo hard-headed, but then I know I am too. :-) Thanks for letting me try all those things, even if no evident money would come out of it and we were starving :-) Thanks for being the other half of HydroloGIS.

HydroloGIS is Tony and Silli.

Last but not least, thanks to all of you that have sympathized with our concept of leading such a small company the meritocratic and open source way. We know you are out there and we have always been honoured when you told us how much you respect and like us.

Happy 10th birthday HydroloGIS, while my eyes are getting slightly wet...