Monday, November 18, 2019

Geopaparazzi 6 is out. Now or never!

It has been an infinite time from the last geopaparazzi release. I only figured when I did the release. With all the work around the Geopaparazzi Survey Server and SMASH the main kid has been left behind.

So today it was the now or never. We are throwing Geopaparazzi out to the market. It will no doubt be the most important release of the year.

The new features can be found in this presentation:
https://www.slideshare.net/moovida/geopaparazzi-state-of-the-art-of-the-digital-field-mapping-application-167930916

Many thanks to several people that helped on the manuals, with fixes and features, that made descriptive and useful bug reports. Brent, Eli, Peter, Tim, Andrew, thanks for the support.

Things will still get quite crazy from here on. We are almost ready to add support for geopackage (also in editing mode... I hope), which is getting more and more used.

So enjoy. Test, Report issues. Enjoy!

Thursday, August 8, 2019

Geopaparazzi 6 betas - testers wanted

Dear all,
as you might have noticed, the upcoming geopaparazzi release will introduce huge changes due to the migration to the graphically accelerated VTM engine in substitution of the old version of mapsforge we were using and that was getting unusable due to https online resources policies on Android.


Also this version of geopaparazzi contains a sparkling new build of spatialite based on proj 6 and is available for 64bit devices. So this also will need some review to make sure it works on most devices.

All this allowed us to have several great functionalities as for example a 2.5 view


or the possibility to have multiple layers loaded (ex. mbtiles on top of online maps). Also this will soon open the possibility for vector tile layers, which is a huge improvement on the long run.

Being that many changes in the game, Brent Fraser suggested to make the beta release available for community testing and I agree with him.

The first beta release has been uploaded and a link will be placed and updated at the end of this post. Everyone can try it out, test it, let us know, possible in the issue tracker.

Each updated beta will have also the translations updated, if you need to check on some strings.

I am quite confident that we can make a beta release every day from now on, if there are changes in the code (fixes) or updated translations.

The links are directly going to the apk, so this can be smoothly done from the device.

Clearly this will also need an updated user manual, which is again a big task. Brent opened a ticket where the task can be monitored in the manual's issue tracker. If you are able to help, please get in touch with us.

My wish is to get the geopaparazzi 6 release out at the "Geopaparazzi: state of the art" presentation at Foss4G 2019.



Thanks to everyone.


List of betas (pick the first to get the newest):




Thursday, May 23, 2019

A new HM tool: the las info viewer

In the open source world there are by now several tools to visualize and manage pointclouds. Over the years in our field of study (forestry) we simply found that viewing pointclouds in 3D is fancy, everyone wants it, but in the end it is mostly useless. That is clearly just my opinion, since all the tools out there have a nice 3D view of the data. Still we felt we needed something more suitable for our needs. And since the Hortonmachine already has a lot of tools to handle las data, why not aggregate them into a simple viewer that would allow us to quickly analyze a set of data, visually understand its issues, content, outlayers, intensity variations, etc etc?

Let's have a look at what we came up til now.

The viewer: once you open it, looks like this:


You can load las/laz files and set filters on any of the properties of a las point, as well as define a subregion or some subsampling. This is particularly useful when you load huge datasets as for example terrestrial scans.

First, let's load a aerial scan from our local administration:


Some header information about the file is shown as well as information about the first point of the file.
By default coloring is done based on elevation.

But one can select also intensity:


A popup on the map view shows some stats about the currently visible data. There for example we can see that intensity reaches over 5000.

Let's try to set an intensity range filter between 0 and 500. The result now is:


Hmmm, it seems we still have some "outlayers". Let's check all the points in which intensity is higher than 300 then:


It seems save to filter some more, let's say 0 to 300:


Well, better, but intensity is anyways always a mess in las data :-D

Let's have  a look at classification:


or impulses (they might be handy to understand smaller areas?):


Well, maybe we could add a DTM and have a feeling about the canopy height model (switching back to coloring by elevation)?


Hmmm, coloring looks odd, there might be some outlayers... the stats, now that the DTM is involved, also tell us the height from the ground. It seems we have trees of around 500 meters. Let's have a look where all point with height major than 80 meters are placed:


Oh, look, this seems a border problem coming from the difference with the DTM. Well, let's just get rid of them by selecting a smaller region:


and load it:


Now this looks way better. Maybe let's try to filter away the ground by thresholding the data post DTM difference at 1.3 meters:


Just know that you can export what you see as new las file or shapefile and use that for some further analyses.

But let us see an example with a more dense pointcloud, coming from a terrestrial laser scan done at the University.

Since the las file is of 3 giga, if is safe to first load it with a sumsampling of 1 point every 1000:


Then we can narrow down the area and reload the data without subsampling:


Looking at the first point table we see that the point's color information is available. Let's use that coloring:



By selecting a subregion and pushing the 3D button we can also check where the markers were placed and where we left our bag :-D


or maybe check the scan area:


It is also possible to enable the horizontal slicing mode. Once done and defined the slice interval and the width of each slice, you can load the data. This will populate the slices combobox with the different slice elevations. By selecting one, the map view will update with the slice:


Let's look for a slice where the trees are better identified:


And now push the extract circles button (mind, the tools is highly experimental and will need some love in future):


You will find that the trees extraction is not so bad (there are some false positives and negatives) and it nicely keeps the information about the radius. If your data had a proper referencing you can then export those to a shapefile.

Conclusions

This is a usual tool done without funding to help us doing our daily job. As such you have to know how to use it and maybe get used to it, else it will turn against you and make you crazy. Also it is not particularly memory saving, even if one can decide what to load into memory.

That said, we will put it in the next hortonmachine release, so if you like, enjoy it!





Monday, October 15, 2018

The Geopaparazzi family upgrades... new server and apps!

In July we released the Geopaparazzi Survey Server and it has been such a great success for ourselves, that we started to miss several features in it and had to develop them.

While doing so we noticed that the Eclipse RAP platform didn't seem to be particularly flexible and supported so we tried a switch to Vaadin... and never looked back since. The whole app has been ported and important functionalities have been added, being the 2 main ones:

The form builder

It is ages now that users ask for a visual form builder. The gvSIG Association has built one for gvSIG that works nicely. But then it still needs to be loaded to the device. Since we started to have a server, we felt this should all be there and allow for easy connection and download.

I think this first version of the form builder is already looking great and very usable:






A more detailed documentation about it can be found in the manual.


The mobile companion app is now able to get the list of forms and install them into the geopaparazzi folder:


The project data download

In order to be able to get project data from the server, it is now possible to load datasets on the server:






These dataset are then available to the mobile app:



The project data are downloaded into a dedicated folder and the geopaparazzi user needs to load them manually. So this is not so sophisticated as it is done in the profiles server.

Export database

While it is best to connect to the online database through the port 9092, which is exposed by the server, in some environments it might not be possible (ex. Heroku allows only one port). In this case it is possible to export the database by downloading it.



Send a log

The log sending has been enhanced to allow the user to add some comment that can help the developers to understand the issue:






Geopaparazzi 5.6.2 released

A new version of geopaparazzi has been released with some bugfixes, enhancements and language updates. This version now allows the user to have multiple forms json files. geopaparazzi will load them all into the add notes view.













Wednesday, July 11, 2018

Do you own a reMarkable? This could help you...

reMarkable HyUtilities is a simple application that helps the reMarkable user  mainly with 3 things that I have been missing. Maybe some of them are available to windows or apple users, but they suck for linux users.
  1. upload of templates. At every update the templates are reset to factory, so my personalized ones are gone. I needed a quick tool to restore them. ssh and scp and even FileZilla are good, but I needed something that didn't require my brain.
  2. upload of graphics. Same as for templates. I customized all possible graphics (splash, reboot, etc), so I need to restore them.
  3. backup. This is important. At the time being there is no way to the raw data. The story that pushed me to this development was a user that reported his reMarkable stolen and was afraid that the thief would delete everything from it before being able to somehow disconnect the cloud account and save the data. Since we are making big use of the device, this is a HUGE issue.

First startup


Download the application from the release page and unzip it on your disk.

To run the application launch the start.bat or start.sh file. Note that this needs java to be installed on your system.

Once launched, you should see something like:



the first time you run it you will need to insert:
  • the host, i.e. the ip address of your reMarkable device
  • the user: root
  • the password to your device
  • the folder on your disk to use as a base for your data. Notice that it is mandatory to have 3 folders already in it:
    • templates: put there all your template images, can be empty
    • graphics: put there all your custom graphics, can be empty
    • backup: can be empty

Templates


If working mode templates is selected, then the local templates should be shown.

If you put the refresh button, the remote ones will be shown.



Push the Upload local button and the app will upload your templates to the device.

Graphics

The working mode graphics works exactly the same way as the templates mode.

Backup

The backup mode is a download only mode at the moment. It enables the backup button:



If that one is pushed, the application will:
  • compress all your data on the device
  • download the tar.gz archive to the local folder
  • remove the archive from the device again
Note that the backup action can be very long going if you have many data on the device. Let the app the time to finish.

Restart device

There is also a restart device button. Once uploads have been done, it is better to restart your device. It can be done manually or through the button.

Warning

On the project page there are big warnings, please read them!

Enjoy!

Thursday, July 5, 2018

Geopaparazzi 5.6.1 is out

Usually when I have time to make bugfixes I also make a release after that. Last time I did fixes I then got caught in the work crossfire and forgot to. I didn't notice, since on my devices I have the most advanced testing devel version.

Yesterday Silvia asked me why the fix she asked for wasn't there yet! :-)
So I noticed. Well, I added a couple of more fixes to the closed list and here we go.

This is mostly a bugfix release, with a few minor usability enhancements that recently a very advanced geopaparazzi user started to report. I am not able to catch up with all his reports, but some of them are in.

Here we go:

Enhancements

  • better feedback about form name and positioning mode (GPS or Map Center) in actionbar
  • all exports now follow all the same pattern. They are exported in the geopaparazzi/export folder and their name is made if the project name + type + timestamp of export
  • project PDF export now has the possibility to export only a selected portion of notes
  • activate button for profiles is now on the main cardview
  • better proportion of forms in portrait mode
  • tile sources icon is now always visible in actionbar
  • dashboard enhancements: visualize number of notes and logs, open notes list on long tap
  • save button in forms now a floating action button to remind user to save

Fixes

  • the force locale didn't have English to allow a user to go back to default
  • fixes on profiles not being visible in landscape mode
  • fix for last background map used not reloaded on app restart
  • fix for crash on pushing back button in form notes
  • fix for crash on issue empty sql query in advanced view
  • fix for issue with saving forms with empty fields
  • avoid data loss in form notes when user exists with back button. Now the user is warned.

 And of course, language updates.

 Enjoy!

Tuesday, July 3, 2018

Geopaparazzi Survey Server

Would you believe it? Only a month ago I was showing the Geopaparazzi Profiles Server developed by the guys at GeoAnalytic and now I am here again to write about the Geopaparazzi Survey Server (GSS).

What does Survey Server even mean? :-)

Well, while the Profile concept is a sophisticated way to handle survey data, background data, forms, spatialite databases and their way of synchronization, the GSS si something much smaller. But in our opinion it reflects the workflow of many, many surveyors and groups of surveyors.




The Geopaparazzi Survey Server (GSS) is a web application that allows geopaparazzi users to synchronize their project data with a central server.

Its companion is an Android app named Geopaparazzi Survey Server Sync (GSSS) available on google play. The app can connect to geopaparazzi projects and synchronize the data contained using the unique device ID to upload the data to the server.

Any device that connects to the server with its ID, will be accepted and if not available, the new id is automatically inserted in the central db.

So this is not about spatialite datasets, but only about geopaparazzi project files, that can contain notes, complex form notes, GPS logs and images.

The server application

The server application is packaged as a docker images and can be installed in the blink of an eye ( well, if your connection is fast). Once you install it, you get a nice and simple web application with a login



a dashboard, a mapview with the possibility to load/unload the data of your surveyors, zoom to it and check information of your notes, gps logs and images:


The surveyors' device that connects to the server is inserted in the surveyors list, if it is not registered already. There a human readable name can be assigned and some basic contact:


The mobile app

The mobile app has been done for android. FYI, we are also working on a desktop version app. There are several reasons for which we decided to go with an external app instead of adding this to geopaparazzi itself. The most important thing is that several cloud synchronization applications are getting born around geopaparazzi these days. We will probably have to let these ideas mature and then at some point it will be possible to converge on the best methodology.

But right now I find it more respectful to have an external app that uses its own way to collect the data from the geopaparazzi projects and send them to the server instance.

So let's have a short look at the Geopaparazzi Survey Server Sync, GSSS :-)

Well, it is a simple simple app with the possibility to load geopaparazzi projects. In it you can see what data are available for upload, i.e. dirty data:



Since the app uses the device id as surveyor id, some simple configurations need to be done:



Once the project file is loaded, the device id is ensured and the server url is inserted... well, just push the upload button! On successful upload the list of notes will be empty and your survey can go on using geopaparazzi.

Installation and training

A complete installation and quickstart guide is available here. Check it out to see all features available.

If you want to get a small training first hand from the developers, we will be giving a workshop about geopaparazzi and GSS at the following locations in the near future:

We would also be very happy to be involved with this stack in projects in developing countries, where data collection and centralization is necessary.
If you are interested, please reach out to us.