Thursday, May 21, 2020

SMASH 1.2.0 is out... and it is packed with goodies!

SMASH 1.2.0 is out in the google and apple stores and it comes packed with a whole set of new features. Not even sure where to start. Here we go...

Welcome and Project View

We never had it, but it was time.When SMASH starts up it need to check some permissions, layers, data, forms and much more. Now you will be notified and see something is going on:


After that the projects view comes. Make your choice, new/old or fast choice between recent?



Data formats

Some new formats are available now.

Raster files in World image format

SMASH now supports loading of raster imagery in the tiff/jpg/png + worldfile (tfw/jgw/pgw) + projection file.

So once you load the rasters in the layer view, it will check the projection.

In this case we have 3 different projections and while 4326 and 3857 are supported. So the first time this happens you will need to tap on the layer to try to download the proj definition for the projection.
If you are lucky enough, the epsg was recognized and the definition downloaded. Now the dataset is loaded. Mind that while on vector data a re-projection is really performed, in case of raster images, the images is just warped using the boundaries. So this could lead to some unwanted distortions. So always bet to use 4326 data for rasters.
The result is quite nice and transparency can be applied where needed.
Here below a jpg loaded in 32632:


 and a tiff loaded in 3857:


Small confession: this is the reason there has been also a version 1.1.0. We found out there were some issues with the projection of the raster imagery files and decided to fix them an release right away.

Shapefile read support

SMASH now supports shapefiles ( outch, who would want them?). 



And they are actually styled with SLD:


And can be queried:


They are and will always be (unless someone pays tons of money) read-only layers. All the editing will be moved to geopackage in future.

WMS/TMS support

TMS has already been supported, now WMS is also here. And there is a beautiful catalog to hold them:





The maps are actually live maps which help to see if a service works even before adding it.

WMS looks the same and both can export and import the definitions:


To add a new definition, since that is a rude thing to do on the device, we tried to make it as comfortable as possible:


Layer management

Layers can now be dragged to reorder them and they can be removed with a right-to-left swipe.
If they support zooming to or styling, it is found in a right-to-left swipe.


Note that this is now harmonized in all list views with zoom and style.

Notes:


and logs:




GPS 

The biggest new is for sure the fact that SMASH now supports background logging, i.e. logging with the screen off. No image here, but you know what that means.

We also added to the settings view:



a GPS live view to be able to play with the settings and see what happens:



As you can see the rose of incoming points is visible in the background, so give an idea of what is going on.



Last but not least, it is possible to activate a counter for GPS points (all vs filtered) to be seen on the main view in the GPS icon:



Other notable things

Debug view

We added a debug view, accessible from the settings:


Better drawers layout

The left and right drawer and the actionbar have been reviewed.
The right drawer is now accessible by icon and contains all the tools, in expandable sumenus.


 As a result the left drawer is now more clean and project related:


And an online help has been added, which is never bad :-)

The GPS info panel has been cleaned up a bit and one can copy each coordinate to the clipboard:




 Notes have labels

We added labels also to project notes:


Grid 

A nice grid plugin has been added to flutter map, so we added it here also:

 

 Reprojection support

As mentioned in the raster imagery part, support has been added for projections. If a projection is not known, it will be required to download its definition. After that vector data will be reprojected properly. This part is still a bit experimental. But it already looks good.

 

Small things

  • sharing the position has now the Openstreetmap url
  • fixes have been done in the forms engine

 Guess that is all! Enjoy!

PS: as usual the user manual is lagging behind a version. Is there some gentle soul out there that would be willing to help?

Monday, May 18, 2020

Geopaparazzi vs. ESRI Collector - a user story

Andrew Mitchell appeared a couple of years ago in the geopaparazzi community and soon marked a differenece. It is one of those users that put effort in making the tool they use better. He soon started to open issues about usability... many issues. And those many issues made me crazy, since he was right and most of the time I had to react with improvements and fixes, but also contributed to a better geopaparazzi.

A couple of days ago he shared his experience on the field with Geopaparazzi and with ESRI Collector. Needless to say that I loved the read.

Pasting the whole content here as it was sent to the geopaparazzi mailinglist:






ESRI Collector vs Geopaparazzi
Background Information
Most environmental consulting firms use Collector. It is the defacto industry standard for GIS data collection. However, I have on most occasions had serious issues with it and now use Geopaparazzi. The following table provides a comparison between the two systems based on slightly more that a years of practice with both systems. In most cases, I would start with Collector and then switch over to Geoparazzi when Collector experienced an issue that was hard to work around. That said, others found Collector to be fine and stable. It could be that the iPhone version is better than the Android version. Even so, many limitations would remain. This review concentrated on issues that have caused grief in the field, rather than what works well.
Geopaparazzi has fewer issues, but some occasionally arise, particularly teething issues with the new version 6. However, any serious issues have rapidly been resolved. At the time of writing Geopaparazzi 6.0.3 was being compared with an up-to-date, professionally managed ESRI ecosystem. 

Collector
Geopaparazzi
Primarily an online data collection system with an off-line capability
Began as an off-line system but has ability to utilise some cloud services for importing base data and form designs. Does not appear to have the ability to upload new or updated data to a live GIS system.
Projects are created in ArcGIS Pro then delivered to an Android device by cellular network
- downloads can be painfully slow
- it is also confusing when you open a downloaded project and are outside the project area as the imagery for your current location will be shown, not the imagery for the defined project area.

Project usually created on the fly on the device by starting a new project.
- loading base maps or spatialite data via USB is very simple and quick.
- it is simple to centre the view on a base map.
Background Images
Background images are limited to those hosted by ESRI online.
Imagery that can be displayed in ArcGIS Pro is not able to be (we can’t figure out how) be displayed as a background image on the mobile device. This prevents recent, high quality imagery such as nearmap from being available in the field.
Spatial misalignments of background imagery from ESRI cannot be corrected.
Potentially imagery is not preserved when Collector is used in online mode when the device screen is turned off. More work is
Background images can be downloaded via cellular network on the fly or you can manually pre-load the area of interest by dragging the screen over the area of interest. This method is very good for linear areas such as under an aircraft flight path. Background imagery persists between sessions.
Background images can also be created in a GIS as an mbtiles file. If the original image is a from a web server such as Open Street Map, then each image level will have a different amount of detail. This is very good for navigation to site. Zoom dependent visibility can also be used in a GIS to show
needed to test this but it seems that the same tiles are downloaded each time they are needed.
If the internet is congested, background imagery will display slowly or a network error will be reported.
It is possible to create tile package files (ptk files) which contain background imagery for Collector, however an add-in is required for ArcMap. These images can then be copied into the device over a USB connection. This capability has not been tried by us.
or hide project layers and this information will be burned into the mbtiles when they are created.
Any imagery that can be displayed in a GIS can be converted to mbtiles. I often use a hillshade blended with an aerial photo.
If the webserver imagery is not located correctly, it is possible to create mbtiles with the imagery shifted to its correct location.
Navigation
North up display
Heads up display or north up display – user selects
Data collection
Both are similar. Both open the keyboard for pick list entries. This is a pain as the keyboard takes up space that could be used to show more picklist entries.
Drawing lines and polygon features in collector is very easy and very refined
Tracks
No tracks by default, although there is a workaround.
Records tracks when track logging is activated. Basis of track point recording can be adjusted in settings
Data management
In online mode, no data management is necessary.
In off-line mode, data has to be synced. Sometimes there are issues with syncing.
Uploaded data is available to most users via Arc Maps Online and to the GIS team via ArcGIS Pro.
Retrieving the data for third party applications requires extra steps such as exporting the data to a folder that is reachable by the applications. The data cannot just be copied out of ArcMaps online. The ability to select rows is disabled. Copying the feature database from the server to a local machine also did
There is some cloud support, but I have not used it. It requires setting up a Postgres Server and some configuration.
I just copy the gpap file from the device and paste it into a folder on my windows PC. A micro USB cable is required for the physical connection.
Prior to copying the gpap file, I may choose to export the photos and gps tracks, which are stored within the gpap database.
Managing this data requires some procedures and an organised filing system but the data can be opened and processed by any suitable application.
not make the data available. The database claims to be empty on local machine but has data on the source machine.
Viewing form data
Form data can be viewed by clicking on points on the map in the field or by clicking on points displayed in ArcOnline. A mini- web page is opened showing the collected information.
Data is collected in JSON format, with the JSON form specification also saved in each record. It is a bit like pdf forms, where the data is saved inside the pdf document and has to be extracted later.
This would be a pain if you do not run a system that can do this (gv-SIG) and are not able to write your own.
The great advantage of this approach is that form design can be evolved in the field. Data already collected is totally unaffected.
Extracted data can be converted to a GIS data file such as a shapefile, just added to a database table or rendered as html or kml.
Usability
In Collector, it is easy to press the back button too many times and to quit the project and even the application by accident.
Issues can occur if a data point is right on the edge of the defined project area. It is not-possible/very difficult as Collector resists the edge of the project area being dragged into the centre of the screen. This is an issue if you have to approach the target via areas that are outside the project area.
Symbol size in our projects is very small, usually round dots. If the colours used are similar, it is hard to tell what the colour is. Also if the coloured dot is used on a strongly coloured background, our eyes will colour balance and the perceived colour will shift.
Screens are prone to clutter. If contour lines are used, then seeing the data can be like looking through spiders web.
In Geopaparazzi, accidentally quiting the current project or the application is very unlikely. Quiting has to be deliberate.
Screen position is not limited to the project area.
Symbol size, colour and shape is under user control.
Screen clutter is manageable. Layers that are only used for navigation can be burned into the background layer.
Multi-user/multi-session experience
This is Collectors’ strongest point. When a
Geopaparazzi is not well suited to live
project is synced, data collected or updated by other people becomes visible. For the field operator, responsibility for managing the data ends as soon as the data is synced. If the data fails to sync, and this can happen, then the data is lost (iPads and Collector!).
ArcPro is required to make Collector projects so GIS support is required.
updates in the field as far as I know.
It is possible to create a new project everyday and to import data from that project into a GIS. I have written an application that can append data from incoming gpap files into a master database, however as far as I know, the Geopaparazzi ecosystem otherwise lacks this capability.
Being able to start a new project anywhere at anytime provide great flexibility.


Thursday, April 2, 2020

SMASH 1.0.0 is out! Your Geopaparazzi for IPhone and IPad

It is a while now that we have been talking about SMASH and its first testversions.

Back in this presentation at Foss4G Bucharest we were showing first results already.

Well, now it is finally out in version 1. And here is what it looks like...

on IPad:


on IPhone

and on Android


SMASH supports a subset of features that Geopaparazzi has, but adds (apart of IOS support, which is huge) much much usability and is way userfriendly.

Forms are compatible with those of geopaparazzi, but also support icons:


...also  in the form items.


SMASH supports various formats as for example, Geopackage (vector and tiles), MBTiles, Mapsforge, GPX and online TMS services:


Vector layers can be queried using a configurable tap radius:



SMASH also synchronizes with the Geopaparazzi Survey Server to gather project data:


and to incrementally upload work done:







SMASH finds its place on the geopaparazzi website, which has been reviewed to showcase Geopaparazzi, SMASH and the GSS. It also features a comparison table.

So run to http://www.geopaparazzi.org for more general info and here to see SMASH directly!







Sunday, December 15, 2019

Geopaparazzi 6.0.1 released - geopackage support and labels

Well, the new geopaparazzi release had been requested by users that were not able to load spatialite tables that had > 3 dimensions. Indeed there was a bug that was giving that problem.

So we fixed the bug and since we had some quite tested code about Geopackage at hand, we tried to insert it and use it the same way we use Spatialite. It proved to be fairly easy and so we decided to bring out this version with the following support:

  • vectors (feature) layers: read, write (speak editing) and style mode, but for 4326 srid only (so that no reprojection is necessary)
  • tiles: but only for 3857 srid (so they fit in the current tiling system)



I personally think it is quite awesome. Geopackage is more limited than spatialite, but its data preparation tools are way more stable and spread around GIS environments. QGIS even uses it as its default vector format.

Also in the hortomachine project we have tools to quickly create geopackage databases, add shapefiles to them, tilesets from raster files and also to style the layers to be geopaparazzi ready.









One other thing we fixed is vector layer labelling. Many users have been crying about missing vector labels. Well, it has been really fun to see how many people and working groups of people use geopaparazzi with spatialite for their surveys. I really wish they would also contribute from time to time.

Anyways, we decided to add back vector labels. Given the limitations
imposed by the rendering framework, which would make things very
difficult in 3D space and add a layer to be handled for data layer, this is how we solved it, blocking the visualization of labels in the 2D space:






Might seem strange at first test, but I can assure you it works quite nicely.

There is also collision handling on a per layer  basis. This might show it better:








Enjoy!!






Monday, November 18, 2019

Geopaparazzi 6 is out. Now or never!

It has been an infinite time from the last geopaparazzi release. I only figured when I did the release. With all the work around the Geopaparazzi Survey Server and SMASH the main kid has been left behind.

So today it was the now or never. We are throwing Geopaparazzi out to the market. It will no doubt be the most important release of the year.

The new features can be found in this presentation:
https://www.slideshare.net/moovida/geopaparazzi-state-of-the-art-of-the-digital-field-mapping-application-167930916

Many thanks to several people that helped on the manuals, with fixes and features, that made descriptive and useful bug reports. Brent, Eli, Peter, Tim, Andrew, thanks for the support.

Things will still get quite crazy from here on. We are almost ready to add support for geopackage (also in editing mode... I hope), which is getting more and more used.

So enjoy. Test, Report issues. Enjoy!

Thursday, August 8, 2019

Geopaparazzi 6 betas - testers wanted

Dear all,
as you might have noticed, the upcoming geopaparazzi release will introduce huge changes due to the migration to the graphically accelerated VTM engine in substitution of the old version of mapsforge we were using and that was getting unusable due to https online resources policies on Android.


Also this version of geopaparazzi contains a sparkling new build of spatialite based on proj 6 and is available for 64bit devices. So this also will need some review to make sure it works on most devices.

All this allowed us to have several great functionalities as for example a 2.5 view


or the possibility to have multiple layers loaded (ex. mbtiles on top of online maps). Also this will soon open the possibility for vector tile layers, which is a huge improvement on the long run.

Being that many changes in the game, Brent Fraser suggested to make the beta release available for community testing and I agree with him.

The first beta release has been uploaded and a link will be placed and updated at the end of this post. Everyone can try it out, test it, let us know, possible in the issue tracker.

Each updated beta will have also the translations updated, if you need to check on some strings.

I am quite confident that we can make a beta release every day from now on, if there are changes in the code (fixes) or updated translations.

The links are directly going to the apk, so this can be smoothly done from the device.

Clearly this will also need an updated user manual, which is again a big task. Brent opened a ticket where the task can be monitored in the manual's issue tracker. If you are able to help, please get in touch with us.

My wish is to get the geopaparazzi 6 release out at the "Geopaparazzi: state of the art" presentation at Foss4G 2019.



Thanks to everyone.


List of betas (pick the first to get the newest):




Thursday, May 23, 2019

A new HM tool: the las info viewer

In the open source world there are by now several tools to visualize and manage pointclouds. Over the years in our field of study (forestry) we simply found that viewing pointclouds in 3D is fancy, everyone wants it, but in the end it is mostly useless. That is clearly just my opinion, since all the tools out there have a nice 3D view of the data. Still we felt we needed something more suitable for our needs. And since the Hortonmachine already has a lot of tools to handle las data, why not aggregate them into a simple viewer that would allow us to quickly analyze a set of data, visually understand its issues, content, outlayers, intensity variations, etc etc?

Let's have a look at what we came up til now.

The viewer: once you open it, looks like this:


You can load las/laz files and set filters on any of the properties of a las point, as well as define a subregion or some subsampling. This is particularly useful when you load huge datasets as for example terrestrial scans.

First, let's load a aerial scan from our local administration:


Some header information about the file is shown as well as information about the first point of the file.
By default coloring is done based on elevation.

But one can select also intensity:


A popup on the map view shows some stats about the currently visible data. There for example we can see that intensity reaches over 5000.

Let's try to set an intensity range filter between 0 and 500. The result now is:


Hmmm, it seems we still have some "outlayers". Let's check all the points in which intensity is higher than 300 then:


It seems save to filter some more, let's say 0 to 300:


Well, better, but intensity is anyways always a mess in las data :-D

Let's have  a look at classification:


or impulses (they might be handy to understand smaller areas?):


Well, maybe we could add a DTM and have a feeling about the canopy height model (switching back to coloring by elevation)?


Hmmm, coloring looks odd, there might be some outlayers... the stats, now that the DTM is involved, also tell us the height from the ground. It seems we have trees of around 500 meters. Let's have a look where all point with height major than 80 meters are placed:


Oh, look, this seems a border problem coming from the difference with the DTM. Well, let's just get rid of them by selecting a smaller region:


and load it:


Now this looks way better. Maybe let's try to filter away the ground by thresholding the data post DTM difference at 1.3 meters:


Just know that you can export what you see as new las file or shapefile and use that for some further analyses.

But let us see an example with a more dense pointcloud, coming from a terrestrial laser scan done at the University.

Since the las file is of 3 giga, if is safe to first load it with a sumsampling of 1 point every 1000:


Then we can narrow down the area and reload the data without subsampling:


Looking at the first point table we see that the point's color information is available. Let's use that coloring:



By selecting a subregion and pushing the 3D button we can also check where the markers were placed and where we left our bag :-D


or maybe check the scan area:


It is also possible to enable the horizontal slicing mode. Once done and defined the slice interval and the width of each slice, you can load the data. This will populate the slices combobox with the different slice elevations. By selecting one, the map view will update with the slice:


Let's look for a slice where the trees are better identified:


And now push the extract circles button (mind, the tools is highly experimental and will need some love in future):


You will find that the trees extraction is not so bad (there are some false positives and negatives) and it nicely keeps the information about the radius. If your data had a proper referencing you can then export those to a shapefile.

Conclusions

This is a usual tool done without funding to help us doing our daily job. As such you have to know how to use it and maybe get used to it, else it will turn against you and make you crazy. Also it is not particularly memory saving, even if one can decide what to load into memory.

That said, we will put it in the next hortonmachine release, so if you like, enjoy it!