RAMADDA, take two: browsing the tree

Assume you have a Ramadda server with some groups created and some data uploaded, something that look like:



Want to access it programmatically and browse it? Here is the code to do so, shown in an example that traverses the repository tree and prints out all of the entries.


The main dump method:

    /**
     * Dumps the tree of the repository.
     * 
     * @throws Exception
     */
    public void dumpTree() throws Exception {
        List postEntries = new ArrayList();
        postEntries.add(HttpFormEntry.hidden(ARG_SESSIONID, repositoryClient.getSessionId()));
        postEntries.add(HttpFormEntry.hidden(ARG_OUTPUT, "xml.xml"));
        String[] result = repositoryClient.doPost(repositoryClient.URL_ENTRY_SHOW, postEntries);
        if (result[0] != null) {
            System.err.println("Error:" + result[0]);
        }
        outputStream.println(result[1]);
        Element response = XmlUtil.getRoot(result[1]);

        // the root id
        String entryId = response.getAttribute("id");
        ClientEntry rootEntry = repositoryClient.getEntry(entryId);
        outputStream.println(entryToString(rootEntry));

        dumpRecursive(rootEntry, TAB);
    }

The recursive method and the toString method:

    /**
     * Recursively traverses the entries and dumps its childs and subchilds.
     * 
     * @param entry the start entry.
     * @param tab the tabulator characters to use.
     * @throws Exception
     */
    private void dumpRecursive( ClientEntry entry, String tab ) throws Exception {
        List childEntries = getChildEntries(entry.getId());
        for( int i = 0; i < childEntries.size(); i++ ) {
            ClientEntry childEntry = childEntries.get(i);
            outputStream.println(tab + entryToString(childEntry));
            dumpRecursive(childEntry, tab + tab);
        }
    }

    /**
     * Extract some base information from the {@link ClientEntry entry}.
     * 
     * @param entry the entry.
     * @return the string representation.
     */
    public String entryToString( ClientEntry entry ) {
        String name = entry.getName();
        String id = entry.getId();
        Resource resource = entry.getResource();
        String type = resource.getType();
        return name + " ( id=" + id + ", type=" + type + ")";
    }

And here the most important method, that gets childs from entries:

    /**
     * Creates a {@link List} of {@link ClientEntry}s for a particular entry.
     * 
     * @param parentId the id of the parent entry.
     * @return the lsit of child entries.
     * @throws Exception
     */
    public List getChildEntries( String parentId ) throws Exception {
        String[] args = new String[]{ARG_ENTRYID, parentId, ARG_OUTPUT, "xml.xml", ARG_SESSIONID,
                repositoryClient.getSessionId()};
        String url = HtmlUtil.url(repositoryClient.URL_ENTRY_SHOW.getFullUrl(), args);
        String xml = IOUtil.readContents(url, getClass());
        Element root = XmlUtil.getRoot(xml);

        List childEntries = new ArrayList();
        List children = XmlUtil.getElements(root, "entry");
        for( int i = 0; i < children.size(); i++ ) {
            Object object = children.get(i);
            String id = XmlUtil.getAttribute((Node) object, "id");
            ClientEntry entry = repositoryClient.getEntry(id);
            childEntries.add(entry);
        }
        return childEntries;
    }

The result launch on the above repository is:

morpheo ( id=02c47d07-6f7d-473b-b104-b922348f7d51, type=unknown)
   documents ( id=660cb2ce-99f5-4d4b-9682-2f408c2b105e, type=unknown)
      client code ( id=9977cbc1-2589-49c8-b7e0-9b944970d169, type=storedfile)
      call_4_papers.odt ( id=33a026f4-87f3-4c78-8c1d-d14da597aa06, type=storedfile)
      rasterlite-how-to ( id=0a138888-b5ee-42bf-be3e-f78c8c3a5232, type=storedfile)
      hydrologis logo ( id=209c182d-e22c-48b2-933a-8d03ab4b72d9, type=storedfile)
   netcdf ( id=91b9a338-d68e-4411-bdd3-b3e5a3f111c6, type=unknown)
      cami_0000.nc ( id=3a74f349-a4bf-4f9d-916d-83f4511e0877, type=localfile)
      water.nc ( id=400a17cc-f33c-40f4-a896-b2bffbed2f84, type=storedfile)
      water_surf.nc ( id=6b69040b-7a96-46dc-95b2-0d6ae044f948, type=storedfile)
   water_surf.nc ( id=eded7693-8b6e-4941-87f5-2fc9bb2a6bb7, type=storedfile)

RAMADDA

You might ask yourself what ramadda is, and I can tell you that the Repository for Archiving, Managing and Accessing Diverse DAta is awsome!

It is really a while that I search for a way to store diverse data, most of them grids, in a repository where the metadat would be preserved and in case even editable. A place where one could extract subsets of data from the datasets. A place where one could access the data also directly from the GIS... yes, I know, everything is thinking about OGC and some WCS and whatever else.
But the summer of code opened me a pandora box full of presents and by choosing the necdf format many possibilities built on open sourced software are available.

Ramadda can be tested on the unidata's demo server, so I won't talk about the many features that one can try out there.

I want to talk about background jobs that can be done from the client code of Ramadda, which Mr. Jeff McWhirter was so kind to introduce me to and help me with.

First, to get started, you need the repository client library, which you can download here. Once the contained libraries are in the classpath, you can start.

Let us assume we start from an existing ramadda repository, that looks like the following:



Let's assume we need to upload a netcdf file that has been produced in JGrass by some model.
The following shows the code needed to do so.

First a client connection has to be instantiated:

RepositoryClient repositoryClient = new RepositoryClient(host, Integer.parseInt(port), base, user, pass);
String[] msg = {""};
if (!repositoryClient.isValidSession(true, msg)) {
  // throw some exception
}

where the isValidSession method in this case also performs a login in the ramadda environment.

Once the connection is done, the upload of any file can be done. The following method does that for you:

    /**
     * Uploads a file to ramadda.
     * 
     * @param entryName name of the entry that will appear in the ramadda server.
     * @param entryDescription a description of the entry that will appear in the ramadda server.
     * @param type the file type that is uploaded.
     * @param parent the parent path inside the repository, into which to upload the file 
     *              ex. morpheo/documents, where morpheo is the base group and documents 
     *              the subgroup.
     * @param filePath the path to the file to upload.
     * @throws Exception
     */
    public String uploadFile( String entryName, String entryDescription, String type,
            String parent, String filePath ) throws Exception {

        Document doc = XmlUtil.makeDocument();
        Element root = XmlUtil.create(doc, TAG_ENTRIES, null, new String[]{});
        Element entryNode = XmlUtil.create(doc, TAG_ENTRY, root, new String[]{});

        /*
         * name
         */
        entryNode.setAttribute(ATTR_NAME, entryName);
        /*
         * description
         */
        Element descNode = XmlUtil.create(doc, TAG_DESCRIPTION, entryNode);
        descNode.appendChild(XmlUtil.makeCDataNode(doc, entryDescription, false));
        /*
         * type
         */
        if (type != null) {
            entryNode.setAttribute(ATTR_TYPE, type);
        }
        /*
         * parent
         */
        entryNode.setAttribute(ATTR_PARENT, parent);
        /*
         * file
         */
        File file = new File(filePath);
        entryNode.setAttribute(ATTR_FILE, IOUtil.getFileTail(filePath));
        /*
         * addmetadata
         */
        entryNode.setAttribute(ATTR_ADDMETADATA, "true");

        ByteArrayOutputStream bos = null;
        ZipOutputStream zos = null;
        try {
            bos = new ByteArrayOutputStream();
            zos = new ZipOutputStream(bos);
            /*
             * write the xml definition into the zip file
             */
            String xml = XmlUtil.toString(root);
            zos.putNextEntry(new ZipEntry("entries.xml"));
            byte[] bytes = xml.getBytes();
            zos.write(bytes, 0, bytes.length);
            zos.closeEntry();
            /*
             * add also the file
             */
            String file2string = file.toString();
            zos.putNextEntry(new ZipEntry(IOUtil.getFileTail(file2string)));
            bytes = IOUtil.readBytes(new FileInputStream(file));
            zos.write(bytes, 0, bytes.length);
            zos.closeEntry();
        } finally {
            zos.close();
            bos.close();
        }

        List postEntries = new ArrayList();
        postEntries.add(HttpFormEntry.hidden(ARG_SESSIONID, repositoryClient.getSessionId()));
        postEntries.add(HttpFormEntry.hidden(ARG_RESPONSE, RESPONSE_XML));
        postEntries.add(new HttpFormEntry(ARG_FILE, "entries.zip", bos.toByteArray()));

        RequestUrl URL_ENTRY_XMLCREATE = new RequestUrl(repositoryClient, "/entry/xmlcreate");
        String[] result = repositoryClient.doPost(URL_ENTRY_XMLCREATE, postEntries);
        if (result[0] != null) {
            outputStream.println("Error:" + result[0]);
            return null;
        }

        Element response = XmlUtil.getRoot(result[1]);
        String body = XmlUtil.getChildText(response).trim();
        if (repositoryClient.responseOk(response)) {
            outputStream.println("OK:" + body);
        } else {
            outputStream.println("Error:" + body);
        }

        Element child = XmlUtil.findChild(response, "entry");
        String entryId = child.getAttribute("id");
        String urlString = "http://" + host + ":" + port + base + "/entry/get/" + entryName
                + "?entryid=" + entryId;
        return urlString;
    }

This method returns a url string, that can be used in the browser to fetch the uploaded dataset.

So if I was to upload a netcdf called water_surf.nc without supplying a particular path, it would have resulted in appearing in the base group, called morpheo in this case:



What I really love about ramadda, is that if the dataset is in netcdf format for example, the metadata are accessible and editable also from the web interface:



There are several ways the data can be accesses, and note in the below picture the opendap link, which is the one the JGrass uses to visualize the dataset or use it in the models:



Ok, but I was writing about programmatically access the data. So how to fetch data from the server? They can be downloaded from ramadda by means of their id. In fact the last part of the above returned url string represents the unique id of the dataset (...?entryid=THEENTRYID).

Downloading a file from ramadda is incredibly easy, since the RepositoryClient class sopplies a method called writeFile that takes the entry id and the output file to which to download as parameters:

repositoryClient.writeFile(entryId, outputFile)

And that is all, thanks to Jeff McWhirter for all the great help he gave me.

Soon I will be glad to describe some deeper integration between JGrass modeling environment and Ramadda.