Transferring jobs via USB stick in TML-A

Hi, I'm Stepan. I am a Senior Software Engineer at TracMap. Today I'm gonna tell you a story about how we implemented transferring jobs via a USB stick in our TML-A.

This feature wasn't originally planned because we already transfer jobs and related data wirelessly to the online job management system. However, it turned out that some of our customers operate in areas without cellular coverage for prolonged periods of time and still need to be able to share jobs between multiple devices.

The first thing that you need to decide on when implementing an "offline" job transfer is what file format you're going to use. Ideally, that should be something standard so that the customers are able to read and write those files by external tools which gives them more power in creating and visualising jobs. In contrast, that's not always the case for the online transfer since the internals of your data transfer protocol is usually hidden from the user. Another important aspect is backward compatibility with our previous generation of products, i.e., being able to seamlessly transfer a job from an older unit to a newer one. Supporting data format generally means writing a parser for it as well as implementing a serializer in order to be able to generate a file of the given format from your device's internal data structures.

We ended up using KML data format (with .kmz extension for gzipped KML) because of its power of representing job geometry, easy-to-read XML structure, standardization and support by tools like Google Earth. It was also one of the formats supported by our previous-generation devices and the online job management system. However, for historical reasons old units supported more than one data format for their jobs so we had to implement additional parsers for SHP files and ZIP archives of SHP files as well. While parsing shapefiles can be done relatively easily by using that library, KML turned out to be trickier due to many different varieties of KML that we had to support for compatibility reasons, and eventually, we used Android's XmlPullParser for just parsing XML tags that we need.

        <Placemark type="tm:Outline">
          <ExtendedData>
            <Data name="template_type">
              <value>coverage</value>
            </Data>
            <Data name="traclink_id">
              <value>1484169</value>
            </Data>
          </ExtendedData>
          <Polygon>
            <outerBoundaryIs>
              <LinearRing>
                <coordinates>
                        170.382313531513972,
                        -45.877627654443017 170.388364595054014,
                        -45.876731326152104 170.391711991904998,
                        -45.884140539101047 170.385660928364985,
                        -45.884678266098597 170.382313531513972,
                        -45.877627654443017
                </coordinates>
              </LinearRing>
            </outerBoundaryIs>
          </Polygon>
        </Placemark>

Example of a paddock serialized to KML.

Unlike online data transfer where most of the validation is performed on the server side, offline job management requires you to put all the checks such as geometry correctness verification or data size threshold on the device side because the user can bring any kind of data on their USB stick. This process may also involve implementing some heuristics to simplify geometry and reduce the number of points and polygons whenever possible, and providing sensible defaults to values that are missed in the input file. Before finding the appropriate limits, we experienced various issues like stack overflows inside API calls to the triangulation library and SQLite commands failing due to too big rows in a table.

Once all the parsers and serializers are done, the next challenge that you need to address is supporting the actual data transfer from and to a USB stick. Although the TML-A unit is built on the Android platform, it turned out to be surprisingly tricky to implement that part of our feature. Firstly, there is only one open-source library that provides the described functionality without having to root your device. Secondly, that library only supports the FAT32 file system and is also not expected to work on oncoming versions of Android. Finally, the library does have some major bugs, one of which affected us directly, and we had to implement our own workaround for that.

Overall, this project is a good example of how the real cost of implementing a feature in terms of the actual amount of work to do can be significantly higher than it seems at the start due to a high number of issues emerging during the development process and requiring to be addressed in order for the project to be completed.