back to index

Testing the Prototype Application

Testing of the prototype application has been limited to small scale testing using the Android emulator and a small number of Huawei U8150 smartphones. In particular medium to large scale testing has proven to be beyond the scope of this exploratory research project. Testing of the application must be carried out on smartphones due to the close integration with the main Serval Project software which cannot be used within the Android emulator. It is intended that as the development of the application continues beyond the prototype stage further testing will be carried out, in particular as interest in the application grows and opportunities for testing during disaster and emergency response exercises become possible. This chapter provides an overview of two possible experiments that could be used for testing in the future.

An Orienteering Style Experiment

An orienteering style experiment is proposed to test the application in relation to the main use case that has driven development of the application up to this point. The underlying principle of the experiment is to have a number of teams explore an area looking for a series of control points. The control points are analogous with incidents that may be added to the map if the application was used during the response to a disaster or emergency event. Example control points include indications that food or water is present, that shelter is available, where a dangerous situation exists, and where people need assistance.

During the experiment the teams will need to record that they have found a particular control point. The effectiveness of the application will be determined by three main factors; the users experience in using the application, the number of control points identified, and the time taken to complete the task. Each team must visit as many control points as possible and record the team specific unique identifying mark for control point. If teams meet each other during the experiment they can share information about the location of control points that they have found. A team cannot record and share the unique team specific control point identifier for another team.

In the first phase of the experiment the teams are provided with a map, a compass and notepaper. The teams must use the map and compass to navigate the defined area and locate as many control points as possible in a set time period. The team uses the notepaper to note the team specific unique control point codes, including the time that the control point was discovered. Teams will also be encouraged to record any observations that they may have during the experiment.

In the second phase of the experiment the teams are provided with an Android powered smartphone that provides access to the standard Android mapping application. As in the first phase of the experiment the team is also provided notepaper to record their observations and the team specific unique control point codes. The team must used the standard Android mapping application to help navigate the defined area. To ensure that a teams recollection of control point placement does not effect the outcome of the experiment the control points will be relocated before the start of this phase of the experiment. As in the first phase of the experiment teams are encouraged to share information with each other if they meet.

In the third phase of the experiment the control points are relocated for a third time. This time the teams are provided with an Android powered smartphone that has the prototype mapping application installed. Each team must use the prototype application to record the location of control points that they discover including the unique team specific control point code. In this phase of the experiment the location of control points will be shared with the other teams automatically using the resilient Ad hoc mesh network provided by the main Serval Project software. Team will also be encouraged to use the voice telephony and messaging aspects of the main Serval Project software to communicate if they wish.

It is anticipated that this experiment will provide data points such as the time taken to find control points, the total number of control points found during the specified time period and if the teams completed the course. Additionally the observations made by the teams will provide insight into their experiences during the three phases of the experiment. Teams will also be encouraged to make observations about their experiences in using the prototype application to assist in determine how the application behaves during a test in the field.

It is anticipated that this experiment will show that when the users of were able to use the prototype application they were able to locate control points more effectively and that the sharing of information became easier. It is hoped that the final phase of the experiment in particular, will show that the prototype application supports an improved user experience by making it easier to collaboratively build knowledge about an area using the map on the device. The second experiment is designed to evaluate the impact that the application has on the ability for two people to coordinate their rendezvous behaviour.

Testing the Impact on Rendezvous Behaviour

The goal of the prototype application is to enhance the ability of users to collaboratively build knowledge about a particular geographic area, such as an area affected by a disaster or emergency event. By collaboratively building knowledge it is anticipated that the user of the prototype application will be better able to coordinate their activities. One such activity is the ability for users to rendezvous at a particular location. To test if the prototype application can have a positive impact on this type of behaviour an experiment similar to employed by Dearman et al. is proposed[#dearman_01].

During this experiment, participants were given a mobile device that displayed a map that was capable of showing three different types of markers. The first marker was a predetermined rendezvous point. The second marker was the geographic location of the user, and the third marker represented the geographic location of a person that they were required to rendezvous with. Experiment participants were required to participate in three different rendezvous scenarios.

In the first scenario the experiment participant was instructed to rendezvous with another person, as represented by the third marker, at a predetermined time and location. This location was displayed on the map and during the experiment participants could see their location, the rendezvous location and the location of the other person on the map.

In the second scenario the experiment participant was instructed to meet with the other person not at a predetermined time or location. When instructed the experiment participant was required to locate the other person on the map and navigate their way across the landscape to the location of the person. During this time the other person was always in motion.

In the third scenario the experiment participant was instructed to meet with the other person at a predetermined location at a specified time. Unlike the first scenario the other person appeared to get lost and so would not arrive at the predetermined location. Therefore the experiment participant would be required track the other person down when it became clear that the person would not be arriving at the specified location.

Using these scenarios it can be possible to observe how a user interacts with the map and the impact that this has on their ability to rendezvous with another person. In the case of the study undertaken by Dearman et al. it was found that:

Participant usage suggests the importance of the relation- ship between their personal location, the location of their partner, and the location of the rendezvous. Additionally, we observed a consistent trend of continual manual refinement of the visible map detail as the proximity between the participant, their partner and the rendezvous decreased. It is evident from this study that there are many different ways to use both a location-aware map application and the application specific features (zoom and pan).[p. 5][#dearman_01].

In particular it was found that users would use the map to maintain awareness of their own location relative to that of the rendezvous point and the other person. It was also found that “the majority of participants … actively refined the map as they progressed through the scenario, zooming into the map to gain greater region specific detail”[p. 6][#dearman_01].

Using these types of scenarios it is anticipated that the ability for the prototype application to enhanced the ability of users to rendezvous with each other will be evaluated. Additionally the experiment will provide an insight into the way in which users interact with the application which may suggest user interface improvements, and the need for new functionality or changes to existing functionality.

It is intended that both this experiment, and the one outlined in the previous section, will be undertaken using real Android powered smartphones using the prototype application and the main Serval Project software. As a side benefit these experiments will also provide an insight into how the resilient Ad hoc mesh network behaves in terrain such as the pine forest on the Flinders University campus. Additionally it will provide an opportunity to field test the ability for smartphones to receive GPS signals in near to real world conditions.