This video shows how the InterFolia application works. On the main screen (0:00 to 0:04), we can find the different menus available. In particular those for classification (multi-organ and individual) and for look over the saved classifications.
From 0:04 to 0:48, we go through a classification performed with a simple organ (leaf). A photo is imported from the phone’s memory before being classified by an embedded deep neural network. The results view shows the top-5 best results (0:14). It is possible to save the desired species with the help of the description images (0:21 to 0:35).
From 0:42 to 1:35, we see the multi-organ classification process. It is similar to the single-organ classification, but it is possible to load a picture for each available organ. The results are then merged to obtain the top-5 best species. For all species in the list, the user has images from botanical book to help to recognize.
Finally, it is possible to find previously saved species through the ‘Local Library’ menu (1:40 to 2:08). The species are sorted by date of recording. Each species is saved with the image, or images, used for classification and is associated with a textual and image description. These descriptions provide additional information about the plant.
The urbanization of society has gradually separated humans from the plant world. For most people, botany remains difficult to understand. It is not easy to decrypt botanical literature because it requires solid theoretical background. In the ReVeRIES project (French acronym that means “dreams” and that stands for Educational, Interactive and Fun Plant Recognition on Smartphones), we propose to use mobile technologies in order to help humans recognize plants that surround them.
First of all, we intend to design mobile learning games that will help users learn about plant characteristics and especially learn the methods, used by expert botanists, to recognize plant families, genera and species. In order to motivate children and botanical neophytes to learn about plants and explore their natural environment, we also intend to use game mechanics for creating fun activities based on plant recognition. The users will be able to improve their skills by comparing their results to those found by the recognition algorithm.
Concerning the image recognition algorithm, we intend to extend the previous prototype to the main exotic woody trees and shrubs. Moreover, we aim to take into account various organs of the plant. This multimodality is essential if we want users to learn and practice the correct recognition method, for which botanists use a variety of organs (i.e. leaf, bark, size of plant, flower, fruit). In addition, the use of organs should greatly improve the algorithm’s accuracy. In terms of image processing, the work done on the leaves cannot be extended directly to flowers, fruits and barks. This will greatly increase the complexity of the data fusion process.
Finally, in order to enhance social awareness of our natural resources, we intend to explore ways to support citizen science. The geolocated photos and information, taken with the application and validated by experts, could be transferred to specialized networks, such as Tela Botanica, integrated into the OpenStreetMap geographic information system and mobilized by local institutions to support actions and projects involving citizens. This addresses problems related to the field of Volunteered Geographical Information such as coordinating the data provided by open contribution with institutional data, collected and maintained by official agencies.
Here is the dataset that we used to evaluate InterFolia (CVPPA
2021).
Download barks
Download fruits/flowers
Download leaves
This application has been developped in ReveRIES project context funded by ANR (ANR-15-CE38-0004).
For more information, please contact Laure Tougne (*) or Carlos Fernando Crispim Junior (*) at LIRIS.