Algorithm Update: Detection and Mapping of Plastics on Beaches using MAVs
By Dr Stefan Leutenegger, Lecturer in Robotics at Imperial College London, scientific advisor to The Plastic Tide.
During the past months, I have been working with my student Ben Hahn at Imperial College London, on developing algorithms and software for The Plastic Tide to automatically detect and map plastic and marine litter on coastlines using drone imagery. Ben deserves the bulk of the credit, as he carried out most of the work described below for his final year project.
The drone images gathered by The Plastic Tide are uploaded to Zooniverse, a citizen science website where thousands of citizen science volunteers created hundreds of thousands of tags of what is and is not plastic litter. Ben used this unprecedented dataset to train the machine learning algorithm, specifically a type called a Convolutional Neural Network (CNN). The network uses these thousands of training examples of labelled plastic pieces – such that it will finally be able to tell what is a plastic piece and what not in an unseen image all by itself.
We have furthermore developed a second algorithm, it is implemented in a piece of software that refines the position (roughly known from GPS) and orientation from which the drone images were taken. This is achieved by exploiting the fact that characteristic features of the environment are observed across several images of the same beach. This image registration technique allows us to know where the drone was and what exact direction it was facing, allowing us to precisely map the detected pieces of plastic in geographic coordinates. This means that not only can we use the CCN to identify plastics; we can accurately locate the items on beaches.
Below, we show some examples of the detection that the CCN outputs – remember, it has not seen these images before, but relies on the “memory” of similar-looking tagged training images. In other words it has learned, from the citizen science inputs, where to find the plastics!
Above, a version of the network is used, that simply detects plastic pieces, without any discrimination of the exact type. Note that the results are not perfect: the network detects the same piece twice at different sizes in the left two images. It also detects something very small in the left image – which by closer inspection might not be a mistake, but an actual small piece of green-ish plastic that’s quite hard to spot even for a trained human!
The next set of test images shows a version, where the network was trained to distinguish seven different classes of plastics: plastic fragments; ropes, strings, fishing gear; plastic drink bottles; all other types of drink bottles; all other plastics or litter items; uncertain. It is certainly a harder task for the algorithm output correct predictions in this setting. In fact, we observed that people tagging the images disagree very often.
However, we are quite satisfied with these example results, given the difficult nature of the task. By closer inspection, we find a mistake in the bottom right image above: on the right, a transparent plastic part was missed. Also, the object class labels in the bottom middle image are incorrect, whereas the detections of plastic parts as such are correct.
We conducted an extensive quantitative analysis, reporting the results here would go beyond the scope of a blog post. While the algorithm on its default parameter setting, on average, outputs about twice as many erroneous or inaccurate detections as correct ones, it does accurately detect around 25% of the pieces of plastic, which we find very promising initial results given the very challenging nature of the task
In the foreseeable future, we suggest to further improve these results and extend our algorithms and software in numerous ways. First of all, we need to rigorously analyse the dataset from Zooniverse in an automated manner: we need to make sure the tagging is consistent, cross-check, filter and merge the bounding boxes and labels in each image as provided by the several annotators. We would also like to evaluate the accuracy of the “human plastic detector” – in order to have a baseline for our algorithms.
Furthermore, since this training and evaluation, the dataset has massively grown to nearly 1 million tags on many more images. We thus anticipate that using the largely increased number of training examples will further boost the accuracy of the CNN detections.
We are also very excited about further developing mapping infrastructure: it should finally allow us to exactly geo-locate and quantify the plastics in terms of number of pieces, and also generate approximate volume and mass predictions of plastics and litter on beaches!
The framework we’re working hard on we’ll share openly and hope that others will use it to detect plastics and litter themselves using their own imagery, and perhaps to improve it themselves too!
In the meantime we need your help, tag more plastics and litter here! Get started straight away, help us improve the algorithm and fight the rising tide of plastics!