Lilypad: getting the most out of remote inspection data

by Alex Haigh, Futureworx Architect
​

​

Over the last few weeks, our team here at Marshall Futureworx has been sharing stories and insights highlighting specific aspects of Lilypad, our first product. We have walked through the journey from opportunity to product; we have outlined why offshore wind is the ideal first application; and we have explained why Lilypad builds logically on the current technological and regulatory state of play in uncrewed aviation.

What we haven’t yet touched on is data—a very substantial part of the Lilypad offering. In this article, I will fill that gap by explaining how the Futureworx team is exploiting the opportunities and tackling the challenges presented by a remotely operated inspection system.

Lilypad’s approach to data acquisition and analytics

For the sake of brevity, I am assuming a passing familiarity with Lilypad’s ecosystem comprising UAVs, enclosures, remote operators, and customers (if not, our dedicated page is a good place to start). Instead, I will focus specifically on inspection data acquisition and analytics.

Let’s start with acquisition. This process starts when a permanently-deployed Lilypad inspection UAV is launched and flown by a remote pilot to a turbine to complete its automated inspection task (using GNSS and LiDAR to fly a standardised inspection route). Throughout its tasking, the UAV will hop between installed Lilypad enclosures, where it will rapidly recharge using a battery swapping mechanism.

During this short turnaround period, the inspection data gathered by the UAV is downloaded within the enclosure and securely uploaded—in the form of images and associated metadata - to the cloud via a secure SATCOM link.

The next step is analytics. The inspection image data is provided in a standardised, adaptable format, and can then be processed using state-of-the-art deep learning computer vision algorithms, which perform an initial damage classification and triage of the data, prioritising which images require further review by a human expert.

Bespoke customer inspection reports can then be created and uploaded to the customer interface portal, allowing customers to conduct their own analysis and plan any next steps.

In addition to being able to take immediate actions (such as requesting further inspections or repairs), operators can analyse large volumes of inspection data acquired over long periods, gaining valuable insights that can inform future inspection and maintenance needs.

Data analytics: providing a flexible solution

In the early stages of the project, we realised that there are already several mature and low-code/no-code computer vision solutions for damage recognition and classification in the wind turbine sector.

Furthermore, initial industry engagement taught us that different customers may have individual approaches when it comes to inspection data analytics; this could include a preferred 3rd party provider or even their own in-house analytics solution.

With this in mind we set about developing a system that outputs inspection imagery and metadata in an industry-standard format, tailorable to the customer’s data analytics tool of choice. Equally, we have opted to partner with an industry-recognised data analytics software provider.

Lilypad’s advantage: higher-quality, more useful data

AI models are only as good as the data fed into them - and this is an area where Lilypad will truly shine.

Lilypad’s UAVs reside on-site, meaning they can conduct automated inspections considerably more frequently, opportunistically and quickly than any other incumbent method of inspection.

This will drastically increase both the volume and quality of available data, ensuring that operators can develop consistent and expansive datasets in a standardised format. This, in turn, will improve the training of AI-assisted reporting and the quality of analytics that can be delivered.

More frequent data acquisition can help manage the risk of turbine damage, enabling the customer to closely monitor blade damage as it develops, rather than having to make decisions on replacing or repairing blades at yearly inspection intervals.

We also believe the ability to inspect more frequently is important for the future innovations that may be required in wind turbine design. To name one example, there are questions around how larger and further offshore wind turbines could interfere with air traffic control and military radar signals; this may drive future innovations in turbine design and/or blade materials. While these designs and materials can be tested in the lab, knowing how these features endure when deployed offshore over several years brings uncertainty. What the Lilypad system can enable is a tailored, more frequent inspection regime for future wind turbine innovations/upgrades over the life of the wind farm, enabling a quicker adoption of the latest technology.

Tackling high-volume remote data transfer

For all the reasons mentioned above (and more), the resident nature of Lilypad UAVs is a key source of strength compared to other offshore inspection methods - but it also presented a significant development challenge for the Futureworx team.

Since the system is deployed offshore and will rely on SATCOM links, transferring high volumes of data could become expensive and time-consuming. This is not a problem faced by approaches involving rope access by humans or UAVs operating in high bandwidth cellular networks.

This is where a bit of Futureworx ingenuity came into play.

We realised that there was no need to upload raw imagery from the Lilypad enclosures—in fact, only a fraction of data in each image is actually relevant to the inspection analysis process.

Working from this realisation, we set about developing an in-house tool that combines image processing software with some light AI and data processing to compress data into a manageable size and format to allow for efficient transfer via data link.

The result is a significant reduction in the size of inspection imagery, meaning wind farm operators can enjoy all the data acquisition benefits of Lilypad without feeling constrained by SATCOM data transmission times and costs.

Plenty of room for growth

There is nothing preventing the deployment of Lilypad to power data-driven analytics beyond our intended first application (offshore wind turbine blades).

In fact, Lilypad could easily be repurposed into a comprehensive platform for several offshore purposes: for example, the system can be tailored to inspect (and deliver imagery from) turbine jackets and nacelles, or other types of infrastructure such as offshore power substations.

Besides the many benefits of Lilypad highlighted by my Futureworx colleagues so far, the ability to integrate and analyse high volumes of data across multiple within a single tool will prove transformative for operators: it will help them optimise resources across for an entire wind farm (or multiple wind farms!) while yielding a new level of insight into predictive maintenance needs.
​