Generated Assets: Underground Cable

Solution Overview

Using node-based connectivity, we modelled the electrical network as a graph, where nodes represent grid objects (such as supply points and substations), and edges represent the cables connecting them. A graph-based approach was a natural fit for this problem, as the core requirement was to trace all connected components between predefined termination nodes.

Initially, NetworkX (a Python library for the creation, manipulation, and analysis of complex networks and graphs) was used to build and traverse graphs. This worked well for validating logic on a single feeder and allowed rapid experimentation with tracing behaviour. However, this approach does not scale efficiently across the full network, where millions of nodes and connections must be processed.

To address scalability, GraphFrames was introduced. GraphFrames is a Spark-native graph processing library designed for large-scale, distributed graph analytics, making it well-suited to our data volumes. Its strength lies in efficiently identifying connected components across the entire network. However, GraphFrames offers limited built-in support for complex tracing logic, whereas NetworkX provides rich and flexible traversal capabilities.

As a result, a hybrid approach was adopted to leverage the strengths of both tools. This hybrid design combines the scalability of Spark and GraphFrames with the flexibility of NetworkX, enabling efficient, large-scale network tracing without sacrificing analytical capability. The process works as follows:

• GraphFrames is used first to identify connected components, effectively breaking the entire network into smaller, independent sub-networks.
• Using Spark’s applyInPandas functionality, each connected component is then processed independently. applyInPandas allows custom Python logic to be executed on grouped Spark data while still benefiting from distributed execution.
• Within each component, NetworkX is used to construct an in-memory graph.
• Start and end (termination) nodes are identified within that graph.
• Tracing is then performed to determine all components and paths between those termination points.