DataFrame™ Loader

Bulk load and integrate ILI, CIS and DA surveys to the enterprise database with powerful data alignment capabilities and flexible format support.

A common problem among pipeline operators is how to integrate survey data with the enterprise data management system. DataFrame™ Loader solves this problem by allowing the user to easily load, align, and publish inline inspection, close interval and direct assessment surveys to the enterprise database. Typically, surveys use an independent referencing system, such as an odometer or interval, which does not correspond to pipeline stationing. Before survey data is integrated with the other datasets, it should be aligned to known pipeline locations to account for the difference between stationing and odometer readings, a process called Macro Alignment. DataFrame Loader examines the control and survey data to find the best alignment to ensure a valve from the survey will be aligned to the known valve location with the same pipeline stationing. In essence, the alignment brings survey data into a single frame of reference with the enterprise data. All users of the data benefit from standardized formats and consistent storage locations, thereby improving the decision-making process. Users may also retrieve the original survey from the database if they wish to examine the details of the survey.

Survey and Assessment Loading

Surveys come from a variety of different vendors, and every vendor has a unique way of delivering the data. For this reason, DataFrame Loader is user-configurable to allow users to be able to take any survey and define how the data will be published to the enterprise database. The application gives the user the ability to bulk load survey data from raw files, including GPS files, without needing a database administrator. Furthermore, DataFrame Loader verifies the quality of the data against the database and will produce an error log if any issues are found. Once the data has been published, defect characteristics can be analyzed much more thoroughly because the anomalies have been integrated with the other enterprise datasets.

Pipeline Feature Bulk Loading

DataFrame Loader offers the capabilities to bulk load external facility data to the pipeline database. Furthermore, the flexibility of the application reduces the time needed to format and cleanse the data before it is loaded. You can quickly load an unlimited amount of data without the preprocessing and time historically required to get new data into the system.

Intelligent Data Alignment

Integrating inline inspection, close interval survey and direct assessment data into the enterprise database is made simple with intelligent data alignment. In addition to macro alignment, powerful new micro alignment capabilities of the updated feature matching interface allows alignment of new surveys to historical runs at a very precise weld-to-weld level. The user defines the alignment parameters for the control and survey data, then DataFrame Loader uses a DNA matching process to identify likely matches between the two datasets, streamlining and automating the alignment process; however, the user remains in complete control of the alignment with the ability to manually match and separate aligned pairs.

Data Validation

Detailed data validation controls the quality of the data being loaded, helping ensure that imperfect data is not introduced to the database. As the data is being published, database code and stationing values are verified to ensure the published data is accurate and consistent. If an error is found, a detailed error log is created to describe the error.

Data Mapping Interface

The data-mapping interface provides robust data mapping options to describe data translation and data conversion. These features greatly reduce the time needed to format and cleanse the preloaded data.