Anatomy of the RIAWELC Dataset: A Useful Project or Pseudo-Academic Fluff?
In nondestructive testing (NDT) and weld radiography, high-quality open-access data is notoriously scarce. Consequently, the emergence of any new dataset immediately catches the engineering community's attention. In 2023, the paper "RIAWELC: A Novel Dataset of Radiographic Images for Automatic Weld Defects Classification" [1] was published. The authors introduced a public dataset consisting of 24,407 X-ray image patches, claiming to have gathered "raw industrial data from real-world manufacturing" and reporting a CNN classification accuracy exceeding 93%.
However, a rigorous engineering audit reveals a fascinating picture. First, this "novel" dataset was directly scavenged from the well-known GDXray benchmark [2]. Second—and far more dangerous in practice—even setting aside data provenance, the dataset suffers from fundamental methodological flaws that render the claimed accuracy a complete illusion. Let's break down what went wrong.
1. Data Genesis: Familiar Faces
In the paper's preamble [1], the authors rightly lament that the classic GDXray benchmark (Welds series) [2] contains only 68 images, which is insufficient for training deep CNNs directly. They promptly offer a solution: their proprietary RIAWELC dataset, supposedly captured on a production line.
Yet, comparing the sample crops presented in the paper [1] with the classic GDXray collection [2] yields instant déjà vu:
The characteristic longitudinal cracks, clustered gas pores, and even the duplex wire image quality indicator combs from BAM Berlin match down to the exact pixel. Opening the project's official GitHub repository [1] provides undeniable confirmation:
The filenames in RIAWELC feature unmistakable prefixes like bam5_... and RRT-12R_...:
bam5_Img2_A80_S1_[11][4].png
RRT-12R_Img1_A80_S1_[3][32].png
RRT-12R_Img1_A80_S1_[3][34].png
Inside GDXray itself [2], weld series are divided into directories W0001, W0002, W0003, and W0004, where files follow the pattern W0003_0001.png ... W0003_0068.png. However, sitting right inside W0003 is an internal W0003_readme.txt file documenting the exact mapping:
The TIFF files were converted to PNG format as follows (2015/Feb/11)
W0003_0001.png is RRT-09R.tif
W0003_0002.png is RRT-101R.tif
W0003_0003.png is RRT-102R.tif
W0003_0004.png is RRT-104R.tif
W0003_0005.png is RRT-105R.tif
W0003_0006.png is RRT-106R.tif
W0003_0007.png is RRT-107R.tif
W0003_0008.png is RRT-108R.tif
W0003_0009.png is RRT-10R.tif
W0003_0010.png is RRT-11R.tif
W0003_0011.png is RRT-12R.tif
W0003_0012.png is RRT-13R.tif
...
W0003_0066.png is RRT-98R.tif
W0003_0067.png is RRT-99R.tif
W0003_0068.png is bam5.tif
Thanks to this table, the correspondence is 100% conclusive: W0003_0011.png is RRT-12R.tif, while W0003_0068.png is bam5.tif. The authors of RIAWELC [1] simply took the original TIFF scans from GDXray [2] and sliced them with an 80×80 sliding window (hence the A80 suffix), retaining the source filename and appending grid coordinates [row][col].
Truth be told, had the researchers transparently stated: "We prepared a standardized, cropped patch-based benchmark derived from GDXray to spare users the effort of parsing long strip radiographs"—the engineering community would have been genuinely grateful. The real problem lies in how this data was sliced and partitioned.
2. Why This Dataset Is Unusable: 3 Critical Flaws
Flaw 1. Fatal Spatial Data Leakage
The original GDXray Welds series [2] contains only 68 long strips. How did the authors extract 24,407 patches from just 68 plates? By sliding an 80×80 window with massive overlap (a step size of mere pixels).
Then the authors committed a textbook sin of applied ML: they dumped all 24,000 cropped patches into a single pool and performed a random split across Train (65%), Validation (25%), and Test (10%) [1].
As a consequence, patch [3][32] landed in test, while adjacent overlapping crops [3][31] and [3][33], shifted by barely a millimeter and containing the exact same pore or crack tip, were assigned to train. The network evaluated on the test set is not detecting defects—it is recognizing the visual texture and baseline grain of specific physical plates seen during training. This explains the triumphant 93.3% accuracy reported in the paper [1], which will collapse to near-zero on the very first independent weld joint captured by a different radiographic system.
Flaw 2. Loss of Spatial Context (80×80 Window Size)
An industrial radiograph is not a handwritten digit. At typical scanning resolutions, an 80×80 patch covers an excessively narrow strip. It captures either an isolated section of weld bead reinforcement or a fragment of a crack without its endpoints. The model is completely deprived of spatial context: it cannot see the heat-affected zone, fusion boundaries, or bevel geometry. Distinguishing a lack-of-penetration defect from natural base metal thickness variations becomes pure guesswork.
Flaw 3. Artificial Class Balance Disregarding Real-World Topology
On real production radiographs, 90–95% of the weld area represents sound metal (No Defect). The authors densely cropped defective regions, artificially inflating the crack and porosity classes, and then leveled the distributions in their dataset summary table [1]. The model is trained in a sterile fantasy world where metal is uniformly riddled with cracks, guaranteeing an avalanche of False Positives when deployed in real production inspection.
3. How to Prepare an NDT Dataset Properly
The only viable gold standard for industrial inspection is a Group Split (partitioning by physical specimens):
- Source Isolation: An entire physical plate (source file) belongs strictly to either Train or Test. Never partition overlapping patches of the same weld across folds.
- Asymmetric Stride: A dense sliding window with overlap is permissible exclusively for Train as a form of geometric data augmentation. For Val/Test, patch extraction must be strictly non-overlapping (stride equals window size) to ensure realistic, uninflated metrics.
Example of correct specimen-based group splitting for GDXray files [2] using Python:
from pathlib import Path
import numpy as np
from sklearn.model_selection import GroupShuffleSplit
# Collect all raw radiographs (full physical plates)
raw_plates = sorted(list(Path("./gdxray_welds").glob("*.tif")))
plate_ids = [p.stem.split("_")[0] for p in raw_plates] # Grouping by physical specimen
# Honest split: each physical plate goes entirely to either train or test
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(raw_plates, groups=plate_ids))
train_plates = [raw_plates[i] for i in train_idx]
test_plates = [raw_plates[i] for i in test_idx]
print(f"Plates for training: {len(train_plates)}, Plates for testing: {len(test_plates)}")Conclusion
The RIAWELC dataset [1] is a textbook example of how chasing vanity metrics and superficial novelty undermines practical utility. Repurposing open benchmark imagery is standard practice; misrepresenting it as proprietary industrial data and introducing catastrophic spatial data leakage via random patch splitting is a methodological dead end.
For practical NDT applications, this dataset cannot be used off the shelf. The only reliable approach is to take original TIFF scans from GDXray [2], build an in-house preprocessing pipeline with strict physical specimen isolation, and rigorously validate models against joints they have never encountered before.
References and Sources
-
RIAWELC: A Novel Dataset of Radiographic Images for Automatic Weld Defects Classification
Benito Totino, Fanny Spagnolo, Stefania Perri
Dept. of Informatics, Modeling, Electronics and Systems Engineering, University of Calabria, Rende, Italy
Dept. of Mechanical, Energy and Management Engineering, University of Calabria, Rende, Italy
International Journal of Electrical and Computer Engineering Research (IJECER), Vol. 3, No. 1, pp. 13–17, 2023. DOI: 10.53375/ijecer.2023.320.
Project repository: https://github.com/stefyste/RIAWELC
📄 Download paper: RIAWELC_A_Novel_Dataset_of_Radiographic_Images_for.pdf -
GDXray: The Database of X-ray Images for Nondestructive Testing
Domingo Mery, Vladimir Riffo, Uwe Zscherpel, German Mondragón, Iván Lillo, Irene Zuccar, Hans Lobel, Miguel Carrasco
Journal of Nondestructive Evaluation, Vol. 34, No. 4, Art. 42, 2015. DOI: 10.1007/s10921-015-0315-7.
Project repository: https://github.com/computervision-xray-testing/GDXray
📄 Download paper: 2015_GDXray.pdf