ALICE TPC — Automated Quality-Control Data Pipeline

A write-up of work I did as a data-engineering intern on the ALICE
experiment at the GSI Helmholtz Centre for Heavy Ion Research (Feb–Jun
2026), building the automation that turns raw detector data from CERN into
quality-control plots for the Time Projection Chamber (TPC).

Note on the code. This repository is a technical write-up. The full
production scripts run against GSI/ALICE infrastructure and the ALICE grid;
they are shared here only in sanitized, illustrative form. Anything
collaboration-internal is described rather than dumped.

The problem

The TPC is ALICE's main tracking detector. During data-taking it emits
time-series files summarising detector behaviour (drift, gain, track
quality) over time. Two things had to happen, reliably and unattended:

1. Get the data from CERN to GSI. New time-series files are produced on the
   CERN grid; physicists at GSI need them mirrored onto the local Lustre HPC
   cluster, continuously and without duplication or corruption.
2. Turn the data into quality-control plots. Tens of terabytes of detector
   data have to be reduced into a compact set of QC plots (per readout sector
   and over time) that a human can scan to spot problems.

I built the automation for both halves.

Part 1 — CERN ⇄ GSI synchronization pipeline

Stack: Bash, the ALICE AliEn grid tools (alien.py, alien_cp), ROOT,
GNU xargs for parallelism, run on a cron schedule.

A single daily entry point refreshes the GSI mirror. The pipeline:

• Discovers what's missing by listing the relevant grid paths per LHC
  period and diffing them against what already exists on Lustre — so only new
  files are ever copied (idempotent re-runs are cheap).
• Copies in parallel with 16 concurrent alien_cp workers driven by
  xargs -P, with a configurable degree of parallelism.
• Validates every file three ways before accepting it:
  1. transfer succeeded and the target exists,
  2. the local size is within a tolerance of the grid size, and
  3. the ROOT file actually opens, contains the expected tree, and is not a
     "zombie" / partially-written file.
• Self-heals: failed copies are retried up to 3×; corrupted files are
  deleted and re-queued; a per-period retry list is kept for anything still
  failing.
• Logs safely under concurrency using flock-guarded, run-scoped logs so
  16 parallel workers never interleave a log line.

Over the run this covered ~168,000 files across the 2024–2026 data periods.

The integrity check is the part I'm most happy with — it's the difference
between "the copy command exited 0" and "the file is actually usable
downstream." In essence:

isGoodRootFile() {
  local file="$1"
  [[ -s "$file" ]] || return 1            # non-empty
  ROOT_FILE="$file" root -l -b -q <<'EOF'  # open it in ROOT and prove it's sane
{
  TFile* f = TFile::Open(gSystem->Getenv("ROOT_FILE"), "READ");
  if (!f || f->IsZombie())            gSystem->Exit(1);   // unreadable / truncated
  if (!f->Get("treeTimeSeries"))      gSystem->Exit(1);   // missing payload
  if (f->Recover() != 0)              gSystem->Exit(1);   // needed recovery -> corrupt
  gSystem->Exit(0);
}
EOF
}

A file that needs Recover() was written partially — exactly the kind of
silent corruption that, left in place, poisons every plot built from it later.

Part 2 — ROOT/C++ quality-control analysis

Stack: C++17, ROOT RDataFrame, TChain, multithreading, TProfile /
TH2D / TScatter, Minuit fits.

A C++ program reads the synced time-series (a TChain over the per-run ROOT
files, with zombie files skipped) and produces the QC plots that physicists
actually look at, for both detector sides (A and C):

• Mean-DCA profiles — 18 TProfiles (one per sector) of the mean distance
  of closest approach versus time: a direct read-out of tracking quality drift.
• σ(DCA) trends — per-time-chunk Gaussian fits of the DCA distribution,
  with dynamic fit ranges from each histogram's RMS and Minuit convergence
  checks, so bad fits are dropped instead of silently skewing the trend.
• dE/dx heatmaps — TH2D maps of energy loss over time × φ for the full
  track and each readout region (IROC, OROC1–3), with a 95th-percentile cut so
  a few outliers don't blow out the colour scale.
• q/pₜ scatter — DCA versus time coloured by charge-over-momentum.

Processing 70+ TB in reasonable time meant treating I/O as the bottleneck:

ROOT::EnableImplicitMT(8);                 // multithreaded event loop

chain->SetBranchStatus("*", 0);            // read nothing by default...
for (auto* b : neededBranches)
    chain->SetBranchStatus(b, 1);          // ...only the branches we use

chain->SetCacheSize(100 * 1024 * 1024);    // 100 MB branch cache
for (auto* b : neededBranches)
    chain->AddBranchToCache(b);
chain->StopCacheLearningPhase();

Selective branch activation plus a sized branch cache is what makes a 70 TB
pass tractable instead of I/O-bound. Events are processed in fixed-size chunks
to bound memory, every stage is timed, and a custom run-scoped logger records
status per plot so a failed batch is easy to trace.

What I took away

• Trust nothing about large file transfers. Exit codes lie; sizes lie a
  little; only opening the file and checking its contents tells the truth.
• At TB scale, I/O is the design. The interesting performance work was in
  not reading data, not in the arithmetic.
• Automation is mostly failure handling. The happy path is a few lines; the
  retries, validation, locking and idempotency are the actual engineering.

Athanasios Tasis — Computer Engineering (MEng), University of Patras.
Work performed at GSI Helmholtz Centre (ALICE Collaboration).