← back

eQECC - Automated Equivalence Checking for Quantum Error-Correction Codes

This project started as the benchmark implementation for my Bachelor’s thesis, Automated Equivalence Checking of Stabilizer Codes, and was later extended into a replication package for the paper Methods for Automated Equivalence Checking of Quantum Error-Correction Codes. The result is both a collection of algorithmic prototypes and a framework for empirically evaluating their runtime and memory performance. The final hybrid algorithms were also contributed to MQT QECC, where they provide the user-facing equivalence checks.

The motivating research question for equivalence checking is the following:

Given two descriptions of quantum error-correction codes, do they represent genuinely different codes, or are they the same up to an admissible transformation?

This matters when comparing code constructions, removing duplicate entries from code databases, or reusing an expensively constructed encoding circuit for an equivalent code. The difficulty is that the same mathematical code can have very different computational descriptions, while two physically different code spaces may still be equivalent under the relation of interest.

Take the Steane code, drawn below as a with its characteristic triangle of seven qubits with three colored plaquettes. The base construction sits in the center. On the left, the same code appears with its qubits relabeled, so every stabilizer acts on different qubit indices. On the right, local Clifford gates (HH or HSHS) are instead applied to individual qubits of the original, which changes the Pauli operators the stabilizers consist of. Written down as matrices, the three look nothing alike, yet they are equivalent under the respective transformations.

Three drawings of the Steane code: a relabeled version on the left, the original construction in the center, and a version with additional local Clifford gates H and HS on the right.

Two Degrees of Freedom

I represent a general ⟦n,k,d⟧\llbracket n,k,d\rrbracket stabilizer code as the binary symplectic tableau of its stabilizer group, which characterizes its code space,

[ X∣Z ]∈F2r×2n,r=n−k,\left[\,X\mid Z\,\right] \in \mathbb{F}_2^{r\times 2n}, \qquad r=n-k,

whose rows encode independent stabilizer generators, and a CSS code as its separated check matrices

HX∈F2rx×n and HZ∈F2rz×nr=n−k=rx+rz,H_X \in \mathbb{F}_2^{r_x\times n} \text{ and } H_Z \in \mathbb{F}_2^{r_z \times n} \qquad r=n-k=r_x+r_z,

This representation of an abstract mathematical object (the stabilizer group of a stabilizer code) as a concrete matrix exposes two independent sources of freedom, which have to be accounted for in my equivalence checking algorithms.

Representation Freedom

The chosen generators are not unique. Reordering generators or multiplying them produces a different matrix but preserves the stabilizer group, and therefore the code space. In the tableau or check-matrix representation this is an invertible row transformation

S⟼RS,R∈GL⁡(r,F2).S \longmapsto R S, \qquad R\in\operatorname{GL}(r,\mathbb{F}_2).

Consequently, comparing two tableaus or check matrixes SS and S′S' entry by entry would be incorrect. What matters is their row space.

Physical Freedom

The second freedom comes from the definition of equivalence. Meaning, two codes that can be transformed into each other using the physical transformations, are considered equivalent. I primarily looked at two transformations which only concern the code space:

Example

Both freedoms are summarized in the figure below. In the mathematical domain on the left, the abstract Steane code and a relabeled copy are related by a qubit permutation π\pi—a physical transformation. Representing those abstract codes as data objects in the computational domain on the right, there are many different matrices representing one and the same abstract code—such as its standard self-dual check matrices HXH_X and HZH_Z, but also a combined symplectic tableau using a different generator basis. These different representations are essentially related by row operations RR—the representation freedom. An equivalence checker only ever sees such matrices as its input, but has to decide whether any two of them describe equivalent codes—aka it as to recognize the red dashed connections, which compose the two possible types of transformation.

Left, in the mathematical domain: two labelings of the Steane code related by a qubit permutation pi. Right, in the computational domain: several matrix representations of each code, related by row operations R, with dashed red arrows marking that all of them must be recognized as equivalent.

Together, both freedoms give the central equation of the project:

S′=R S T.S' = R\,S\,T.

Here, RR absorbs the arbitrary choice of stabilizer generators and TT is the admissible physical transformation. The different checking problems arise by changing the structure permitted for TT, or the concrete representation SS and S′S'.

Equivalence Problems

In my thesis I formulated and considered four different equivalence problems. The last one is omitted in the paper, which focuses on pairwise equivalence problems. All of these checks use the phase-free symplectic representation. Thus, stabilizer phases are ignored and permutation equivalence is understood up to a possible physical Pauli correction.

Pairwise Permutation Equivalence of Stabilizer Codes

Are two general stabilizer codes the same up to relabeling their physical qubits? A qubit permutation has to move the corresponding XX and ZZ columns together.

Formal problem

With PP a permutation matrix, the decision problem is

∃P∈Sn, ∃R∈GL⁡(r,F2):[ X′∣Z′ ]=R[ X∣Z ][P00P].\exists P\in\mathfrak{S}_n,\ \exists R\in\operatorname{GL}(r,\mathbb{F}_2): \qquad \left[\,X'\mid Z'\,\right] = R \left[\,X\mid Z\,\right] \begin{bmatrix} P&0\\ 0&P \end{bmatrix}.

Pairwise Permutation Equivalence of CSS Codes

The same question for CSS codes, where the generator bases of the two check matrices may change independently, but the qubit permutation must again be shared.

Formal problem ∃P∈Sn, RX∈GL⁡(rX,F2), RZ∈GL⁡(rZ,F2):HX′=RXHXP,HZ′=RZHZP.\exists P\in\mathfrak{S}_n,\ R_X\in\operatorname{GL}(r_X,\mathbb{F}_2),\ R_Z\in\operatorname{GL}(r_Z,\mathbb{F}_2):\quad \begin{aligned} H_X' &= R_X H_X P,\\ H_Z' &= R_Z H_Z P. \end{aligned}

Pairwise Local-Clifford Equivalence of Stabilizer Codes

Are two stabilizer codes the same up to a local Clifford operation the physical qubits? Modulo Pauli operations and global phases, a single-qubit Clifford has six possible actions on the Pauli axes.

Formal problem

In the binary symplectic representation, an nn-qubit local Clifford becomes

Q=[ABCD],Q= \begin{bmatrix} A&B\\ C&D \end{bmatrix},

where A,B,C,DA,B,C,D are diagonal and each local block

Qi=[aiibiiciidii]∈Sp⁡(2,F2)Q_i= \begin{bmatrix} a_{ii}&b_{ii}\\ c_{ii}&d_{ii} \end{bmatrix} \in\operatorname{Sp}(2,\mathbb{F}_2)

describes the action on qubit ii. The pairwise problem is therefore

∃Q, ∃R∈GL⁡(r,F2):[ X′∣Z′ ]=R[ X∣Z ]Q.\exists Q,\ \exists R\in\operatorname{GL}(r,\mathbb{F}_2): \qquad \left[\,X'\mid Z'\,\right]=R\left[\,X\mid Z\,\right]Q.

Existence of a local-Clifford-equivalent of CSS Codes

The thesis additionally asks whether a given stabilizer code is local-Clifford equivalent to any CSS code.

Formal problem ∃[ X′∣Z′ ]=[HX00HZ]∈F2r×2n:∃Q, ∃R∈GL⁡(r,F2):[ X′∣Z′ ]=R[ X∣Z ]Q.\exists \left[\,X'\mid Z'\,\right] = \left[\begin{smallmatrix}H_X&0\\0&H_Z\end{smallmatrix}\right] \in \mathbb{F}_2^{r\times 2n}: \exists Q,\ \exists R\in\operatorname{GL}(r,\mathbb{F}_2): \left[\,X'\mid Z'\,\right]=R\left[\,X\mid Z\,\right]Q.

Algorithmic Toolbox

I implemented and evaluated several complementary strategy families rather than assuming that a single method would dominate. For each equivalence problem, 5 concrete prototypes were implemented. In their structure, the implementations seem rather random, and for the thesis they mostly were to be honest. For the paper, I categorized and structured them a bit more intentionally, like it is explained below.

Invariants and Signatures

An invariant is a property preserved by the selected equivalence relation. Examples include the code parameters, ranks of tableau submatrices, zero- and duplicate-column counts, low-order linear dependencies, stabilizer weight enumerators, and low-degree local invariants.

For example, the stabilizer weight enumerator

W(S)w=∣{s∈S∣weight⁡(s)=w}∣W(\mathcal{S})_w = \left|\left\{s\in\mathcal{S}\mid\operatorname{weight}(s)=w\right\}\right|

is preserved by both permutations and local Clifford operations. A mismatch proves inequivalence immediately; a match is only a necessary condition and cannot prove equivalence.

For permutation problems I also adapted coordinate signatures from classical coding theory, namely from Sendrier’s Support Splitting Algorithm. Each qubit receives a fingerprint derived from the weight enumerator of a punctured code’s hull. Qubits with different signatures can never be mapped to one another, as indicated below, so the signatures partition the candidate permutation space into classes. They can both reject an instance globally and make a later exact check smaller.

Two permutation-equivalent four-column matrices and a bipartite mapping graph in which columns may map only to columns with the same signature.

These methods explicitly traverse the physical transformation group and resolve representation freedom at every step, which is abstractly shown below. The baseline is brute force, while more involved variants avoid redundant or unpromising paths through automorphism groups, canonicalization (implementing the KLS normal form really drove me crazy), or orbit traversal when representing a stabilizer code as a graph state.

A group-action search tree traversing physical transformations while resolving representation freedom at each visited transformation.

The graph-state variant relies on the fact that a local Clifford operation on a graph state corresponds to a local complementation of the underlying graph. In the example below, complementing at node 00 toggles every edge within its neighborhood (light lilac). The orbit traversal explores all graphs reachable by such steps, discarding those it has already visited.

A five-node graph before and after local complementation at node 0: edges among the neighbors 1, 2 and 4 of node 0 are toggled.

These methods are very efficient when the physical transformation space stays small, otherwise the traversal takes too much time. Even the graph-state orbit can grow exponentially in the number of qubits.

Isomorphisms

Another strategy removes the arbitrary generator basis by construction and encodes the remaining physical freedom into a colored graph. Thus, we mapped both the representation transformations and the physical transformations to the same kind of freedom, namely, a color-preserving relabeling of the graph-vertices, such that

C∼C′  ⟺  G(C)≅G(C′).C\sim C' \iff G(C)\cong G(C').

For general stabilizer codes, graph vertices can represent every stabilizer-group element and its incidences with qubits and Pauli labels. In the example below, this full stabilizer-group construction is applied to two exemplary codes which indeed are permutation equivalent, as indicated by the existing graph isomorphism. The freedom of the graph isomorphism then simultaneously handles the permitted reordering and the basis-independent structure.

Two isomorphic bipartite graphs connecting all stabilizer-group elements to the qubits they act on, with orange edges for X support and green edges for Z support.
Constructed graphs of the permutation-equivalent codes with stabilizer groups ⟨IIX, XZI⟩ (left) and ⟨XXZ, IXZ⟩ (right), with the isomorphism mapping qubits 0 → 1, 1 → 2, 2 → 0.

The same idea adapts to local-Clifford equivalence, only the graph construction is a bit different to allow for an independent Pauli-basis change of each qubit, while keeping their order intact. Below is one such constructed graph for the paper’s running example of a code with stabilizer group ⟨IXXX,ZZIZ⟩\langle IXXX, ZZIZ\rangle.

Full-group graph for local-Clifford equivalence: the four stabilizer elements IIII, IXXX, ZZIZ and minus ZYXY on top, connected to triangles of Pauli nodes X, Y, Z for each of the four qubits.

Both constructions grow with the full stabilizer group, i.e. with 2r2^r. For CSS codes, a slightly different construction instead uses the circuits (not quantum or hardware circuits) of the binary matroids induced by HXH_X and HZH_Z, as shown using the same running example below. This shifts the exponential cost from the number of generators rr to the number of qubits nn.

Matroid-circuit graph: circuit nodes of H_X in orange and of H_Z in green, each connected to the qubits 0 to 3 it contains.

These methods can be extremely effective while the graph stays small. Their main failure mode is not the isomorphism test itself, but the time and memory needed to construct an exponentially growing graph.

Constraint Solving

Encodings of the equivalence problems into constraint systems or linear systems of equations handle both freedoms simultaneously, and can be solved by delegating the system to a specialized solver.

For SAT, boolean variables describe a row transformation RR and either a permutation or local Clifford witness. An auxiliary tableau AA can then be constrained from both sides, as sketched below: it must result from physically transforming one input SS by PP or QQ, and from changing the generator basis of the other input S′S' by RR. The formula is satisfiable exactly if both paths can meet, and a satisfying assignment directly yields the witness.

SAT encoding: S is mapped to the auxiliary tableau A by a physical transformation P or Q, and S prime is mapped to A by a row transformation R.

For local-Clifford equivalence I also implemented the graph-state algorithm of Van den Nest, Dehaene, and De Moor. It reduces the problem to a linear system over F2\mathbb{F}_2 with 4n4n unknowns, followed by constraints ensuring that every local block is symplectic. Within its valid domain it is polynomial and particularly fast.

Limitation of the Graph-State Reduction

Representing a stabilizer code as a graph state, which itself can be represented by a simple undirected graph, enables algorithms such as orbit traversal or the aforementioned polynomial-time algorithm to solve local-Clifford equivalence. However, the use of these algorithms first requires the transformation of a stabilizer code into a stabilizer state, and then a graph state—and here I noticed a slight issue, which made my life a lot harder.

Turning a stabilizer code into a stabilizer state introduces kk reference qubits through a Choi-Jamiolkowski-style purification. This requires choosing logical XX and ZZ operators, or equivalently a concrete logical encoding of the code space. That choice is not intrinsic to the code space, so consequently, needs to be accounted for as an additional representation freedom within my checks. As sketched below, an encoding map VV and its precomposition V∘UV \circ U with a logical Clifford UU have the same image C\mathcal{C}, so they describe the same code, but fix different logical frameworks.

An encoding map V from the logical space into the physical space and the map V composed with a logical Clifford U both have the same image, the code space C.

Two descriptions of the same code may use logical frameworks related by an arbitrary kk-qubit Clifford. For k≥2k\geq2, that Clifford may entangle the reference qubits, while the graph-state equivalence algorithm only permits local operations there, as is visualized below. As a result, a graph-state method that is complete for stabilizer states does not automatically remain complete for general stabilizer codes.

An encoding map connecting reference input qubits to physical output qubits, enclosed by red code-equivalence and lilac local-Clifford-equivalence boundaries.
Code-to-state transformation via the encoding map (Enc): a stabilizer state on k reference and n physical qubits. Code equivalence permits arbitrary k-qubit Cliffords on the reference subsystem and local Cliffords on the physical qubits (red); standard local-Clifford equivalence permits only local Cliffords on every qubit (lilac).

The implemented graph-state algorithms can therefore only be used (without additional restrictions) when k<2k<2: there is no logical freedom when k=0k=0, and for k=1k=1 every logical Clifford is still a single-qubit operation. Identifying this boundary was one of the most interesting theoretical outcomes of the thesis.

Refute - Refine - Decide

The experiment verified my expectations and showed that no single backend is best for every code class, equivalence notion, and parameter range. I therefore combined the methods into an input-dependent hybrid architecture:

  1. Refute: Compare cheap invariants for every input and more expensive invariants only where their measured benefit justifies their cost.
  2. Refine: For permutation equivalence, partition qubits by their signatures so that later methods only consider mappings within compatible classes.
  3. Decide: Select a sound and complete backend based on the code family and dimensions: group-action search for very small search spaces, graph or matroid isomorphism while their representations remain manageable, the graph-state linear system within its valid domain, and SAT as the broad fallback.

The flowchart below shows this architecture. Invariants refute (pink) the instance potentially which leads to an early exit, and signatures refine (cyan) the permutation search space before the dispatcher chooses a fitting strategy to decide the instance (lilac), routing each instance along one of the dashed edges to the strategy that suits it: group-action search for a small physical transformation space, isomorphisms for small representation freedom, and constraint solving as the reliable fallback.

Refute-Refine-Decide flowchart: invariants in a pink refute stage, signatures in a cyan refine stage, and a dispatcher in a lilac decide stage choosing between group-action search, isomorphisms, and constraint solving.

This ordering creates an intentional runtime asymmetry. An inequivalent pair may be rejected by the first inexpensive mismatch, whereas an equivalent pair generally has to reach a complete decision procedure.

The library-facing checks return witnesses when possible:

from mqt.qecc import (
    are_local_clifford_equivalent,
    are_permutation_equivalent,
)

permutation = are_permutation_equivalent(code_a, code_b)
local_cliffords = are_local_clifford_equivalent(code_a, code_b)

# None means that no such equivalence exists. Otherwise, the result
# explicitly describes the transformation mapping code_a to code_b.

As a note, the hybrids constructed in the thesis differ from those considered in the paper, as I constructed the latter one with a bit more data from the empirical evaluation.

Evaluation

Problem Instances

I evaluated the implementations in two complementary regimes. The randomized suite covered a grid of code parameters up to n=47n=47, with up to ten equivalent and ten inequivalent pairs for each (n,k)(n,k) setting and algorithm. A second suite used 18 named, structured codes ranging from the small 3-qubit repetition, 5-qubit perfect, and Hamming codes to Golay, color, rotated-surface, and bivariate bicycle codes with up to 144144 physical qubits. This combination was important: dense, random instances expose broad scaling behavior, while structured codes reveal whether the resulting hybrid still works on the kinds of constructions one might actually want to compare.

The benchmark pairs were generated deliberately. Positive instances start from one code and apply a known random qubit permutation or local Clifford, followed by a random change of stabilizer-generator basis. For negative instances, simply sampling two independent codes would make many comparisons unrealistically easy. I instead applied a short two-gate Clifford or CNOT perturbation and retained the pair only after an exact SAT or matroid backend certified inequivalence where feasible. Larger CSS cases use a separate permutation-invariant certificate. All randomized experiments derive from a master seed of 4242, and input generation and certification happen before the timed call.

Resource Restriction

Every algorithm call was isolated in its own supervised worker and received a wall-clock budget of 54005400 seconds and a memory budget of 1313 GiB. The supervisor combined an address-space limit with periodic monitoring of the worker’s complete process group, so memory allocated by child processes was counted as well. On a timeout or memory violation it terminated the whole group, while recording timeouts, memory failures, execution errors, wrong answers, and input-generation failures separately. The successful-call timer surrounds only the algorithm itself; worker startup and result transfer are excluded.

I also collected diagnostics beyond elapsed time, for example solver decisions when comparing SAT implementations, or deciding stage when measuring hybrids.

Insights

Invariants and Signatures

Cheap invariants, like the code parameters or matrix ranks, are always applied. The question was whether the more expensive ones are worth it. Preliminary experiments narrowed the candidates down to linear column dependencies and signatures for permutation equivalence, and a low-degree local invariant for local-Clifford equivalence.

Their job is to reject as many inequivalent instances as possible, and they do so remarkably well, especially on CSS codes, as the mean rejection rates on the randomized inequivalent instances show:

Linear column dependenciesSignaturesLocal invariant
Non-CSSCSSNon-CSSCSSGeneral
43.8 %90.5 %46.0 %99.0 %29.4 %

However, a high rejection rate alone does not make an invariant useful. It also has to be cheaper than the decision work it tries to avoid—it doesn’t make sense to compute an invariant for minutes that can only hopefully reject the instance, when a full decision algorithm would have decided it within seconds. The parameter maps below therefore compare each invariant’s runtime with that of the fastest decision method for the same parameters: blue cells favor the invariant, red ones the decision method, and the pink outlines mark where I consider the invariant worthwhile. Linear column dependencies only pay off for larger codes, signatures on general stabilizer codes only for small ones (they scale badly with the number of generators rr), and the local invariant only for small nn, where it actually is the most discriminant anyways. The specialized signature implementation for CSS codes, is the exception and beneficial throughout the whole parameter space. Thus is probably an implementation artifact, because for the CSS case I could make use of heavily optimized F2\mathbb{F}_2 libraries, whereas I needed to handroll a lot of F4\mathbb{F}_4 arithmetic myself in Python for the adaptation to general stabilizer codes.

Three parameter maps of the runtime of linear column dependencies, signatures, and the local invariant relative to the fastest decision method, with pink outlines marking where each invariant is useful.

Signatures have a second job: refining the permutation search space by partitioning the qubits into classes. The maps below show the fraction qˉ\bar{q} of qubit pairs they tell apart, from 00 (no refinement) to 11 (every qubit distinguished). On average, two random distinct qubits are distinguished with a probability of 86 %86\,\% for general stabilizer codes and 69 %69\,\% for CSS codes. The refinement saturates as rr grows, while the computation time keeps growing, as the timed-out cells in the upper right of the left map show.

Two parameter maps of the fraction of qubit pairs distinguished by signatures, for general stabilizer codes and CSS codes.

So invariants and signatures are beneficial, but only in parts of the parameter space, and the hybrid has to apply them depending on the input parameters.

Decision Methods

The most important result was not a universal winner strategy, but a map of where each paradigm performs best, shown below. Apart from general selection guidelines, these maps also provided the exact parameter thresholds for the dispatcher of the hybrids.

Three parameter maps showing the empirically fastest equivalence-checking method for each combination of physical qubits and stabilizer generators: permutation equivalence for stabilizer codes, permutation equivalence for CSS codes, and local-Clifford equivalence for stabilizer codes.

Group-action search is only competitive while the physical search space is small. The exhaustive search over permutations, combined with the rank-based row-space check, wins for CSS codes with few qubits in the lower left of the middle map. Traversing the LC-orbit while canonicalizing into KLS form at each step is fastest for a few isolated settings, but shows no consistent trend.

Isomorphisms win where their encoded object stays small: the full-group encoding for few stabilizer generators (low rr), the matroid-circuit encoding for few qubits (low nn). Isolating their runtime, as in the maps below, shows how steeply the cost rises with the graph size, until memory becomes the actual limiting factor (crosses). Surprisingly, the permutation construction runs out of memory before the local-Clifford one, despite having fewer nodes in our examples. This is actual an implementation artifact: the graph-isomorphism backend does not support colored edges, so the permutation encoding represents them by auxiliary nodes (aka blowing up the graph size even more).

Three parameter maps of the mean runtime of the full-group encodings for permutation and local-Clifford equivalence and the matroid-circuit encoding, with crosses marking runs that ran out of memory.

SAT is the strongest general baseline. It covers the widest parameter range for both permutation and local-Clifford equivalence of stabilizer codes, and never ran out of memory, an important property for a fallback solution for big codes. The one surprise was permutation equivalence of CSS codes: as the comparison below shows, there are some parameter ranges where the CSS codes are slower by a magnitude compared to same-size general stabilizer codes. This is not caused by the concrete representation, as CSS codes given as full tableaus behave the same, and choosing a different generator basis to make a CSS code look non-CSS also does not help. Instead, the separated X/ZX/Z structure couples the constraints less, so a wrong partial permutation is refuted much later by the solver (but I go into way more detail about that in the paper). Ironically, the same structure is what makes other things like signatures so effective on CSS codes.

Two parameter maps of the mean SAT runtime for permutation equivalence: tableau encoding on general stabilizer codes and check-matrix encoding on CSS codes, the latter being much slower.

The graph-state linear system has a noticeable conversion overhead for small codes, but within its restricted domain k<2k<2, it is the fastest local-Clifford method.

Hybrid Checkers

Finally, I evaluated the complete implemented hybrids with conservative thresholds derived from the experiments above, on randomized variants of the structured codes. In the table below, the color of each cell encodes the mean runtime for equivalent (++) and inequivalent (−-) instances, and its label names the stage that terminated most runs: CI/EI cheap/expensive invariants, S signatures, BF exhaustive (“brute force”) search, MI/GI matroid/graph isomorphism, SAT, and LSE for the graph-state linear system.

Table of the hybrid checkers on 18 structured codes from the Bell pair to bivariate bicycle codes, with cells colored by mean runtime and labeled by the stage that terminated most runs.

The most visible effect is an asymmetry between equivalent and inequivalent inputs. An equivalent pair can never be rejected early and always has to reach a complete decision procedure, whereas cheap invariants terminate many inequivalent permutation instances, and signatures become decisive for larger CSS codes.

In numbers, most cases with at most 3737 physical qubits had mean hybrid runtimes below 1010 seconds. Even at n=144n=144, inequivalent CSS permutation instances were decided in about 310310 seconds on average, while inequivalent local-Clifford instances remained below 1616 seconds on average. For equivalent local-Clifford instances, every evaluated n=72n=72 case completed with a mean of 4949 seconds, while larger cases increasingly reached the timeout. These numbers are empirical feasibility results under the paper’s resource budget, not worst-case guarantees.

Replication Package

The repository preserves the research prototypes, tests, structured code instances, thesis benchmarks, and the complete experimental pipeline used for the paper, which is structured as follows:

paper/
  benchmarks/       long-running data collection
  experiments/      deterministic extraction and aggregation
  visualizations/   generation of the paper's figures

The paper pipeline deliberately separates three phases for each figure that is used in the paper, as shown below. Only the first phase generates codes, runs algorithms, and consumes meaningful compute time, on a benchmark server. Its raw measurements in data/collected/ are the only thing transferred to the local machine, where a deterministic extraction step turns them into figure-ready tables, which the visualization step then draws. This makes it possible to inspect the exact measurements and aggregation rules, regenerate the figures without rerunning days of benchmarks, or reproduce the entire experiment from the original seeded inputs.

As a note, I committed the actually used data from the paper in the repo as well.

Replication pipeline: phase 1 collects measurements on a benchmark server into data/collected, which is the only artifact transferred to the local machine, where phase 2 extracts figure-ready tables and phase 3 visualizes them.

Acknowledgements

I really really enjoyed working on this project as a thesis, and am super happy about my (unofficial) advisor Tom, who introduced me to that topic and really helped me a lot getting started—even months before I actually started the thesis officially. And he helped me get a job as a student assistant at the CDA, for which I am super thankful. Obviously I am also super grateful for mu official advisor and co-author Ludwig who I finished the thesis with in the end, and who was super supportive when I threw the idea of a potential publication into the room. And last but definitely not least, my supervisor, co-author, and employer, Robert, whose guidance, motivation, and approachability I cannot appreciate and cherish enough :)