Frameworks compared
A Translation, Not a Race
One model, built in two frameworks.
- Role
- Project author
- Classification
- Reproducible experiment
Problem
Comparisons fail when the models quietly differ.
How it was solved
I forced identical images and asserted identical parameters.
- Same images
- Same model
- Same run
Result
815,018
parameters in both, checked on every push
Built with
- TensorFlow
- Keras
- Docker
- GitHub Actions
Reproducible experiment / Frameworks
Rebuilding a model in a second framework is easy to do loosely. The useful question is which differences between the two cannot be removed at all, and what each one costs.
Both models, asserted815,018 parameters
One split, two frameworks, and one difference that stays
This diagram in words
- Split exported source — Produced by running torch. The other side reads it rather than rebuilding it.NumPy
- Architecture translated process — Rebuilt layer for layer in the second framework.Keras
- Count derived by hand artifact — From the architecture, not from either framework's summary.
- Differences corrected process — Initialisers, epsilon, padding and ordering, each handled explicitly.TensorFlow
- Asserted every push gate — A silent change in either framework fails the build rather than the comparison.GitHub Actions
- The irreducible one artifact — Kernel reduction order. Not a defect, and not removable.
Read the full background
Problem
Why this work exists
A framework comparison is worth nothing if the two models see different images or are not the same model. Both are easy to get wrong quietly: two random number streams cannot be made to draw the same augmentations, and two block diagrams can agree while the models underneath them do not.
Contribution
What I can claim
Exported the exact training indices by running torch rather than reproducing them in NumPy, translated the architecture with the parameter count derived by hand and asserted in CI, and enumerated the five differences between the frameworks with the correction each one needed.
Method
Making two frameworks answerable to the same number
Comparing two deep-learning frameworks is usually a comparison of two different experiments wearing the same name.
4 of 5 corrected for. The last is the floor on any comparison of this kind, and both models are asserted at 815,018 trainable parameters on every push.
Fix
The data split is exported by running one framework and then reused by the other.
Translate
The architecture is rebuilt rather than ported.
Derive
The parameter count is worked out by hand from the layer definitions, so there is a number to be wrong about that does not come from either framework.
Correct
Each known difference between the two is applied deliberately.
Assert
The count is then asserted on every push, so the two models cannot silently diverge later.
What this does not show
No accuracy comparison is published, because the second run has not been made: the results directory is empty and the repository's own table records it as pending. What is established is that the two models are the same model, which is the precondition for a comparison rather than the comparison itself.
How it was built
System
Workflow and decisions
The architecture, the split and the parameter count are three separate artefacts that have to agree. The count is checked without training, the split is checked before training, and the training script refuses to run if either disagrees.
- 1Export the split by running torch
- 2Translate the architecture
- 3Derive the parameter count by hand
- 4Correct for each framework difference
- 5Assert the count on every push
- TensorFlow
- Keras
- NumPy
- Docker
- GitHub Actions
Evidence
What is actually versioned
Derived by hand per layer and asserted on every push by a check that needs no data, no compile and no fit, because the training runs elsewhere and the number was otherwise only as true as somebody's memory of the last run.
13,500 train and 1,500 validation, exported by running torch rather than reproduced in NumPy, because torch's random stream cannot be replicated there and guessing would have made the comparison meaningless without saying so.
Weight decay, the learning-rate schedule, channel dropout and the augmentation definitions were each corrected for. The reduction order of the kernels is accepted as the irreducible floor on any comparison of this kind.
How the work is checked
Quality controls
How the work is checked
- The parameter count is asserted rather than stated, by a check that runs on every push in seconds
- The split travels as exported indices, so the two frameworks cannot quietly see different images
- Weight decay is halved for Keras, because the penalty enters the loss there rather than the gradient and its derivative carries a factor of two
- The two dropout rates are kept as separate constants, because the count cannot catch them being confused
- CIFAR-10 itself is never committed; only the index file travels
Limitations
Where the evidence stops
- The Keras run has not been made. The repository's results directory is empty and its own table records the Keras accuracy as pending, so no comparison figure is published here
- Even with every correction applied, kernel reduction orders differ between frameworks and devices, so an exact match was never available
- The parameter count is a floor, not a proof: it catches a wrong channel width or a missing bias, and it cannot catch the two dropout rates being confused, which is the worked example in the repository
- One architecture on one subset of one dataset, which says nothing general about either framework
What this changed in my practice
What this changed in my practice
The assertion that protects the comparison also shows its limit. The reference uses 0.25 in the conv blocks and 0.5 in the classifier head. Reusing the conv rate halves the regularisation in the head, and because dropout carries no weights the count still lands on 815,018. So the count is where the checking starts rather than where it ends.