MZQMohd Zamin Quadri
All work

Mathematics seminar

Same Answers, Different Weights

When are a network's weights unique?

Role
Seminar author
Classification
Academic research
Institution
Technical University of Munich
Inspect repository
What the code constructs: one network, the same network with its hidden units relabelled and two of them sign-flipped, and the verdict of comparing them.

  1. Problem

    Different weights can compute the same function.

  2. How it was solved

    I built such networks and checked them numerically.

    1. Build
    2. Permute and flip
    3. Verify

  3. Result

    2

    symmetries confirmed in code

  4. Built with

    • PyTorch
    • NumPy
    • Jupyter
    See the evidence on GitHub

Academic seminar / Mathematics

Two networks with identical weights do the same thing. The interesting question is the reverse: if two networks agree on every input, must their parameters be the same?

What is publishedno numerical finding

Two networks, two different questions about sameness

must not passsame answers, different weightsA small networkFully connected, small enoughto reason about exhaustively.Relabelled and flippedHidden units permuted, and twoof them sign-flipped.Agree on inputs?The two networks evaluated onsampled points.Parameters align?Under that exact relabelling,not up to anything looser.Unrelated networkPut through the identicaltest, and required to fail it.IdentifiedBoth checks pass and thecontrol fails, or the claim is…
Agreeing on every input and having matching parameters are different claims, which is why they are two gates rather than one. The unrelated network is there so the test is shown capable of rejecting something.

This diagram in words

  1. A small network source — Fully connected, small enough to reason about exhaustively.
    PyTorch
  2. Relabelled and flipped process — Hidden units permuted, and two of them sign-flipped.
    NumPy
  3. Agree on inputs? gate — The two networks evaluated on sampled points.
  4. Parameters align? gate — Under that exact relabelling, not up to anything looser.
  5. Unrelated network source — Put through the identical test, and required to fail it.
  6. Identified artifact — Both checks pass and the control fails, or the claim is not made.
    unittest

01 / 7One networkOne input, 4 hidden units, one output. A small fixed layout, not a trained model.

Hidden units
4
Activations the prototype builds with
tanh, sigmoid, ReLU
Trained
No - both networks are constructed
Two small networks side by side, each an input, four hidden units and an output. As the sequence runs, the second network's hidden units slide into a new order - the permutation two, zero, three, one - with faint leaders showing which unit went where, and then two of them flip sign, drawn as their pair of edges inverting from the accent colour to the warning one. The two networks compute the same function afterwards, which is the subject: agreeing on every input does not make two parameter sets the same. Two checks are then listed, the permuted and sign-flipped network agreeing and an unrelated network being rejected by the same comparison, so the test is shown to be capable of failing. The diagnostics it runs follow - exact and near-exact hidden-unit clones, sampled non-degeneracy, inactive contributions on sampled inputs - and the sequence closes on what is not claimed: three activation families are implemented, no trained model or numerical finding is published, and the formal conditions live in the three cited papers rather than in this prototype.
Read the full background

Problem

Why this work exists

Two networks with identical weights always compute the same function. The reverse does not follow, and a fully connected network can hide equivalent behaviour inside different weights, which is what makes a weight worth interpreting an open question rather than an obvious one.

Contribution

What I can claim

Built the prototype that constructs a network related to another by relabelling its hidden units and flipping signs, verifies that the two agree numerically on sampled inputs and that their parameters align under that relabelling, and runs the same comparison against an unrelated network so that the check is one that can fail.

Method

A check that is able to fail

Two networks with identical weights compute the same function.

  1. Build

    A small fully connected network, chosen to be understandable rather than capable.

  2. Relabel

    A second network is constructed from the first by permuting its hidden units and flipping signs in matched pairs.

  3. Agree

    The two are compared on sampled inputs and agree.

  4. Align

    So the parameters are checked separately, under the relabelling.

  5. Reject

    Finally the identical comparison runs against an unrelated network, and rejects it.

  6. What this does not show

    These are finite numerical diagnostics on sampled inputs. They can find exact clone pairs and inactive contributions; they do not prove global functional equivalence, and they do not satisfy the hypotheses of any published identifiability theorem. The repository versions no configuration, model or result, and this section claims none.

How it was built

System

Workflow and decisions

A seminar prototype supervised at Technical University of Munich. One module builds networks, one constructs the relabelled and sign-flipped relatives, one runs the diagnostics, and the notebooks are where they are explored rather than where results are kept.

  1. 1Build a small fully connected network
  2. 2Relabel its hidden units and flip signs
  3. 3Check the two agree on sampled inputs
  4. 4Check the parameters align under the relabelling
  5. 5Reject an unrelated network put through the same test
  • PyTorch
  • NumPy
  • Jupyter
  • unittest

Evidence

What is actually versioned

What is publishedno numerical finding

The repository's experiments and results directories intentionally hold no versioned configurations, no trained models and no findings, and it makes no claim of experimentally establishing identifiability for any trained model or activation family. That is its own scope statement, and this page claims nothing beyond it.

What the code constructs2 transformations

relabel the hidden units, then flip signs in matched pairs. The constructed network is then checked for numerical output agreement and parameter alignment against the one it came from.

What the diagnostics look for3 checks

exact and near-exact hidden-unit clones, sampled non-degeneracy, inactive contributions on sampled inputs. They are finite and numerical, on sampled inputs, across tanh, sigmoid, ReLU.

How the work is checked

Quality controls

How the work is checked

  • The comparison is run against an unrelated network as well as a related one, so it is a check that can fail rather than one that always passes
  • Output agreement and parameter alignment are checked separately, because the first without the second is the whole phenomenon being studied
  • The scope statement is in the README rather than left implicit, and it is the first thing after the question
  • The activation names in the code are described as the assumptions this prototype considers, with the formal conditions left to the cited papers

Limitations

Where the evidence stops

  • Finite numerical diagnostics on sampled inputs. They can find patterns such as exact clone pairs or inactive contributions; they do not prove global functional equivalence
  • They do not satisfy every hypothesis of any published identifiability theorem, and the repository says so rather than implying otherwise
  • Source code and exploratory notebooks, not a validated empirical benchmark. Nothing is versioned under experiments or results
  • A seminar prototype. The mathematics it points at belongs to the cited papers rather than to this repository
What this changed in my practice

What this changed in my practice

A check that always passes is not a check. Constructing the equivalent network is the easy half; the half that makes it mean anything is running the identical comparison against a network that should be rejected, and watching it be rejected.

Next evidence milestone: Hold a diagnostic to a theorem. The references the repository cites, Fefferman 1994, Bona-Pellissier, Miche and Malgouyres 2022, Petzka and Trimmel 2020, state conditions the prototype's checks do not test, and the useful next step is to make one check correspond to one hypothesis rather than to an intuition about it.

Source and documentationDiscuss relevant work