MZQMohd Zamin Quadri
All work

News text classification

The Line That Refuses to Run

A news classifier comparing TF-IDF and DistilBERT.

Role
Author
Classification
Engineering prototype
Inspect repository
The loader's three outcomes, and the vocabulary statistic that explains why the stopping one is the default.

  1. Problem

    A failed download can quietly become a fake score.

  2. How it was solved

    The run stops if the real dataset is missing.

    1. AG News
    2. TF-IDF and DistilBERT
    3. One evaluation

  3. Result

    It stops

    no real data, no number

  4. Built with

    • Hugging Face Transformers
    • scikit-learn
    • PyTorch
    See the evidence on GitHub

Research project / Text classification

The dataset will not download and there is a synthetic fixture sitting right there. Falling back to it would keep the pipeline green. What should the code do instead?

Why the fallback is not automatic86% of its vocabulary

A loader that refuses, and a split read once

dataset unavailableselection frozentunes the transformerLoad AG NewsThe synthetic fixture existsand is not substituted when…TF-IDF baselinesLogistic regression, linearSVM, random forest.DistilBERTFine-tuned through theTrainer.Run stopsNo fallback. A green pipelineon invented text measures…Validation splitStratified. The only splitanything is tuned against.Published test splitRead once, for the finalnumber. No choice is made…Metrics reportClassification metrics andconfusion matrices; latency…
Tuning has exactly one legal path - the return edge into the transformer from the validation split. The test split is entered once, from the left, and nothing is selected after it.

This diagram in words

  1. Load AG News gate — The synthetic fixture exists and is not substituted when the download fails.
    Datasets
  2. TF-IDF baselines process — Logistic regression, linear SVM, random forest.
    scikit-learn
  3. DistilBERT process — Fine-tuned through the Trainer.
    Hugging Face Transformers · PyTorch
  4. Run stops sink — No fallback. A green pipeline on invented text measures nothing.
  5. Validation split store — Stratified. The only split anything is tuned against.
  6. Published test split gate — Read once, for the final number. No choice is made after it.
  7. Metrics report artifact — Classification metrics and confusion matrices; latency optional.

01 / 7One request, three ways outLoad AG News. The junction has an exit for success, one for failure and one for the fixture.

AG News loaded
evaluate on the published test split
raises, and stops
the default when the download fails
templates, on request
only with --synthetic, for offline development
A junction with three exits, drawn from the loader's own branch. A request to load AG News travels into it as the reader scrolls and takes the default exit, which raises and stops: when the download fails the loader produces nothing rather than silently substituting the synthetic fixture it ships. A third exit leads to that fixture and carries a barrier across it, because it is reachable only by passing --synthetic explicitly. The reason follows as four bars, one per class, each showing the fixture's distinct vocabulary with the exclusive part filled inside it: 34 of World's 41 words, 38 of Sports' 44, 39 of Business' 43 and 35 of Sci/Tech's 41, which is 86 per cent of the whole vocabulary appearing in exactly one class. A bag of words would separate that perfectly, so a score measured on it would describe the template list rather than any model. The sequence closes on the two tracks the project compares - TF-IDF classifiers and a fine-tuned DistilBERT, evaluated the same way - and on the three rules governing which split is touched when.
Read the full background

Problem

Why this work exists

A pipeline that depends on a download will eventually fail to get it, and there is almost always a fixture nearby that would let it keep going. Taking that option produces a number - and a number produced from template text is indistinguishable, in a results table, from one that means something.

Contribution

What I can claim

Built both tracks and the shared evaluation, and made the loader refuse: when AG News cannot be fetched it raises, and the synthetic fixture is reachable only by passing --synthetic. The fixture is for developing code offline, and the code says so rather than treating it as a fallback.

Method

What to do when the data does not arrive

Two tracks, one task, one evaluation: TF-IDF baselines against a fine-tuned transformer on news headlines.

  1. Request

    The pipeline needs a public dataset, fetched at run time.

  2. Fail

    There is a synthetic fixture in the repository for developing offline.

  3. Refuse

    So the loader raises instead.

  4. Count

    The reason is in the fixture itself.

  5. Compare

    With real data the two tracks are then measured the same way, on a test split kept out of tuning entirely.

  6. What this does not show

    No result is published and none is claimed. The repository versions no data, no trained model, no checkpoints, no metrics and no run metadata, so there is no accuracy, no F1 and no verdict on whether the transformer earns its cost here. The vocabulary counts on this page describe the synthetic fixture, which exists for offline development and must not be used to report model capability.

How it was built

System

Workflow and decisions

A loader that can refuse, two training paths behind one interface, a shared evaluation, and inference helpers for locally saved artefacts. Models and results are untracked by design, which is why the repository can be read without any of it being mistaken for a measurement.

  1. 1Load AG News, or stop
  2. 2TF-IDF: logistic regression, linear SVM or random forest
  3. 3DistilBERT: fine-tuned through the Hugging Face Trainer
  4. 4a stratified validation split for tuning the transformer
  5. 5the published test split for the final evaluation only
  6. 6no tuning of model choices or thresholds against the held-out split
  7. 7Report classification metrics and confusion matrices, with latency optional
  • Python
  • Hugging Face Transformers
  • Datasets
  • scikit-learn
  • PyTorch

Evidence

What is actually versioned

Why the fallback is not automatic86% of its vocabulary

The synthetic fixture is 20 hand-written sentences, 5 per class across World, Sports, Business, Sci/Tech. Counting the words in them, 86% appear in exactly one class - so almost every word gives its label away and a bag of words would separate them perfectly. A score measured there would look like a result and be a description of the template list, which is the number an automatic fallback would have produced.

What is publishedno model result

The repository versions source and notebooks but no downloaded data, no trained model, no checkpoints, no metrics, no plots and no run metadata. It states that it therefore makes no verified accuracy, F1, latency, parameter or model-comparison claim, and this page makes none.

What is compared2 tracks, one evaluation

TF-IDF (logistic regression, linear SVM or random forest); DistilBERT (fine-tuned through the Hugging Face Trainer). Both are scored the same way on the same task, which is the only arrangement under which the comparison means anything.

How the work is checked

Quality controls

How the work is checked

  • The loader raises on a failed download rather than substituting its own fixture, so the pipeline cannot quietly produce a number about nothing
  • The synthetic path exists for offline development and is gated behind an explicit flag, with the README stating it must not be used to report capability
  • Transformer tuning uses a stratified validation split, and the published test split is kept for the final evaluation only
  • The README names what must be recorded before runs are compared - dataset revision, package versions, hardware, seed, training arguments, split policy - and notes that seeding does not make GPU operations deterministic

Limitations

Where the evidence stops

  • Nothing is measured here. No data, model, checkpoint or metric is versioned, so there is no accuracy to compare and no claim that either track wins
  • The synthetic fixture is not a dataset. Its results are not comparable to AG News or to real news classification, and the page quotes it only to explain why it is not used by default
  • Single-task, single-dataset. Whether a fine-tuned transformer earns its cost over TF-IDF is exactly the kind of question this repository is set up to ask and has not answered
  • A learning and research project. It is not a deployed content-moderation, routing or editorial decision system, and the README says so first
What this changed in my practice

What this changed in my practice

The failure mode I was guarding against is not a wrong number, it is a plausible one. A fallback that keeps the pipeline green converts a missing dataset into a results table, and by the time anyone reads that table the distinction is gone. Counting the fixture's vocabulary afterwards was the part that convinced me: eighty-six percent of the words are giveaways, so the fallback would not have scored badly - it would have scored beautifully.

Next evidence milestone: Run it and publish the comparison properly - the dataset revision, the seed, the hardware and the metrics together, as one artefact. The interesting question is not which track wins but by how much, and whether that margin survives being repeated.

Source and documentationDiscuss relevant work