Random Forests Can Teach Neural Networks. And Vice Versa. With 94.7% Agreement.
- mahdinaser
- May 14
- 4 min read
Knowledge distillation has a well-known shape: a big teacher model trains a smaller student model of the same family. GPT-4 distills into GPT-4-mini. ResNet-152 distills into ResNet-18. Tree-based ensemble distills into a single shallow tree. Teacher and student speak the same architectural language.
What happens when they don't?
I built a systematic study of cross-paradigm knowledge distillation — bidirectional knowledge transfer between Random Forests and Deep Neural Networks. Full code and data: cross-paradigm-knowledge-distillation. The headline result: student models achieve 94.7%+ agreement with their cross-paradigm teachers, in both directions.
The bidirectional setup
┌───────────────────────┐ ┌────────────────────────┐
│ │ ◄── KD ── │ │
│ Random Forest │ │ Deep Neural Network │
│ (Tree-based) │ ── KD ──► │ (MLP / DNN) │
│ │ │ │
└───────────────────────┘ └────────────────────────┘
Teacher OR Student Student OR TeacherMost knowledge-distillation research lives inside one paradigm. This study asks a different question: can a tree-based ensemble teach a neural network — and can a neural network teach a tree-based model back? The answer in this experiment is yes in both directions, with surprising fidelity.
Direction 1 — Random Forest teaches a Neural Network
The Random Forest is the teacher; an MLP or DNN is the student. The student has to learn to mimic the forest's decisions, but the two models do not share parameters, gradients, or representational structure. Three distinct transfer methods were tested:
RF (Teacher) NN (Student)
─────────── ────────────
Probability ────► Probability Matching ────► Softmax outputs
Distribution
Feature ────► Feature Alignment ────► Hidden-layer
Importance activations
Leaf Index ────► Decision Boundary ────► Decision
Patterns boundary lossOf the three, probability matching produced the cleanest results: the NN student learns to match the soft probability distribution the RF teacher would have produced for each input. The result on the breast cancer dataset (binary classification, 569 samples, 30 features):
RF → MLP, probability matching
──────────────────────────────────────────
Teacher accuracy ▓▓▓▓▓▓▓▓▓▓ 95.61%
Student accuracy ▓▓▓▓▓▓▓▓▓▓ 95.61%
Teacher-student agreement ▓▓▓▓▓▓▓▓▓░ 94.74%
Confidence correlation ▓▓▓▓▓▓▓▓░░ 82.17%
Paradigm compatibility ▓▓▓▓▓▓▓▓░░ 0.80Same test accuracy. 94.74% of the time they make the same prediction. Their confidence scores correlate at 0.82. The NN has effectively learned to be the RF in disguise — without ever seeing a tree.
Direction 2 — Neural Network teaches a Random Forest
Now flip it. The neural network is the teacher. The Random Forest is the student. This is the harder direction — the RF has to absorb knowledge from a continuous, distributed representation it has no direct equivalent for.
NN (Teacher) RF (Student)
──────────── ────────────
Soft ────► Probability Matching ────► Tree training
probabilities weights
Hidden-layer ────► Feature Alignment ────► Augmented
activations feature space
Ensemble ────► Ensemble Guidance ────► Tree split
predictions criteriaResult on the same dataset:
MLP → RF, feature alignment
──────────────────────────────────────────
Teacher accuracy ▓▓▓▓▓▓▓▓▓▓ 95.61%
Student accuracy ▓▓▓▓▓▓▓▓▓▓ 95.61%
Teacher-student agreement ▓▓▓▓▓▓▓▓▓░ 94.74%
Confidence correlation ▓▓▓▓▓▓▓░░░ 67.82%
Paradigm compatibility ▓▓▓▓▓▓▓░░░ 0.75Same accuracy. Same agreement rate. The confidence correlation drops from 0.82 to 0.68 — the RF student is more decisive than the NN teacher, which makes architectural sense (RFs produce sparser probability outputs than well-calibrated NNs).
The full picture: both directions side by side
┌────────────────────────────────────────────────────────────────┐
│ │
│ Metric RF → MLP MLP → RF │
│ ───────────────────────────────────────────────────────── │
│ Teacher accuracy 95.61% 95.61% │
│ Student accuracy 95.61% 95.61% │
│ Teacher-student agree 94.74% 94.74% │
│ Confidence correlation 82.17% 67.82% │
│ Paradigm compatibility 0.80 0.75 │
│ Best method Prob. matching Feature align. │
│ │
└────────────────────────────────────────────────────────────────┘Both directions land in the same neighborhood of accuracy and agreement. The asymmetry shows up only in confidence — and that asymmetry is interpretable.
Why does this work at all?
Distillation across architectures works when both models are converging on the same underlying decision function — even if they represent it very differently internally. The forest's leaf-path decisions and the network's gradient-shaped manifold are both approximations of the same f(x). Knowledge distillation transfers the function, not the representation.
Decision function f(x)
──────────────────────
▲
│
┌───────────────┴───────────────┐
│ │
│ Two paradigms approximate │
│ the SAME function │
│ in different ways │
│ │
└───────────────────────────────┘
│ │
▼ ▼
┌────────────────┐ ┌──────────────────┐
│ RF: piecewise │ │ NN: continuous │
│ axis-aligned │ │ smooth manifold │
│ decisions │ │ representation │
└────────────────┘ └──────────────────┘Once you see it this way, cross-paradigm distillation isn't surprising — it's a confirmation that the function being learned matters more than the architecture learning it.
When does this actually matter in production
You have a working tree-based model in production and need to deploy it to an environment that only supports neural inference (mobile, browser, accelerator hardware).
You have a deep network that's accurate but uninterpretable; distilling into an RF gives you a stand-in for audit, explanation, and feature-importance reporting.
You're consolidating an ML stack and want a smaller, paradigm-consistent set of production models without sacrificing the accuracy gained from architecturally diverse training experiments.
You're building model-risk governance for a regulated industry and need a model in one paradigm to validate decisions made by a model in another.
Reproducing
git clone https://github.com/mahdinaser/cross-paradigm-knowledge-distillation.git
cd cross-paradigm-knowledge-distillation
pip install pandas openpyxl scikit-learn torchExcel results in data/raw/. Per-experiment JSON outputs in data/raw/experiments/{rf_to_mlp, mlp_to_rf, rf_to_dnn, dnn_to_rf}/. Datasets used: breast cancer (binary), digits (multi-class), synthetic (multiple sizes).
Author: Mahdi Naser Moghadasi, PhD — Senior ML Engineer
Comments