Dr. Michael Fromm @effi288

@JJitsev @wrwagox @eliebakouch You can use it soon and try it out The best is probably to use your own internal benchmarks that have not yet be made it into your training data.

@austriaaann @eliebakouch x.com/effi288/status…

Dr. Michael Fromm @effi288

16 hours ago

@eliebakouch Thanks for pointing that out, we followed in the paraphrasing the Nvidia datasets. We will remove the GPQA benchmarks in an updated version of the tech report! Its good that we released everything so that it was possible to find!

keyboard_arrow_left Previous keyboard_arrow_right Next

1 1 26 2K 4

"TinyMATH For each of the 7500 examples in the MATH training set, we generate 100 new, similar problems. We then create Python code solutions to the newly for each problem (TinyMATH-PoT), and two flavors of conversational English discussing these solutions (TinyMATH-MIND). In aggregate, this yields 1.14B tokens of novel, synthetic data targeted to improve performance on the MATH benchmark. A microanneal consisting of all of these new tokens in a 50/50 ratio with web data yields 13.2 points of improvement in the MATH benchmark and 13.9 points in GSM8K." Olmo 3 does the same for the MATH benchmark, so you can assume everyone does it. arxiv.org/pdf/2512.13961 Good that we built in the open🧐

@eliebakouch huggingface.co/datasets/Idavi… It is also listed here as train split. huggingface.co/nvidia/NVIDIA-…

@eliebakouch Thanks for pointing that out, we followed in the paraphrasing the Nvidia datasets. We will remove the GPQA benchmarks in an updated version of the tech report! Its good that we released everything so that it was possible to find!

keyboard_arrow_left Previous keyboard_arrow_right Next

They used 6B tokens, we only used 2B tokens. We also mentioned it openly in the tech report btw: "As part of the SFT mixture, we additionally include QA-base (∼0.05% of the pool; the 1.43B English tokens are counted under the SFT category and the 1.87B German tokens under the Ger- man category in Table 8), paraphrased training splits of 25 standard NLP benchmarks in English and German, analogous to the paraphrase-augmented benchmark training data in Olmo 3’s mid- training mix (e.g. TinyMATH, Dolmino Flan) [64] and the benchmark-seeded synthetic data in Nemotron 3 [61]."

@JJitsev @eliebakouch huggingface.co/nvidia/NVIDIA-…

@JonasAndrulis @kimmonismus We used at least one month of 1024 x Nvidia B200 for synthetic data generation. We especially focused on rare data such as long context data and knowledge intense German data (climbmix translation to German), also a lot of post training data.

Fair question! The nemotron_h files are the architecture implementation (hybrid Mamba-2/attention), which we adopted from Nvidia's open Nemotron 3 Nano design. The weights are trained from scratch: 27T tokens over three phases on Telekom's Munich cluster, random init. That's why the repo needs those modeling files. Proof is verifiable: we release intermediate checkpoints from early training (a fine-tune can't produce a checkpoint trajectory starting from random init), the full training code, and per-source data accounting.

@RoRoSeSe_ @Karl_Lauterbach Ist es für deine Kunden kein Problem Unternehmensdaten an US/China zu geben? Oder nutzen diese "on-premise open-weights Modelle"?

Die chinesischen Modelle sind alle open-weights, nicht open-source: keine Trainingsdaten, kein Trainingscode. Soofi released Gewichte, Checkpoints, Code und die komplette Datenbilanz (OSAID 1.0-konform, Details inkl. der einen Ausnahme im Tech Report). In der Kategorie tatsächlich offener Modelle sind wir state-of-the-art. Wollen wir uns in Deutschland/Europa darauf verlassen das künftig China/USA für uns Open Weights Modelle baut? Das war jetzt der erste (erfolgreiche) Versuch in der Größe. Aktuell trainiert auch schon Soofi L - 120B-A12B. (4x größer)

Stimmt, bei reiner Benchmark-Leistung liegt GLM 5.2 vorn. Aber GLM ist open-weights, nicht open-source: keine Trainingsdaten, kein Trainingscode. Soofi released Gewichte, Checkpoints, Code und die komplette Datenbilanz (OSAID 1.0-konform, Details inkl. der einen Ausnahme im Tech Report). Wir werden auch noch größere Modelle trainieren, aktuell trainiert Soofi L 120B-A12B. Wir planen aber nochmals größere Modelle - stay tuned :)

Hi, bitte ne Mail an [email protected] mit dem Anwendungszenario im Unternehmen und deinem HF Username. Dann hier access request stellen: huggingface.co/collections/So… Falls möglich gerne das Feedback an [email protected] schicken, wir werden das Feedback in der zweiten Posttraining Iteration einfließen lassen!

@meyer_peace18 @a_ratzenberger Alle Datenquellen sind im Tech Report genau aufgelistet unter Tabelle 6/8 im Appendix.

📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with deliberately up-weighted German. What’s in the report: 🏆 Strongest fully open model in our evaluations on BOTH the English and German aggregates — ahead of Olmo 3 32B and Apertus 70B (full methodology in the report) 📋 Radical transparency: complete per-source data accounting, all hyperparameters, training + eval code, checkpoints — everything under permissive licenses 🇩🇪 Trained end-to-end on Deutsche Telekom’s Industrial AI Cloud in Munich — sovereign AI infrastructure on German soil Soofi S combines frontier-level capability with the highest measured aggregate long-context decode TPS, and unlike full-attention dense baselines maintains high throughput as context grows. The figure plots Capability Index versus measured aggregate decode TPS/GPU at 40K context and batch 32. The Capability Index averages five benchmark groups, i.e., Code, GSM8K, GPQA-Diamond, English aggregate, and German aggregate, after normalizing each group to the best plotted model. Aggregate decode TPS/GPU is measured with a TP=1, one-B200 vLLM latency-subtraction protocol.

@eliebakouch @kimmonismus There is a new post on Evaluation x.com/maxidahl/statu… We basically used the Olmo 3 Evaluation suite and our evaluation stack is fully open source (in contrast to the evaluation suite from Nemotron imho)

Max Idahl @maxidahl

21 hours ago

Tech report on Soofi-S-base is out: A 30B-A3B Mixture-of-Experts Model based on @NVIDIAAI's Nemotron 3 architecture, trained on 27T tokens with a strong focus on German. Supports up to 1M context length. Its a decent model and a solid first step. And SOOFI-L-base is already

keyboard_arrow_left Previous keyboard_arrow_right Next

4 4 25 9K 10

@eliebakouch @kimmonismus That is the updated Figure, we had an old placeholder value inside and already mentioned it under the initial post.

Hi, bitte ne Mail an [email protected] mit dem Anwendungszenario im Unternehmen und deinem HF Username. Dann hier access request stellen: huggingface.co/collections/So… Falls möglich gerne das Feedback an [email protected] schicken, wir werden das Feedback in der zweiten Posttraining Iteration einfließen lassen! Aktuell läuft schon das Training von Soofi L 120B-A12B in NVFP 4!

@MTorygreen Here is the original post from the team. Happy to answer questions :) x.com/effi288/status…

Dr. Michael Fromm @effi288

5 days ago

📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with

85 170 1K 355K 490

Kleine, aber wichtige Korrektur: Soofi ist kein nachtrainiertes Nemotron. Wir haben von Grund auf trainiert, 27 Billionen Tokens, eigene Datenmischung, komplett auf deutscher Infrastruktur. Übernommen haben wir die offene Architektur (wie jedes moderne Modell auf der Transformer-Architektur von Google aufbaut). Ziel war aber, die komplette Trainings-Pipeline in Europa aufzubauen. 'Bahnbrechend' behaupten wir nicht; 'erstmals in Deutschland auf diesem Niveau machbar' schon. Alle Details offen: x.com/effi288/status…

Dr. Michael Fromm @effi288

5 days ago

📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with

85 170 1K 355K 490