Industry News

Baidu Open-Sourced a 280B Base Model: Is Raw Pretraining the New AI Flex?

Baidu released ERNIE 5.5 Flash Base: 280B parameters, 14B active per token, 33.5 trillion training tokens, FP8 and BF16 weights. No chat tuning, no permissive licence, and a benchmark table Baidu ran on itself.

On 5 November, Baidu published the weights of ERNIE 5.5 Flash Base on Hugging Face. The headline specification is a sparse mixture-of-experts model with 280 billion total parameters that activates 14 billion of them for each token it processes. Baidu says it was pretrained from scratch on 33.5 trillion tokens, with a training context extended to 128,000 tokens and support for up to 1 million tokens at fine-tuning and inference time.

That is a large model. It is also, deliberately, not a product. The checkpoint Baidu uploaded is the state of the network at the end of pretraining — before mid-training, before instruction tuning, before any reinforcement learning. There is no chat template, no refusal behaviour, no tool-calling convention. Plug it into a chat window and you get an autocomplete engine with an encyclopaedic memory and no manners.

That choice is the story. Releasing a base model is a different commercial and technical move from releasing a chatbot, and reading the release as if it were the latter is how people end up disappointed by a model that was never asked to be helpful.

What Baidu actually put on the disk

The arithmetic first, because it determines who can run this at all. 280 billion parameters stored in FP8 is roughly 280 GB of weights before you allocate a single byte of KV cache. The BF16 build doubles that. Neither fits on a workstation, and the FP8 path is the realistic one for a small cluster.

Baidu published both formats as safetensors, and says SGLang and vLLM can serve the model through a completions API. One implementation detail from the setup guide is worth flagging to anyone who tries it: every input must begin with a beginning-of-sequence token. Skip it and the model will not fail loudly — it will just produce worse output, which is a much harder bug to find.

The context claims need separating. A 128,000-token training context is a pre-training fact. The 1-million-token figure is a capability Baidu says the architecture supports at fine-tuning and inference time, and it is a claim nobody outside Baidu has measured yet. Long-context quality at that length is exactly the kind of number that looks fine in a config file and falls apart on real documents, so treat the 1M as an invitation to test rather than a specification.

What is not in the release matters just as much. There is no post-trained sibling yet. Baidu says a post-trained ERNIE 5.5 Flash will follow, without a date. Until it does, everything anyone says about this model's conversational quality is speculation.

A base checkpoint is a different product, not a weaker one

The intuitive reading of “base model” is “unfinished model”, and that reading is wrong in a specific, practical way.

A base checkpoint contains no one else's decisions. No house style, no safety classifier's edge cases baked into the weights, no opinion about how long an answer should be. For a team building a model for legal document review, a low-resource language, or an internal support workflow, that is the valuable property. You are not fine-tuning around someone else's personality; you are starting from a clean distribution and teaching it yours.

The cost side is equally concrete. Instruction tuning on top of a base model is not a small job: you need supervised data, usually a preference or RL stage after it, and, for mixture-of-experts architectures specifically, more tuning experience than a dense model demands because routing behaviour is easy to disturb. Baidu is effectively publishing the expensive first 33.5 trillion tokens and leaving the expensive last mile to whoever has the domain data.

This is a coherent strategy, and it is not unique to Baidu. It is also the reason the release's benchmark table compares base against base rather than against chat models. Cross-comparing a base checkpoint with a tuned assistant tells you almost nothing.

The benchmark table, and the three numbers worth quoting

Baidu evaluated the checkpoint against three other recently released open base models of similar size: DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base. The company says all four were run through the same harness, the same prompts, the same few-shot examples and the same LLM judge — which is the right method and also the method that is easiest to get subtly wrong.

The three figures that carry the most signal:

  • MMLU at 93.3 for ERNIE against 89.4 for DeepSeek-V4-Flash-Base, on a five-shot setup.
  • HumanEval+ pass@1 at 82.3 against 69.5. That is a gap of nearly 13 points on a code benchmark, which is a large enough difference that it should either replicate independently or be explained by something in the harness.
  • LiveCodeBench pass@1 at 40.1 against 31.9. LiveCodeBench is the more interesting of the two code numbers precisely because its problems are drawn from recent contests, which limits how much of the score can come from memorisation.

Now the caveat, stated as plainly as it deserves. These are vendor-run numbers from a single harness that Baidu configured, published by the company making the claim, with no independent leaderboard entry to check them against. The direction of the result is plausible — a newer base checkpoint beating older base checkpoints is the expected outcome — but the magnitude is unaudited. Anyone making a procurement decision on a 13-point HumanEval+ gap is making it on marketing until someone re-runs the eval.

The honest way to use this table is as a shortlist generator, not a verdict. It tells you three or four models worth putting on your own evaluation set. It does not tell you which one will win there.

Where the architecture is doing the real work

The specification sheet is the least interesting part of this release. The design choices are where the engineering is, and they are all pointed at the same problem: long context is expensive, and most of the cost is waste.

Baidu describes the attention stack as Latent-Gated Sparse Attention, which combines three ideas. The first is a split between two attention paths: one that operates over a compressed representation of history, and one that can still reach individual historical positions when precision matters. Most tokens never need the second path, so paying for it everywhere is the waste being eliminated.

The second is a low-rank gate. Instead of learning a separate projection just to decide what to let through, the model reuses a low-rank latent it already computes for attention. Baidu puts the saving at a 75% reduction in gating projection parameters. That is a vendor figure, but the design logic is sound and the direction is the one the whole field has been moving in.

The third is a virtual head attention trick that improves expressiveness during training and then gets fused into the attention kernel at inference time, so it costs close to nothing to serve.

The routing design is the other half. Baidu calls it Multi-Router Latent MoE: several independent routers vote on which experts to activate, rather than one router deciding alone. The stated goal is to decorrelate routing scores from actual expert usefulness — a single router can develop a systematic preference for experts that score well but contribute little. Baidu says the number of active experts, the expert compute and the communication volume are all unchanged, which is the important part: this is a quality intervention, not a capacity intervention.

The training objective is the piece I find most interesting. Baidu introduces ECHO, which drops the usual auxiliary multi-token-prediction module and instead reuses the backbone's own final layer to predict subsequent tokens. The model learns to plan a sequence's direction using hardware it already has. Baidu says it trained only a single ECHO step, and that the reused layer then works as a draft model for speculative decoding — meaning the same parameters that improve reasoning also make inference cheaper. If that holds up outside Baidu's own tests, it is a genuinely elegant piece of engineering.

Secondary write-ups describe a 43-layer transformer; I have not read Baidu's technical report directly, so treat the layer count as second-hand.

The licence is the part to read twice

The weights ship under the ERNIE 5.5 Flash Base Model License Agreement v1.0. That is Baidu's own agreement, not Apache 2.0 and not MIT, and the distinction is not cosmetic.

Custom model licences typically carry conditions that a permissive licence does not: acceptable-use clauses that bind downstream users, attribution requirements, thresholds that trigger different terms at scale, or restrictions that interact with local regulation. None of that is unreasonable for a model this size, and none of it is unusual in 2026. But “open weights” and “open source” are different claims, and this release supports the first, not the second.

The practical instruction is boring and gets skipped constantly: have someone who reads licences read this one before it goes anywhere near a commercial pipeline. The cost of that hour is trivial against the cost of discovering a clause after you have shipped.

Why labs keep shipping bases instead of chatbots

Three reasons, and they compound.

The first is that a base release cannot be benchmarked as an assistant, so it cannot lose an assistant comparison. A frontier lab that publishes a chat model invites direct comparison with every other chat model on the same day. A base checkpoint invites comparison on pre-training efficiency, where the metrics are chosen by the publisher.

The second is that it outsources the alignment work. Post-training is where safety behaviour, refusal quality and tool use get decided. Publishing before that stage puts those decisions in the hands of downstream users, which is defensible on openness grounds and convenient on liability grounds simultaneously.

The third is ecosystem. Every team that fine-tunes on your base becomes a party with an interest in your tokenizer, your serving stack and your next release. Weights are cheap to distribute and expensive to switch away from.

None of this makes the release cynical: the base checkpoint genuinely is the more useful artefact for a certain class of buyer, and aligned incentives do not make it worse. It does mean that “Baidu open-sourced a 280B model” is a headline that needs a second clause attached to it every time.

For teams doing that comparison, the practical problem is not model quality. It is that the number that decides the project — cost per finished task, after fine-tuning, serving and retries — is not on any leaderboard. A 14B-active model with a clean base and an awkward licence can be the right answer for a narrow domain and the wrong answer for a general assistant, and the benchmark table will not tell you which. The same reasoning is what aibookmarks.net applies to the rest of the stack: it organises AI tools by the specific job they do and the cost of getting that job done, rather than by launch date or ranking, so a comparison starts from your workload instead of from someone else's scoreboard. That framing matters more in a week when a 280B base model and a 15-second video model both count as “new AI” and are not remotely substitutes for each other.

The honest limits

  • The checkpoint is pretrained only. Chat quality, instruction following and refusal behaviour are unknown properties that depend entirely on whoever post-trains it.
  • Every benchmark number is Baidu's, from Baidu's harness. No independent replication exists as of publication.
  • The 1-million-token context is a stated capability, not a measured one. Long-context quality at that length is untested publicly.
  • The licence is bespoke. Commercial use is not simply permitted by default; it is governed by an agreement with conditions.
  • The post-trained version, which is what most people actually want, has no release date.

What to watch

Two things will decide whether this release matters beyond the week it landed.

The first is independent evaluation. If a third party reproduces the HumanEval+ and LiveCodeBench gaps on the same base-model footing, Baidu has a genuine pre-training efficiency result. If the gaps shrink under a neutral harness, this becomes another vendor table, and the field's habit of quoting them unverified gets a little worse.

The second is what the community builds on top. A base checkpoint's value is revealed by the fine-tunes it attracts, and those take weeks to appear. Watch whether domain-specific ERNIE 5.5 Flash variants show up with published recipes and honest evaluation — that is the signal that the base is actually workable, in a way no launch table can demonstrate.

In the meantime the correct posture is the boring one. Put it on the shortlist, run your own eval, read the licence, and do not plan around a post-trained model that does not exist yet.

---

Original author: Baidu's ERNIE team, from the ERNIE 5.5 Flash Base release notes (5 November 2026) and the model card on Hugging Face, with benchmark figures as published by Baidu. Compiled and rewritten by the AI Bookmarks research desk. Vendor-reported numbers are labelled as such throughout.

Published by AI Bookmarks · More AI news →

← Back to News