Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

In-Place Tokenizer Expansion for Pre-trained LLMs

Source

arXiv

arxiv.org/abs/2607.15232v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti, Tim Seyde, Maxime Labonne, Alexander Amini, Mathias Lechner
  • Venue: arXiv: cs.CL
  • Published: 2026-07-16
  • Collected: 2026-07-17
  • URL: http://arxiv.org/abs/2607.15232v1
  • PDF: https://arxiv.org/pdf/2607.15232v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers have developed an in-place method called Tokenizer Expansion to upgrade a pre-trained model's tokenizer when language priorities shift, focusing on improving efficiency for users of languages added later. The method is applied to LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to produce LFM2.5-8B-A1B with a larger tokenizer. This new tokenizer uses fewer tokens for Hindi and Vietnamese by up to $4.0 imes$ compared to the source, leading to a potential speedup of $2.2$-$3.7 imes$ in decode speed for these languages on reference devices.

What the Document Says

  • The paper presents Tokenizer Expansion, an in-place method to upgrade a pre-trained model's tokenizer when the model producer controls its design.
  • The method is applied to LFM2-8B-A1B to produce LFM2.5-8B-A1B with a larger tokenizer.
  • The expanded tokenizer uses fewer tokens for Hindi and Vietnamese by up to $4.0 imes$ compared to the source.
  • This leads to a potential speedup of $2.2$-$3.7 imes$ in decode speed for these languages on reference devices.

Key Concepts Mentioned

  • Tokenizer Expansion
  • Pre-trained models
  • Language priorities
  • LFM2-8B-A1B
  • LFM2.5-8B-A1B
  • Hindi
  • Vietnamese

Claims Made by the Document

  • Tokenizer Expansion can upgrade a pre-trained model's tokenizer.
  • The method results in fewer tokens for Hindi and Vietnamese using the expanded tokenizer.
  • The larger vocabulary of the expanded tokenizer leads to a potential speedup in decode speed for these languages.

Evidence Used

  • The text mentions that Tokenizer Expansion is an in-place method used to upgrade a pre-trained model's tokenizer.
  • It states that the method was applied to LFM2-8B-A1B to produce LFM2.5-8B-A1B with a larger tokenizer.
  • The text notes that Hindi and Vietnamese use fewer tokens with the expanded tokenizer compared to the source.
  • It mentions that this leads to a potential speedup in decode speed for these languages.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.