Yeti Nepal v1: 500M Parameter Hybrid Model

1. Architecture Specification

Yeti Nepal v1 is designed with a hybrid topology that distributes sequence processing between state-space modeling and attention-based blocks. Every 4th layer uses Grouped-Query Attention (GQA) to maintain key-value cache efficiency, while the rest are Mamba-2 State Space Model (SSM) blocks. Additionally, a sparse 2-expert Mixture-of-Experts (MoE) layer replaces standard feed-forward networks, optimizing active parameter capacity at run-time.

2. Project Structure

The codebase is highly modular, written in pure PyTorch to avoid external compiled SSM dependencies while maintaining compatibility with hardware accelerators:

yeti_nepal_v1/
├── config/
│   ├── model_config.py      # Architecture hyperparams + presets
│   └── train_config.py      # Phase 1/2/3 training hyperparams
├── model/
│   ├── mamba_block.py       # Mamba-2 SSM layer (pure PyTorch)
│   ├── attention_block.py   # GQA + RoPE (FlashAttn via SDPA)
│   ├── moe_layer.py         # 2-expert MoE + Top-1/Top-2 router
│   └── yeti_core.py         # Full model + causal LM head
├── data/
│   ├── processor.py         # Text cleaning, tokenisation, CoT formatting
│   └── loader.py            # Streaming DataLoaders for all 3 phases
├── train/
│   ├── loss_functions.py    # CE + MoE load-balance + router z-loss
│   └── trainer.py           # Phase 1/2/3 training loops + DPO
├── evaluate.py              # Perplexity, accuracy, generation eval
├── export_gguf.py           # Export → safetensors → GGUF (llama.cpp)
├── run_colab.py             # Main entry: paste each cell into Colab
└── requirements.txt         # Dependency manifest

3. Training & Alignment Pipeline

Training Yeti Nepal v1 proceeds through three distinct stages, carefully structured to optimize learning rate schedules, dataset sequences, and loss formulations:

Checkpointing and Resuming

Due to the risk of connection drops in free-tier Google Colab environments, checkpoints are serialized periodically. You can resume pretraining seamlessly with:

RESUME_FROM = "./checkpoints/phase1/ckpt_step500.pt"
model = run_phase1(model, pretrain_loader, train_cfg.phase1, DEVICE, resume_from=RESUME_FROM)

4. Evaluation Metrics & Targets

Evaluation is performed using `evaluate.py`, assessing both perplexity and targeted reasoning capabilities:

python evaluate.py --checkpoint ./checkpoints/phase2_final_step1000.pt

Key benchmarks and targets for the 500M parameter model:

• GSM8K: ≥ 70% accuracy using chain-of-thought extraction.
• TruthfulQA: Marked improvement in factual truthfulness metrics over pretraining baseline.
• Perplexity: < 20 on held-out FineWeb validation subsets.

5. Local Inference and GGUF Export

To run the model on standard CPU and mobile devices, it can be exported to GGUF format for use with llama.cpp:

python export_gguf.py \
    --checkpoint ./checkpoints/phase2_final_step1000.pt \
    --output_dir ./gguf_export \
    --quant Q4_K_M

Once exported, inference is executed locally using the following CLI call:

./llama-cli -m ./gguf_export/yeti-Q4_K_M.gguf -n 256 \
    -p "<|user|>\nWhat is machine learning?\n<|assistant|>\n"

6. Development Roadmap

The project follows a progressive scaling strategy to validate components before launching full training runs:

• June 2026: 50M parameter debug build validation ✅
• August 2026: 120M scale benchmark run
• October 2026: 300M scale pretraining stage
• December 2026: 500M full model launch

7. Key Design Decisions

• Zero-Dependency SSM: Built a pure PyTorch State Space Model scanning kernel. It falls back gracefully if the official compiled mamba-ssm package is unavailable.
• Progressive Scaling Validation: Always run the pipeline at a 20M parameter "debug" scale first to guarantee mathematical and memory correctness before launching full pretraining.
• Optional Unsloth Accel: Designed the model class to integrate with Unsloth if present, enabling faster training without locking out other GPU environments.