MarkBase Admin 31427770b1 v2: Apply tokenizer UTF-8 fix + Engine writeFloats helper ...

- Tokenizer fix: collect <0xXX> bytes and decode as UTF-8
  (fixes Chinese/non-ASCII character decoding)
- BPETokenizer + HuggingFaceTokenizer: both updated
- Engine.swift: added writeFloats() utility method
- FloatWeights struct added to Layer.swift (bf16 support)
- attnQBits/KBits/VBits/OBits detection added to Model.swift
- bf16 layer weight support from commit 48c0347 cherry-picked

2026-07-05 13:41:48 +08:00

..

BPETokenizer.swift

v2: Apply tokenizer UTF-8 fix + Engine writeFloats helper

2026-07-05 13:41:48 +08:00

HuggingFaceTokenizer.swift

v2: Apply tokenizer UTF-8 fix + Engine writeFloats helper

2026-07-05 13:41:48 +08:00

SentencePieceTokenizer.swift

v2: Initial clean branch with unit tests + CI/CD pipeline

2026-07-05 13:29:25 +08:00

Tokenizer.swift

v2: Initial clean branch with unit tests + CI/CD pipeline

2026-07-05 13:29:25 +08:00