warren/markbaseengine
1
0
Fork 0
Code Issues Pull Requests Actions 14 Packages Projects Releases Wiki Activity
31 Commits 2 Branches 0 Tags
319f29bf694612175f028fafd1b58277520f3f4c
Commit Graph

3 Commits

Author SHA1 Message Date
MarkBase Admin 319f29bf69 v2: fix multi-language support - use SentencePieceTokenizer for GemmaTokenizer
CI / unit-tests (push) Blocked by required conditions
Details
CI / build (push) Waiting to run
Details
CI / lint (push) Blocked by required conditions
Details
2026-07-06 18:22:59 +08:00
MarkBase Admin 31427770b1 v2: Apply tokenizer UTF-8 fix + Engine writeFloats helper
CI / build (push) Has been cancelled
Details
CI / unit-tests (push) Has been cancelled
Details
CI / lint (push) Has been cancelled
Details 
- Tokenizer fix: collect <0xXX> bytes and decode as UTF-8
  (fixes Chinese/non-ASCII character decoding)
- BPETokenizer + HuggingFaceTokenizer: both updated
- Engine.swift: added writeFloats() utility method
- FloatWeights struct added to Layer.swift (bf16 support)
- attnQBits/KBits/VBits/OBits detection added to Model.swift
- bf16 layer weight support from commit 48c0347 cherry-picked
 2026-07-05 13:41:48 +08:00
MarkBase Admin 8a66b9086a v2: Initial clean branch with unit tests + CI/CD pipeline
CI / build (push) Has been cancelled
Details
CI / unit-tests (push) Has been cancelled
Details
CI / lint (push) Has been cancelled
Details 
- Started from ac75faa (initial E4B-MarkBase integration)
- Kept Sources/ (all engine code) + Package.swift + .gitignore
- Removed all ad-hoc tests, documentation, scripts, Python files
- Added Tests/00_Unit/ (MathTest, TokenizerTest, SamplerTest)
- Added .gitea/workflows/ci.yaml (build + unit tests + lint)
- Added Scripts/check_resources.sh (memory-aware test runner)
- Added Tests/Manifest.json (resource requirements for all tests)
- Focus: 4-bit quantized models only
 2026-07-05 13:29:25 +08:00