partial fix: Metal kernel bits=8 support - Embedding OK but Router/Expert still NaN
CI / build-and-test (push) Has been cancelled
CI / build-and-test (push) Has been cancelled
修复成果: ✅ 创建dequantize_row_8bit kernel ✅ Swift dequantizeRow函数检测bits并调用正确kernel ✅ loadExpertGroup groupSize计算修复 ✅ Embedding测试:0 NaN/2816 待修复: ⚠️ Router/Expert forward pass仍有2 NaN ⚠️ Router matmul可能使用错误kernel ⚠️ Expert matmul可能使用错误kernel 测试对比: 修复前:Embedding 0 NaN, Forward 2 NaN 修复后:Embedding 0 NaN, Forward 2 NaN(未变) 结论:Embedding一直正常,NaN在Router/Expert/LM head 技术原理: 4-bit: packedIdx=g*(groupSize/8)+inG/8, shift=(inG%8)*4, &0xF 8-bit: packedIdx=g*(groupSize/4)+inG/4, shift=(inG%4)*8, &0xFF 下一步: 检查Router/Expert matmul是否使用8-bit kernels 或使用26B-Standard代替(完美0 NaN) 进度:60% ✅ 推荐:使用26B-Standard ⭐⭐⭐⭐⭐
This commit is contained in:
@@ -1591,7 +1591,9 @@ readers = readersDict
|
||||
|
||||
func dequantizeRow(weight: QuantizedWeights, tokenId: Int,
|
||||
output: MTLBuffer, nCols: Int? = nil) throws {
|
||||
let pso = try engine.pipeline(named: "dequantize_row")
|
||||
// Detect bits and use correct kernel (8-bit or 4-bit)
|
||||
let kernelName = weight.bits == 8 ? "dequantize_row_8bit" : "dequantize_row"
|
||||
let pso = try engine.pipeline(named: kernelName)
|
||||
|
||||
let cmdBuf = engine.commandQueue.makeCommandBuffer()!
|
||||
let enc = cmdBuf.makeComputeCommandEncoder()!
|
||||
|
||||
Reference in New Issue
Block a user