BM25 Search with Chinese Synonym Expansion Completed #2

New Issue

warren · 2026-03-30T00:20:36+08:00

warren commented

2026-03-30 00:20:36 +08:00

Status Report: BM25 Search Implementation Completed

✅ Completed Features

BM25 API Integration
- New endpoints: /api/v1/search/bm25 and /api/v1/n8n/search/bm25
- Full API key authentication support
- Tested on port 3003 (development server)
Chinese Text Processing
- OpenCC integration for Traditional/Simplified Chinese conversion
- Jieba tokenizer for Chinese text segmentation
- Proper handling of already-tokenized database content
Synonym Expansion Infrastructure
- Full synonym expansion mechanism implemented
- Support for multiple JSON synonym files via environment variables
- Intelligent query expansion with OR logic grouping
- Copyright compliant: Default empty mapping, users provide own data
Technical Foundation
- Fixed tokenization issue for multi-character words
- Proper TSQUERY generation for PostgreSQL full-text search
- Comprehensive unit tests and integration tests
- All linting and formatting checks pass

🔧 Configuration Status

Component	Status	Notes
BM25 Search	✅ Ready	API endpoints live on port 3003
Synonym Expansion	✅ Ready (disabled by default)	Users must provide their own synonym files
OpenCC Conversion	✅ Ready	Automatic Simplified→Traditional conversion
Authentication	✅ Ready	Requires `X-API-Key` header

📋 Next Steps for Users

To enable synonym expansion, users need to:

Create custom synonym JSON file (format: {"word": ["syn1", "syn2"]})

Set environment variable:

export MOMENTRY_SYNONYM_FILE=/path/to/your/synonyms.json

Restart application

Example file available at: docs/examples/custom_synonyms.json

📚 Documentation

Main guide: docs/SYNONYM_CONFIGURATION.md - Complete configuration instructions
Environment setup: .env.development updated with example paths
API documentation: BM25 endpoints documented in the configuration guide

⚠️ Important Notes

No default synonym data - System starts with empty mapping for copyright compliance
Hybrid search 500 error - Not addressed per instructions (not currently used)
Database stores Traditional Chinese - Queries automatically convert Simplified→Traditional
All existing search endpoints continue working - Vector-only search unaffected

Configuration: .env.development (updated with synonym file instructions)
Documentation: docs/SYNONYM_CONFIGURATION.md (complete user guide)
Example file: docs/examples/custom_synonyms.json (template)
Core implementation: src/core/text/synonym_expander.rs
Database integration: src/core/db/postgres_db.rs
API endpoints: src/api/server.rs

The system is now ready for production use with BM25 text search. Synonym expansion can be enabled at any time by customers providing their own synonym files.

## Status Report: BM25 Search Implementation Completed ### ✅ Completed Features 1. **BM25 API Integration** - New endpoints: `/api/v1/search/bm25` and `/api/v1/n8n/search/bm25` - Full API key authentication support - Tested on port 3003 (development server) 2. **Chinese Text Processing** - OpenCC integration for Traditional/Simplified Chinese conversion - Jieba tokenizer for Chinese text segmentation - Proper handling of already-tokenized database content 3. **Synonym Expansion Infrastructure** - Full synonym expansion mechanism implemented - Support for multiple JSON synonym files via environment variables - Intelligent query expansion with OR logic grouping - **Copyright compliant**: Default empty mapping, users provide own data 4. **Technical Foundation** - Fixed tokenization issue for multi-character words - Proper TSQUERY generation for PostgreSQL full-text search - Comprehensive unit tests and integration tests - All linting and formatting checks pass ### 🔧 Configuration Status | Component | Status | Notes | |-----------|--------|-------| | BM25 Search | ✅ **Ready** | API endpoints live on port 3003 | | Synonym Expansion | ✅ **Ready (disabled by default)** | Users must provide their own synonym files | | OpenCC Conversion | ✅ **Ready** | Automatic Simplified→Traditional conversion | | Authentication | ✅ **Ready** | Requires `X-API-Key` header | ### 📋 Next Steps for Users To enable synonym expansion, users need to: 1. **Create custom synonym JSON file** (format: `{"word": ["syn1", "syn2"]}`) 2. **Set environment variable**: ```bash export MOMENTRY_SYNONYM_FILE=/path/to/your/synonyms.json ``` 3. **Restart application** **Example file** available at: `docs/examples/custom_synonyms.json` ### 📚 Documentation - **Main guide**: `docs/SYNONYM_CONFIGURATION.md` - Complete configuration instructions - **Environment setup**: `.env.development` updated with example paths - **API documentation**: BM25 endpoints documented in the configuration guide ### ⚠️ Important Notes - **No default synonym data** - System starts with empty mapping for copyright compliance - **Hybrid search 500 error** - Not addressed per instructions (not currently used) - **Database stores Traditional Chinese** - Queries automatically convert Simplified→Traditional - **All existing search endpoints continue working** - Vector-only search unaffected ### 🔗 Related Files - Configuration: `.env.development` (updated with synonym file instructions) - Documentation: `docs/SYNONYM_CONFIGURATION.md` (complete user guide) - Example file: `docs/examples/custom_synonyms.json` (template) - Core implementation: `src/core/text/synonym_expander.rs` - Database integration: `src/core/db/postgres_db.rs` - API endpoints: `src/api/server.rs` **The system is now ready for production use with BM25 text search. Synonym expansion can be enabled at any time by customers providing their own synonym files.**

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: warren/momentry_core#2