Thank you for your interest in contributing to Kagome C++! This document provides guidelines for contributing to the project.
- Code of Conduct
- Getting Started
- Development Environment
- Coding Standards
- Testing
- Pull Request Process
- Issue Reporting
- Documentation
We are committed to providing a welcoming and inclusive environment for all contributors. Please be respectful and constructive in all interactions.
- C++23 compatible compiler: GCC 12+, Clang 15+, or MSVC 2022+
- CMake 3.25+
- libicu-dev: Unicode support
- libfmt-dev: String formatting
- libarchive-dev: Archive handling
- pkg-config: Build configuration
# Fork the repository on GitHub, then clone your fork
git clone https://github.com/YOUR_USERNAME/kagome-cxx.git
cd kagome-cxx
# Add upstream remote
git remote add upstream https://github.com/rspamd/kagome-cxx.gitmkdir build && cd build
cmake ..
make -j$(nproc)
# Run tests
ctest
./kagome_tests- IDE: Visual Studio Code with C++ extensions, CLion, or similar
- Debugger: GDB, LLDB, or IDE integrated debugger
- Static Analysis: clang-tidy, cppcheck
- Memory Checking: AddressSanitizer, Valgrind
- Formatting: clang-format (config included)
# Debug build (default)
cmake -DCMAKE_BUILD_TYPE=Debug ..
# Release build
cmake -DCMAKE_BUILD_TYPE=Release ..
# Debug with sanitizers
cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_SANITIZER=ON ..# Ubuntu/Debian
sudo apt install build-essential cmake pkg-config
sudo apt install libfmt-dev libicu-dev libarchive-dev
sudo apt install clang-tidy clang-format
# macOS
brew install cmake fmt icu4c libarchive pkg-config
brew install llvm # for clang-tidy and clang-format- C++23 Standard: Use modern C++23 features where appropriate
- Concepts: Use concepts for template constraints
- Ranges: Prefer ranges over traditional iterator-based loops
- fmt formatting: Use kagome::format (fmt::format) for string formatting instead of printf/iostream
- std::optional: Use for nullable values
- Smart Pointers: Use RAII and smart pointers for memory management
We use clang-format for code formatting. The configuration is in .clang-format.
# Format all source files
find src include -name "*.cpp" -o -name "*.hpp" | xargs clang-format -i
# Check formatting
find src include -name "*.cpp" -o -name "*.hpp" | xargs clang-format --dry-run --Werror- Classes/Structs:
PascalCase(e.g.,TokenizerConfig) - Functions/Methods:
snake_case(e.g.,tokenize_text()) - Variables:
snake_case(e.g.,token_count) - Constants:
UPPER_CASE(e.g.,MAX_TOKEN_LENGTH) - Namespaces:
snake_case(e.g.,kagome::tokenizer) - Files:
snake_case(e.g.,tokenizer.hpp,lattice.cpp)
// File: include/kagome/tokenizer/tokenizer.hpp
#pragma once
#include <memory>
#include <string>
#include <vector>
#include <optional>
// Third-party includes
#include <fmt/format.h>
// Project includes
#include "kagome/dict/dict.hpp"
#include "kagome/tokenizer/token.hpp"
namespace kagome::tokenizer {
class Tokenizer {
public:
// Public interface first
explicit Tokenizer(std::shared_ptr<dict::Dict> dictionary);
auto tokenize(const std::string& text) -> std::vector<Token>;
private:
// Private members last
std::shared_ptr<dict::Dict> dict_;
};
} // namespace kagome::tokenizer// Good: Use kagome::format (which is fmt::format)
auto message = kagome::format("Processing {} tokens", token_count);
// Avoid: printf-style formatting
char buffer[256];
sprintf(buffer, "Processing %d tokens", token_count);// Define concepts for template constraints
template<typename T>
concept StringLike = requires(T t) {
{ t.c_str() } -> std::convertible_to<const char*>;
{ t.size() } -> std::convertible_to<size_t>;
};
template<StringLike S>
auto process_text(const S& text) -> std::vector<Token> {
// Implementation
}// Use std::optional for nullable returns
auto find_token(const std::string& surface) -> std::optional<Token>;
// Use exceptions for exceptional cases
class TokenizerError : public std::runtime_error {
using std::runtime_error::runtime_error;
};
// Use expected<T, E> when available (C++23)
auto parse_features(const std::string& data) -> std::expected<Features, ParseError>;We use a simple custom test framework. Tests are located in the tests/ directory.
// tests/new_feature_test.cpp
#include "test_framework.hpp"
#include "kagome/tokenizer/tokenizer.hpp"
TEST_CASE("Tokenizer handles empty input") {
auto dict = create_test_dictionary();
kagome::tokenizer::Tokenizer tokenizer(dict);
auto tokens = tokenizer.tokenize("");
REQUIRE(tokens.empty());
}
TEST_CASE("Tokenizer processes Japanese text") {
auto dict = create_test_dictionary();
kagome::tokenizer::Tokenizer tokenizer(dict);
auto tokens = tokenizer.tokenize("こんにちは");
REQUIRE(tokens.size() == 1);
REQUIRE(tokens[0].surface() == "こんにちは");
}# Build and run all tests
make kagome_tests
./kagome_tests
# Run specific test
./kagome_tests --filter "Tokenizer"
# Run with verbose output
./kagome_tests --verbose
# Memory checking
valgrind --tool=memcheck ./kagome_tests- Unit Tests: Test individual components in isolation
- Integration Tests: Test component interactions
- Regression Tests: Ensure compatibility with original kagome
- Performance Tests: Benchmark critical paths
- Rspamd Integration Tests: Test plugin functionality
- Rebase on latest main:
git rebase upstream/main - Run tests: Ensure all tests pass
- Check formatting: Run clang-format
- Static analysis: Run clang-tidy
- Update documentation: Update relevant docs
- Clear title: Describe what the PR does
- Detailed description: Explain the problem and solution
- Link issues: Reference related issues with
#issue_number - Small focused changes: Keep PRs focused on a single concern
- Tests included: Add tests for new functionality
- Documentation updated: Update docs for user-facing changes
type(scope): brief description
Detailed explanation of the changes, including:
- What was changed and why
- Any breaking changes
- References to issues
Closes #123
Types: feat, fix, docs, style, refactor, test, chore
Examples:
feat(tokenizer): add support for custom user dictionaries
- Implement UserDict class for loading custom dictionaries
- Add priority-based dictionary merging
- Include comprehensive tests for user dictionary functionality
Closes #45
fix(rspamd): handle dictionary loading failures gracefully
Previously, dictionary loading failures would crash the plugin.
Now we fall back to a minimal dictionary and log warnings.
Fixes #67
Please include:
- Environment: OS, compiler version, dependencies
- Steps to reproduce: Minimal example demonstrating the issue
- Expected behavior: What should happen
- Actual behavior: What actually happens
- Additional context: Logs, stack traces, etc.
Please include:
- Use case: Why is this feature needed?
- Proposed solution: How should it work?
- Alternatives considered: Other approaches you've considered
- Additional context: Examples, references, etc.
- Use Doxygen-style comments for public APIs
- Include usage examples
- Document parameters, return values, and exceptions
/**
* @brief Tokenizes Japanese text into morphological units
*
* @param text The input text to tokenize (UTF-8 encoded)
* @param mode The tokenization mode (Normal, Search, Extended)
* @return Vector of tokens with morphological information
* @throws TokenizerError if dictionary is not loaded
*
* @example
* ```cpp
* auto tokens = tokenizer.tokenize("こんにちは世界");
* for (const auto& token : tokens) {
* std::cout << token.surface() << "\n";
* }
* ```
*/
auto tokenize(const std::string& text,
TokenizeMode mode = TokenizeMode::Normal) -> std::vector<Token>;- Keep README.md up to date
- Update RSPAMD_INTEGRATION.md for integration changes
- Update DICTIONARY.md for dictionary-related changes
- Include examples and common use cases
- Profile first: Measure before optimizing
- Algorithmic improvements: Focus on O(n) improvements
- Memory efficiency: Use object pooling, avoid unnecessary allocations
- Cache-friendly code: Consider data locality
- SIMD when applicable: Use vectorization for hot loops
# Run performance tests
./kagome_performance_tests
# Profile with perf (Linux)
perf record ./kagome_main "large_input_text.txt"
perf report
# Memory profiling
valgrind --tool=massif ./kagome_main "input.txt"- Maintain source compatibility within major versions
- Follow semantic versioning (SemVer)
- Document breaking changes clearly
- Provide migration guides for major version updates
- Primary: Linux (Ubuntu/Debian, RHEL/CentOS)
- Secondary: macOS (Homebrew environment)
- Future: Windows (with appropriate build system updates)
- Keep dependencies minimal and well-justified
- Prefer header-only libraries when possible
- Use FetchContent for C++ dependencies
- Document system package requirements
- Update version in
CMakeLists.txt - Update
CHANGELOG.md - Create git tag:
git tag -a v1.0.0 -m "Release v1.0.0" - Push tag:
git push upstream v1.0.0
# Build Debian packages
dpkg-buildpackage -us -uc
# Test installation
sudo dpkg -i ../kagome-*.deb
# Test Rspamd integration
sudo systemctl restart rspamd- Issues: Use GitHub issues for bugs and feature requests
- Discussions: Use GitHub discussions for questions
- Documentation: Check existing docs first
- Code review: Ask for review on complex changes
By contributing to Kagome C++, you agree that your contributions will be licensed under the Apache License, Version 2.0.
- Sponsor: TwoFive Inc. for supporting this project
- Original Project: kagome by ikawaha
- Contributors: All contributors to the project
- Dependencies: Open source libraries and tools used in development
This project is proudly sponsored by TwoFive Inc., Japan's leading email security company. Their expertise in email systems and Japanese text processing makes them the perfect sponsor for advancing Japanese tokenization in email security applications.
For sponsorship opportunities, see SPONSORS.md.