OpenCC Source Layout

This directory contains the C++ core library, the public C API, command-line tools, and native extension entry points. The code is organized around a simple conversion pipeline:

1. Load a JSON configuration.
2. Load segmenters and dictionaries referenced by that configuration.
3. Segment input text.
4. Apply one or more dictionary-based conversion stages.
5. Return converted text or inspection data.

The data files that drive this pipeline live outside this directory:

• data/config/*.json: conversion schemes.
• data/dictionary/*.txt: source dictionaries.
• generated .ocd2 files: marisa-trie dictionary binaries.

Main Pipeline

Configuration

• Config.hpp, Config.cpp
  • Finds and parses JSON configuration files.
  • Builds segmenters, dictionaries, dictionary groups, conversions, and converters.
  • Resolves dictionary/resource files through ResourceProvider.
  • Keeps config loading separate from dictionary resource lookup.
  • Uses UTF-16-capable file checks on Windows for internal config and resource paths.
• ResourceProvider.hpp, ResourceProvider.cpp
  • Provides ResourceProvider and FilesystemResourceProvider.
  • FilesystemResourceProvider searches configured resource directories in order and returns a resolved path for lower-level dictionary loaders.
  • Embedded hosts can combine application, plugin, and system OpenCC data directories without changing config JSON.

Segmentation

• Segmentation.hpp, Segmentation.cpp
  • Base interface for segmenters.
• MaxMatchSegmentation.hpp, MaxMatchSegmentation.cpp
  • Maximum forward matching segmenter used by normal OpenCC configs.
  • Uses dictionary prefix matches to keep phrases intact.
• PluginSegmentation.hpp, PluginSegmentation.cpp
  • Runtime-loaded segmentation plugin support.
  • Used by the Jieba plugin.
• Segments.hpp
  • Segment container used between segmenters, conversions, and inspection output.

Conversion

• Converter.hpp, Converter.cpp
  • Owns a segmenter and a conversion chain.
  • Provides high-level string and buffer conversion.
• ConversionChain.hpp, ConversionChain.cpp
  • Ordered list of conversion stages.
• Conversion.hpp, Conversion.cpp
  • One dictionary-backed conversion stage.
• ConversionInspection.hpp
  • Data structures returned by inspection mode.

The core conversion path depends on segmentation and longest-prefix dictionary matching. Character-by-character replacement is not equivalent to OpenCC behavior because phrase priority and multi-stage conversion order matter.

Dictionaries

Interfaces and Shared Types

• Dict.hpp, Dict.cpp
  • Abstract dictionary interface.
  • Supports exact match, prefix match, all-prefix match, and enumeration.
• DictEntry.hpp, DictEntry.cpp
  • Key/value entry representation.
• PrefixMatch.hpp, PrefixMatch.cpp
  • Prefix match result.
• Lexicon.hpp, Lexicon.cpp
  • In-memory collection of dictionary entries.
• SerializableDict.hpp
  • Template helpers for loading serialized dictionaries from files.
• SerializedValues.hpp, SerializedValues.cpp
  • Compact storage for candidate value lists used by .ocd2.

Implementations

• TextDict.hpp, TextDict.cpp
  • Tab-delimited text dictionary.
  • Useful for source data and tests.
• MarisaDict.hpp, MarisaDict.cpp
  • Default .ocd2 dictionary format.
  • Uses marisa-trie for compact prefix lookup.
• DartsDict.hpp, DartsDict.cpp
  • Legacy .ocd dictionary format.
  • Requires Darts support.
• BinaryDict.hpp, BinaryDict.cpp
  • Legacy binary payload support used with Darts serialization.
• DictGroup.hpp, DictGroup.cpp
  • Ordered group of dictionaries.
  • Tries dictionaries in sequence and returns the first usable match.
• DictConverter.hpp, DictConverter.cpp
  • Converts dictionary files between supported formats.

Binary Format Portability

Neither binary dictionary format is portable across platforms, but they fail differently, and the constraints differ per format.

	Byte order	Word size (32/64-bit)
.ocd	dependent	independent (since 1.4.1)
.ocd2	dependent	dependent

• **.ocd is word-size independent.** The Darts double array uses id_type = unsigned int (4 bytes on every supported platform), and DartsDict::SerializeToFile writes the array length as an explicit uint32_t. BinaryDict writes all of its integer fields as fixed-width uint32_t as well. Files written by pre-1.4.1 64-bit builds used native 8-byte size_t fields; BinaryDict::DetectFieldWidth() still recognizes that legacy layout, and LooksLikeLegacy64Field() performs the probe through a native uint64_t read so the detection is correct on big-endian hosts too.
• **.ocd2 is not.** The SerializedValues payload OpenCC writes uses fixed-width uint32_t fields, but the marisa-trie image that precedes it does not: MARISA_WORD_SIZE is derived from UINTPTR_MAX, and the serialized BitVector::Unit / FlatVector::Unit are uint64_t on 64-bit builds and uint32_t on 32-bit builds. A 64-bit-built .ocd2 loaded by a 32-bit build is not reliably rejected either — marisa's Vector<T>::map only checks total_size % sizeof(T), and a multiple of 8 is always a multiple of 4 — so the mismatch can be read as silently wrong data rather than an error.

Forcing MARISA_WORD_SIZE to 32 would not make .ocd2 portable, since byte order would still differ; it would not apply to USE_SYSTEM_MARISA builds (which distributions use), where the format would then depend on which marisa the build linked against; and it would narrow the rank/select word width in MarisaDict's lookup hot path. Making .ocd2 genuinely portable means normalizing the trie image on write and converting back on load, which gives up the in-place mmap that makes the format fast.

The practical consequence is that generated .ocd2 files must be built for the architecture that consumes them, and must never be shipped from an architecture-independent package — see packaging/debian/README.md for how this played out in Debian.

Public APIs

C++ API

• SimpleConverter.hpp, SimpleConverter.cpp
  • High-level C++ wrapper around Config and Converter.
  • Accepts a config name/path, optional search paths, or an explicit ResourceProvider.
  • Throws C++ exceptions on failures.

C API

• opencc.h
  • Stable C ABI for downstream users and bindings.
  • Exposes opencc_open, opencc_open_w, opencc_convert_utf8, opencc_convert_utf8_free, opencc_close, and error helpers.

Windows path semantics need care:

• opencc_open_w(const wchar_t*) is the explicit UTF-16 Windows API.
• opencc_open(const char*) keeps the historical Windows/MSVC narrow-string behavior. Do not silently change its encoding contract without a migration plan.
• New Windows path-taking C APIs should use explicit names such as *_utf8 or *_w rather than relying on ambiguous char* semantics.

Python Extension

• py_opencc.cpp
  • Python module binding built on top of the C++ core.

Command-Line Tools

The native CLI tools live under src/tools.

• CommandLine.cpp
  • Small executable entry point.
  • On Windows/MSVC, obtains command-line arguments through wide Windows APIs and converts them to UTF-8 before dispatching to the CLI core.
• CommandLineMain.hpp, CommandLineMain.cpp
  • Main opencc command implementation.
  • Handles conversion, segmentation/inspection modes, measurement output, stream conversion, and explicit --in-place conversion.
• PlatformIO.hpp, PlatformIO.cpp
  • CLI platform boundary for path-sensitive operations.
  • Handles UTF-8 file open/remove/replace, temporary output files, same-file detection, real path resolution, and Windows argv collection.
• DictConverter.cpp
  • opencc_dict implementation.
• PhraseExtract.cpp
  • opencc_phrase_extract implementation.
• CmdLineOutput.hpp
  • Shared command-line help formatting.

CLI file conversion must remain streaming. Do not replace stream processing with "read whole file into memory" logic. In-place conversion is intentionally opt-in:

• Without --in-place, -i and -o referring to the same actual file is rejected.
• With --in-place, output is written to a temporary file next to the target, then the target is replaced after conversion succeeds.

Platform and Path Handling

OpenCC uses UTF-8 internally for paths unless an API explicitly documents a different contract. Windows code should convert to UTF-16 at the platform boundary.

Important files:

• UTF8Util.hpp, UTF8Util.cpp
  • UTF-8 helpers and platform string conversion helpers.
• WinUtil.hpp
  • Small Windows-only UTF-8/UTF-16 conversion helpers.
• tools/PlatformIO.*
  • CLI path boundary.

Maintenance rules:

• Do not add raw fopen, std::fstream, stat, std::tmpnam, or narrow Win32 path calls to new Windows-sensitive paths.
• Prefer existing platform helpers or add a clearly named boundary helper.
• Keep public API encoding contracts explicit.
• Validate Windows path changes with real Windows CI. Zig cross-compilation is useful for compile/link coverage, but it does not execute Windows runtime path behavior.

Plugins

• plugin/OpenCCPlugin.h
  • C plugin ABI used by loadable segmentation plugins.
• PluginSegmentation.*
  • Host-side plugin loader and adapter.

The Jieba plugin lives outside src under plugins/jieba.

Tests

Most core modules have adjacent *Test.cpp files in this directory. Important test groups include:

• ConfigTest.cpp
  • Config search paths and Unicode config path handling.
• SimpleConverterTest.cpp
  • C++ wrapper behavior and C API basics.
• Conversion*Test.cpp
  • Conversion and inspection behavior.
• MaxMatchSegmentationTest.cpp
  • Segmenter behavior.
• MarisaDictTest.cpp, TextDictTest.cpp, DictGroupTest.cpp
  • Dictionary implementations.
• UTF8*Test.cpp
  • UTF-8 slicing and utility behavior.

CLI tests live in test/CommandLineConvertTest.cpp because they execute the built command-line binary. They cover streaming behavior, Unicode paths, measurement output, inspection modes, and in-place conversion safety.

Build Integration

The source tree is built through both CMake and Bazel:

• src/CMakeLists.txt
• src/BUILD.bazel
• src/tools/CMakeLists.txt
• src/tools/BUILD.bazel

When adding or renaming source files, update both build systems and any direct cross-build scripts that carry explicit source lists.