SILK is an index and search engine built for GEUL. Without decoding entire GEUL documents, it answers queries such as “which documents state that the occupation (P106) is physicist (Q169470)?” using record indexes alone.

StatusEarly development. The search algorithms are being validated, and formats may change before the first release
LanguageGo
InputGEUL pack archives (uncompressed)
LicenseCC0
Repositorygithub.com/geul-org/silk

Core Thesis

Search was never the hard part. Unstructured data was the problem.

Conventional: write first → infer structure later (tokenizing, embedding, analysis)
GEUL:         structure at write time → search is just integer comparison and range scans

A GEUL record already carries fields such as subject, predicate, object, and language as integer IDs at fixed positions. SILK simply extracts those fields and keeps them sorted.

Why It Fits GEUL

Property of GEULWhat SILK gains
A single 16-bit TAG WORD carries the kind (Kind, Codec) and the lengthRecords can be filtered and skipped without a codebook
Edge fields are records at fixed positionsKeys are extracted by position. No separate parsing rules are needed
Controlled vocabulary is integer IDs (Q, P, language)Equality is an integer comparison, with no string normalization
A pack consists of immutable objects and generationsThe index is also built from immutable segments and generations, so snapshots are safe
Conversion is deterministicThe same input yields the same index. Incremental results can be compared byte for byte against an index built from scratch

GEUL documents are never re-sorted. The index is only a sorted view; the published bytes, content hashes, and signatures stay unchanged.

Architecture

Input

SILK reads GEUL pack archives through gogeul. Because the index points to byte offsets inside data packs, the set must store records uncompressed (GEULCOMPRESS=none). zstd compression is for distribution and archiving; the search layer uses uncompressed packs.

Partition Spec

For each TAG kind, a partition key and a sort key are declared as fixed-position fields of the record. Nothing like predicates or languages is hard-coded; the table below shows the defaults for Wikidata documents.

TAGPartition keySort key
TRIPLEPredicateObject
QUALIFIERPropertyValue
LABEL·ALIAS·DESCRIPTIONLanguageSubject
SOURCEnoneOrigin
REFERENCE-NODEnoneObject
  • A key is the first few bytes (a slot) of a field record. If the record fits in the slot, equality is exact; if it is longer, the key only selects candidates.
  • Spec validation rejects partition keys on fields whose position requires a codebook, or on fields of unbounded length.
  • A spec is identified by its content hash. When the spec changes, only that TAG is rebuilt.

Runs, Increments, and Merging

ComponentRole
Sorted runAn immutable segment sorted by (sort key, document, offset) per partition. Searched by binary search
Appended segmentHolds only the newly arrived revisions from an incremental update (silk index update). Scanned
Live-revision bitmapMarks the old revisions of changed documents as dead instead of deleting them
Size-tiered mergingMerges runs of the same size tier with an O(n) merge, and folds into the base run when dead rows pile up
GenerationThe manifest generation is swapped atomically. Readers pin a generation, so snapshots hold even during a merge

Merging can be restricted to idle time (-idle, leaving 30% of CPU and memory free). It stops at partition boundaries and resumes from a checkpoint.

Queries

  1. Equality condition: find the partition and binary-search the sorted runs.
  2. Multiple conditions: merge-join on the document (subject). All conditions must hold within a single document.
  3. Verification: decode candidate records with gogeul, compare them field by field, and read the Edge Id.
silk query -triple P106=Q169470 ARCHIVE
silk query -count -triple P31=Q5 -triple P106=Q169470 ARCHIVE
silk query -label ko -match LABEL:1=Q42 ARCHIVE

Measurements

Measured on Wikidata Q1–Q10000 plus 16,696 property documents (271MB, 7.07 million index records). 8 threads, best of 7 runs, excluding process startup time.

Queries

QueryResult documentsPrevious approach (full scan)SILK index
P106=Q169470 (occupation = physicist)55~20ms0.12ms
P31=Q5 (human)1,579~19ms~1ms
human + physicist54~38ms~1ms
LABEL ko9,746(not possible)~4ms
  • The previous approach scanned the entire TRIPLE record map.
  • For LABEL ko, most of the time goes into reading 9,746 records from the pack for verification.

Size, Build, and Incremental Updates

ItemValue
Index size113MB (15.7B per record on average)
Build~3 seconds (document parsing is single-threaded)
Incremental update (4,217 revisions changed)1.8 seconds
Incremental resultAfter 2 updates, 2 merges, and an interrupted-then-resumed run, query results are byte-identical to an index built from scratch
Write amplification1.31 cumulative (1.71 if rebuilt every time)
  • Extrapolated to all of Wikidata (about 7.5 billion records), the index would be about 110GB (estimate).
  • The cost is size. It is about 4x larger than a simple map at 4B per record. Narrower slots and entity signatures are next to be examined.

Signature Scan (preliminary measurement for entity signatures)

Time to scan 100 million 128-bit signatures with a mask and pattern (i7-14700K, AVX2).

ImplementationTime
NumPy, 1 thread~1s
Go, 1 thread~134ms
Go, 8 threads~27ms
Go, 16–20 threads~23ms (memory bandwidth bound)

What makes the difference is not the language but fused loops without intermediate arrays, plus multithreading.

It is a division of labor, not a replacement.

SILKVector search
Queries handledStructured queries (predicate, object, language, entity conditions)Semantic similarity that is hard to express as structure
ResultThe set satisfying the conditions (exact)A similarity ranking (approximate)
PrerequisiteThe data must be structured in GEULRaw text or embeddings are enough

For queries that look for facts expressed in GEUL, SILK is exact and cheap. For unstructured raw text or queries for “something similar,” vector search is the better fit.

Position and Next Steps

SILK is the search technology used by WMS (the GEUL storage and retrieval coordinator). Judging whether a retrieved statement is correct is the job of a separate adjudication engine.

Next stepDetails
Entity signaturesA fixed-width 16–128-bit bit string per entity. Users declare the fields (enum, one-hot, Bloom, prefix, range). It is an approximate first-pass filter followed by verification. Currently at the planning stage (partly in the library, not yet in the index)
Spec as a GEUL documentDeclare the JSON spec as a GEUL document and read layouts from codec specification documents (DECLARE, FIELD)
Statement assembly indexA Target index that directly finds the TRIPLE a QUALIFIER or SOURCE points to
Preparing for scaleMemory caps for build and merge, parallel parsing during build

The initial concept (2026-03) was a Python/NumPy design that scanned 64-bit entity identifiers with bitwise AND. It has since become a Go engine that indexes GEUL records directly.

Repository and License

  • Repository: github.com/geul-org/silk
  • Install: go install github.com/geul-org/silk/cmd/silk@latest
  • License: our own work is CC0 1.0. Third-party components follow their own licenses (see NOTICE in the repository).