A 56.6 GB Zi Wei Dou Shu dataset, and the case for deterministic AI training data

The most interesting thing about Renhuai123/ziwei-doushu is not that it is a Zi Wei Dou Shu charting engine. It is that the project’s v4.0 dataset, released October 15, contains 674,976 natal charts and 8,774,688 interpretation segments totaling roughly 20.4 billion characters across 56.6 GB uncompressed, and every one of those segments was produced by a deterministic rule engine, not by an AI model. The README states it plainly: “排盘引擎 + 断语库确定性生成，不经 AI，同一输入恒得同一输出” (charting engine plus interpretation library, deterministically generated, not through AI, same input always yields same output).

That is a quiet rebuke to a lot of how the industry now thinks about training data.

What’s actually in the box

The numbers are worth sitting with. The v4.0 dataset covers every real calendar date from 1950 to 2026, times 12 two-hour 时辰 periods, times two genders, which yields 674,976 charts. Each chart gets 13 analysis dimensions (overview, personality, love, career, wealth, health, siblings, children, relocation, social, property, spirit, parents), so 8,774,688 segments. The compressed archive is only 161 MB across 77 per-year zstd files, and the project says all 77 were decompressed and byte-compared against source before release. The prior v3.0 sample set was 518,400 charts at 33.1 GB; v4.0 is a 71% size jump and adds 17 years of coverage.

The generation method matters more than the volume. The project’s interpretation library (断语库) is a hand-curated rule set, and the v4.0 release is specifically a correction pass: quotes attributed to Ni Haixia’s 天纪 lecture series or the 紫微斗数全书 that could not be traced verbatim were deleted or replaced, and each surviving Ni quote now carries an episode and timestamp citation down to the second. Impossible four-transformation (四化) combinations like 七杀化权 or 紫微化忌 were removed. Conditional interpretations that previously hedged “when strong… when fallen…” now show only the branch that applies to the given chart. The knowledge base’s classical-text citations went from 244 to 405, each one verifiable word-for-word in the source book.

That is data curation as editorial work. Someone read the primary sources, checked every claim, and cut what could not be sourced. The result is a corpus with an auditable provenance chain.

The part builders should steal

The project draws an explicit line between what is open and what is not, and the line is instructive. Open: the charting engine in lib/ziwei/ under MIT, the public-domain classical texts, the SEO knowledge graph, a demo UI, both datasets under CC BY 4.0. Closed: the interpretation library itself (db-analysis.ts is a placeholder in the open repo), the AI interpretation prompts, the backend API routes, the user system, and the commercial frontend. The README is candid that the demo UI is a fraction of the production one: the chart page is 75 lines versus 1,227 in the commercial build, the insight panel 467 versus 5,088.

So the open release is genuinely useful as a charting engine and as a training corpus, but it will not hand you the product. That is a defensible split. It also means anyone fine-tuning a small model on the 518,400 input-output pairs in the v3.0 sample set is training on rule-engine output, which is exactly the kind of clean, consistent, license-clear data that synthetic-data pipelines try to approximate and often fail to.

Where the engineering earns its keep

The engine is not a toy. It handles true solar time correction by birth-city longitude plus the equation of time, Chinese daylight saving time for 1986–1991, four configurable leap-month conventions, late-子-hour day rollover, seven brightness tiers for stars with switchable schools, and a 1,100-plus-line pattern library covering classical configurations like 紫府同宫 and 七杀朝斗. The October 20 update even changed the default star-brightness convention to match mainstream charting software, with a note that v4.0’s brightness characters and some brightness-dependent interpretation sentences were generated under the old default and will differ.

That last detail is the kind of honesty most dataset releases skip. The project also flags a known defect in the older v3.0 sample set: about 2,520 samples (0.49%) have impossible dates like February 30, because generation took days 1–30 every month. The engine now rejects nonexistent dates as of October 3, and the next dataset version will not contain them.

The AI angle, stated plainly

The README pitches the sample dataset for “微调小模型的训练语料” (fine-tuning corpus for small models) and “AI 对话的 RAG 检索源” (RAG retrieval source for AI chat). That is the right framing. A 518,400-pair corpus of consistent input-output examples, generated by rules rather than a model, is a rare thing: it has no hallucination surface, no license ambiguity beyond attribution, and no distribution drift between examples. For anyone building a retrieval or fine-tuning pipeline in a domain with formal rules, this is a template.

The commercial product, metisziwei.com, does layer AI interpretation on top. But the dataset deliberately excludes the AI layer, the age-adjusted sentences, and the appended Ni-law paragraphs. What ships open is the deterministic substrate.

The dataset is 20.4 billion characters of rule-generated text with an auditable provenance chain. That is a more useful thing to open-source than another model checkpoint.

What to watch

The open question is whether the correction pass holds up under external scrutiny. The project invites PRs that touch charting conventions only with supporting evidence from classical texts or verifiable comparisons, and it says issues with reproduction steps get priority. Whether the 405 classical citations and the reattributed Ni quotes survive a hostile read is the real test of the v4.0 release.

For AI builders, the takeaway is narrower than the astrology. A domain with explicit rules can produce a large, clean, commercially usable training corpus without a single model call. The ziwei-doushu repo is a working example of that, and the 161 MB compressed size of a 56.6 GB deterministic corpus is the number to remember the next time someone argues that good training data has to come from a generative model.