Cameron D. Campbell 康文林

Family, Social Mobility, and Inequality in China and in Comparative Perspective

Menu
  • Research
    • Abridged CV
    • Full CV (PDF)
    • 2 page CV (PDF)
    • Google Scholar
    • 百度学术
    • ORCID
    • HKUST Repository
  • News
  • Data
    • China Government Employee Database – Qing (CGED-Q) 中国历史官员量化数据库(清代)
      • Download Data
      • Search by Name
      • CGED-Q Jinshenlu Public Release – Resources for Users
    • China Multigenerational Panel Databases 中國多代人口数据庫
      • Download Data
  • Lee-Campbell Group
    • People
    • Projects
    • Publications
  • Photography
    • Photo site 摄影网站
    • Map view
    • Updates
  • Contact
Menu

Consolidating Chinese character variants in preparation for nominative linkage

Posted on July 27, 2016September 2, 2022 by camecamp

While carrying out nominative linkage of records of individuals in different editions of a database we are constructing, we ran into three types of problems:

  • Different variants of the same character were used when entering the same individual’s surname or given named in different editions, preventing a link from being made. This was sometimes the result of the coder’s keying, and in other cases, it seemed to reflect differences in the character in the original.
  • Keying mistakes by coders led to the occasional entry of a simplified character, when a traditional character should have been entered.
  • A character in an individual’s given name was replaced with a homonym, preventing a link from between made.

To deal with this, we wrote code in STATA to

  • Consolidate variants for the purpose of nominative linkage by replacing them to the most common variant that appears in our dataset. This isn’t necessarily the ‘right’ variant and we are only doing the replacement temporarily, when we do linkage. We preserve the original entry in the dataset. Since we are linking on surname, given name, and for anyone who wasn’t in the Banners, province and county of origin, we’re not too worried about the prospect of false positives created by linking someone to someone else who has the exact same province and county of origin, and identical other characters in their surname and given name, but whose name actually included the variant.
  • Translate simplified characters to traditional characters. Again, this is only for linkage. We’re not worried about false positives because for a link to be made correctly, the other person would need to be identical on all the other characters in their province, county and surname and given name, and differ only in that one character was written with a traditional character, rather than the traditional character that is now the simplified version of that traditional character.
  • Translate strings of Unicode characters into pinyin, with tones, and then allow for matches to be made in situations where the combination of province, county, surname and given name were identical except for one character that was nevertheless a homophone.

In the hope that our programs and the tables used for mapping might be useful to someone, we provide them below.

Please keep in mind that these were developed for a very specific purpose, nominative linkage, where false positives were unlikely, because we were matching on additional information, and we considered it very, very unlikely that there could be two people from the same province and county, whose combination of name and surname differed by only one character, and the different characters were both variants of a same character. For our purposes, even a few false positive would not be much of a problem. However, for other purposes, the programs below might be inadequate, and some details we have swept under the web might be crucial. If you are dealing with a situation where more precision is important, you are of course welcome to take what we have done as a starting point and come up with something better. We do hope that you will refer to this blog entry.

The other caveat is that we just don’t have time to provide any help or answer questions. For the time being, these programs are sufficient for our purposes, and we are not inclined to do much more work on them. I can’t incorporate suggestions, and if you have questions about the code or the tables, I may or may not be able to answer. The material we have made available here will be most useful to someone who knows STATA, or some programming language.

Our programs rely on mappings constructed from files we downloaded from the Unicode website: http://unicode.org/charts/unihan.html, in particular Unihan.zip from the folder http://www.unicode.org/Public/UCD/latest/ucd/. Within that zip file, Unihan_Variants.txt was the basis for our mapping of character variants and simplified/traditional variants. Unihan_Readings.txt was the basis for our mappings to pinyin. We imported these files into STATA, and then produced three different files, one for the variant mappings, one for simplified/traditional mapping, and one for pinyin. We did additional processing to narrow down the variant mappings so that they would all converge on the version of a character that was most common in our dataset.

We have a STATA .do file that defines programs to carry out these tasks. It needs three .dta files mentioned above that you will also have to download and place in a folder: Character variants recodes.dta, Simplified to traditional.dta, and Unicode pinyin.dta.

Before executing any of the programs, you will need to define a global macro named conversion that specifies the path to the folder in which the .dta files that they need have been placed:

global conversion “pathname“

Replace pathname with the path to the folder, for example, C:\Users\camcam\Documents\StataFiles

Consolidating variants

The program to consolidate variants, consolidate_variants, expects three arguments:

consolidate_variants varname1 newvarname1 newvarname2

The first, varname1, is a string variable already in memory that will be the subject of the processing. The second, newvarname1, is a string variable to be created that will be where the output is placed. The third newvarname2 will be a string variable to be created which will include a list of the transformations that were made.

The program dices the Unicode string into its constituent characters, converts each character to a Hex representation of its Unicode value, merges with Character variants recodes.dta to obtain a new value if available, and then reassembles everything. Looking at it, there are things I would do to speed it up and simplify it, but I don’t have time.

In the file Character variants recodes.dta, unicode is the ‘before’ version, unicode_cv is the ‘after’ version, and for reference, the Chinese characters corresponding to the before and after Unicode values are included as original_cv and outcome_cv, respectively. Outcome_cv_total is the count of the number of times the outcome character appeared in our data, and was the basis for the decision about which character was ‘before’ and which one was ‘after’.

For reference, here is the Excel file with the variant mappings.

Converting simplified to traditional characters

The program to transform simplified to traditional, to_traditional, is invoked the same way:

to_traditional varname1 newvarname1 newvarname2

newvarname1 will contain the traditional version of varname1, and newvarname2 will contain a list of the changes that were made.

Here is the Excel file with our simplified to traditional mappings.

Converting characters to pinyin

Finally, to_pinyin creates a string of pinyin with tones. It is invoked with two arguments, the first being the name of an existing string, and the second the name of a string to be created.

to_pinyin varname1 newvarname1

Here is the Excel file with our mappings to pinyin.

Stripping out tone marks in the pinyin

We also created a program to strip tone marks out of the pinyin strings created by the code above.

pinyin_no_tone varname1 newvarname1

This is hopefully self-explanatory.

  • Instagram
  • Photography website
  • Bluesky
  • LinkedIn

Recent Posts

  • New piece in Guangdong Social Science

    March 29, 2025
  • New article in Explorations in Economic History

    March 21, 2025
  • China Government Employee Dataset-Beiyang (CGED-BY) added to online search

    February 11, 2025
  • Paper in 历史档案 (Historical Archives) by Chen Jun on mid- and low-level Qing military officials

    October 20, 2024
  • Kinship information in the 同年齿录 and related sources completed in August 2024

    August 28, 2024
  • CGED-Q Meeting at Central China Normal University, July 29, 2024-August 2, 2024

    August 6, 2024

Recent Photography

  • HKUST Guangzhou 香港科技大學(廣州)

    March 29, 2025
  • Guozijian and Confucius Temple in Beijing 北京國子監及孔廟

    March 29, 2025
  • Yonghegong in Beijing 北京雍和宮

    March 29, 2025
  • Sunset at Razor Hill, near the HKUST campus 鷓鴣山日落竟

    February 15, 2025
  • Taiwan Province City God Temple 台灣省城隍廟

    February 15, 2025
  • Zhongzheng District in Taipei 臺北中正區

    February 15, 2025
  • A walk from Oban to Dunbeg and back, in the winter

    February 15, 2025

©2025 Cameron D. Campbell 康文林 | Theme by SuperbThemes