Skip to content
BranchesTags

Repository files navigation

fusionData Developer Manual

Kevin Ummel (kevin.ummel@yale.edu)

Purpose of this Manual

This document is the primary developer manual for fusionACS team members with access to the Yale Milgram High-Performance Computing (HPC) environment. It covers upstream data processing managed by the fusionData package:

  • ingesting spatial data and survey microdata
  • creating predictor variable harmonization
  • generating fusion input datasets
  • executing the fusion process
  • compiling the fusionACS database for analysis and dissemination

Researchers looking to analyze fused U.S. social survey microdata should refer to the end-user fusionACS R package. It is the starting point for any analysis of fusionACS data and does not require access to the Yale HPC.

fusionACS Overview

The purpose of the fusionACS project is to statistically fuse variables from “donor” surveys (e.g., RECS, AHS) onto the American Community Survey (ACS). Fusing donor variables onto ACS respondents creates large-sample, probabilistic simulations of how ACS respondents would have answered donor questionnaires.

The data science platform relies on three core mechanisms:

  1. Shared Predictors: Common socioeconomic, demographic, and housing attributes across donor surveys and the ACS.
  2. Gradient Boosting Models: Probabilistic simulation via variable-$k$, conditional expectation matching using high-performance, gradient boosting machine learning models. Methodological details are described in Ummel et al. (2024), Scientific Data.
  3. Spatial Microsimulation: Integration with ORNL’s UrbanPop framework (Tuccillo et al. 2023, Applied Geography) to estimate ACS respondent location down to Census block groups.

The project’s core functionality is made available via three R packages:

Package Role & Scope Target Audience
fusionACS Analytical Tools: Assembles and analyzes U.S. survey fusion outputs; provides access to a public pseudo-sample of the full fusionACS database. End-user researchers and data analysts.
fusionModel Statistical Engine: Implements the generalized statistical fusion techniques described in Ummel et al. (2024). Data scientists building custom fusion workflows.
fusionData Data Development: Manages U.S. survey ingestion, spatial predictor datasets, predictor variable harmonization, execution of fusion, and compilation of fusion output for subsequent analysis. fusionACS team members operating within the Yale HPC.

The fusionData package is the focus of this document, since it contains the suite of tools needed to execute the fusionACS data development pipeline. As of August 2026, all fusionACS data assets have been migrated to the Yale Milgram HPC environment. Future development of new fusion outputs will occur with the Yale HPC. Any user of the fusionData package will need to have access to the fusionACS project directory in the HPC.

Package Installation

For users whose sole objective is analysis of fusionACS output, it is only necessary to install the fusionACS package. It can be installed on a user’s own/local computer, making use of the public pseudo-sample (HPC access not necessary); this is recommended for analysis development and prototyping.

For users with HPC access who want to do production-level analysis of the complete fusionACS database, the fusionACS package needs to be installed within their HPC user environment. In either case, the installation command is the same:

devtools::install_github("ummel/fusionACS")

For users who want to do “upstream” data development (e.g. add donor surveys, additional fusion runs), it is necessary to install both the fusionModel and fusionData packages within the HPC:

devtools::install_github("ummel/fusionModel")
devtools::install_github("ummel/fusionData")

Loading the fusionData package within the Yale HPC via library(fusionData) automatically changes the working directory to: /gpfs/milgram/project/rao/shared/fusionACS/fusionData

Yale HPC Directory Structure

All fusionACS data assets reside within the main fusionACS project directory: /gpfs/milgram/project/rao/shared/fusionACS

It is helpful to have an understanding of the sub-directory structure.

Sub-Directory Role & Content Description
/fusionData Core data development directory managed by the fusionData package.
/versions Complete, immutable, date-stamped database snapshots (versions) compiled via compileVersion().
/public_releases Public pseudo-samples uploaded as Github data releases for use by the fusionACS package.
/analysis Unstructured directory for (optionally) storing arbitrary analyses and scripts developed by fusionACS team members.

The /fusionData directory contains the following sub-directories:

Directory Path Role & Content Description
/geo-raw Raw, unprocessed spatial/geographic source datasets.
/geo-processed Contains processed spatial dataset files and associated code (*_processed.rds, *_processed.R), geographic concordance files (concordance/geo_concordance_2010.fst, concordance/geo_concordance_2020.fst) and compiled spatial predictors file (geo_predictors.fst).
/survey-raw Raw, unprocessed survey microdata organized by survey and vintage.
/survey-processed Standardized survey microdata (*_processed.fst), associated dictionary files (*_dictionary.rds), and the code used to generate them (*_processed.R).
/survey-sensitive Restricted-access or sensitive administrative microdata
/harmony Code for running the harmony() Shiny app/tool and storage of resulting survey-specific harmonization files (/harmonies).
/universe Code for running the universe() Shiny app/tool to explore available survey microdata in /survey-processed.
/fusion Production output generated by fusionInput() and fusionOutput().
/fusion_ Test output generated by fusionInput() and fusionOutput().
/urbanpop ORNL UrbanPop synthetic population data.
/data Package-wide data objects.

Data Development Pipeline

Generation of fusionACS outputs follows a sequence of steps:

  1. Ingest Spatial Data
  2. Ingest Survey Microdata
  3. Harmonize Predictor Variables
  4. Generate Fusion Inputs
  5. Generate Fusion Outputs
  6. Compile Version and Github Public Release
  7. Perform Analysis

Step 1 is maintained by Kevin, and a typical user will not have reason to engage with it.

Step 2 consists of adding new donor survey microdata (either entirely new or new vintages). This is an important step, but it doesn’t have to be done in the HPC. I typically write and debug a survey ingestion script outside of the HPC and then copy the results into /survey-raw and /survey-processed in the HPC once I am happy with the results.

Steps 3 through 6 need to be done in the HPC, relying on the fusionData and fusionModel packages.

Step 7 can be done inside or outside of the HPC using the fusionACS package, but production-level results yielding accurate point estimates and margin of error is only possible inside the HPC.

Step 1: Ingest Spatial Data

This step is maintained by Kevin, and there is generally no reason for a typical user to modify the associated files.

There are two types of spatial (or geographic) data used within the fusionData package.

First, geographic concordance information provides linkages between different geographic units and their relationships across time. This information generally comes from MCDC Geocorr and is stored across the following files:

  • geo-processed/concordance/geo_concordance_2010.fst
  • geo-processed/concordance/geo_concordance_2020.fst
  • data/bg_crosswalk.rda
  • data/puma_crosswalk.rda

Note on geographic vintages: The definitions and boundaries of some Census geographic units change every decade. Consequently, there are two concordance files: 2010 and 2020, reflecting these changes. Within fusionACS - including for use in fusionACS::assemble() - different vintages are referenced by a “10” or “20” suffix. For example, “tract10” refers to Census tracts using the 2010 definitions, and “tract20” uses the 2020 definitions. Some geographic units are static over time and have no suffix: region, division, state, state_name, state_postal.

Second, spatially-referenced variables from various sources expand the set of potential predictor variables available during model training by adding geographic context to household- or person-level characteristics. In the fusionACS framework, the spatial unit of analysis is the PUMA (Public Use Microdata Area), which is observed for ACS households and imputed for donor households. The various spatial predictor datasets used by the package are compiled into the geo-processed/geo_predictors.fst, which is used during the fusion process to assign spatial predictors to donor and ACS microdata.

Processing spatial datasets consists of the following steps:

  1. Raw Data: Store raw geographic files in /geo-raw/, if necessary. Some datasets can be sourced directly from the web within the processing script.
  2. Ingest and Process: Write an .R processing script and save the processed output as _processed.rds in /geo-processed/ alongside its parent script.
  3. Compile Predictors: Run compileSpatial() to aggregate all spatial predictors to PUMA level and update the spatial predictors database (geo-processed/geo_predictors.fst).

To ensure smooth aggregation to PUMAs, every processed spatial _processed.rds file must satisfy two core requirements:

  1. vintage Column: Must contain a vintage column defining the time period (e.g., 2018, "2015-2016", or "always"). The special value "always" denotes time-invariant metrics, such as long-term climate normals.
  2. Valid Location Columns: Must contain one or more location columns matching variables present in one of the geo_concordance_20XX.fst files (e.g. state, county10, tract10, bg10). These keys allow compileSpatial() to map local geographies to PUMAs.

Some datasets require multiple columns to establish exact spatial concordance. For example, block-group level datasets (such as EPA-SLD) include state, county10, tract10, and bg10 columns to align correctly with the concordance file.

File naming for spatial datasets is flexible. The compileSpatial() function automatically detects and ingests any file in /geo-processed/ ending in _processed.rds. As long as the file ends with that suffix and meets the two structural requirements above, its metrics will be aggregated to the PUMA level and included in the master geo_predictors.fst file.

Important: Whenever spatial data assets are added or modified within the HPC, it is necessary to run fusionData::compileSpatial() to re-compile geo-processed/geo_predictors.fst.

Step 2: Ingest Survey Microdata

Survey ingestion transforms raw survey microdata files stored in /survey-raw into standardized microdata (.fst) and metadata dictionaries (.rds) stored in /survey-processed.

Given the variety of donor survey data structures and conventions, there is no strict procedure for how the processing _processed.R file(s) should be written, provided the script ultimately outputs {survey}_{year}_{H|P}_processed.fst and {survey}_{year}_{H|P}_dictionary.rds. Surveys that include both household and person-level respondent information have two such files - both “H” and “P” variants.

See, for example, survey-processed/RECS/2020/RECS_2020_H_processed.R. It is used to generate survey-processed/RECS/2020/RECS_2020_H_processed.fst and survey-processed/RECS/2020/RECS_2020_H_dictionary.rds.

Best Practices

  • Documentation & Comments: The .R processing scripts should include liberal use of comments to help others understand the codebase later. Comments should explain why a particular transformation or piece of code is included, not just what it does.
  • Variable Descriptions: In all cases, the _processed.R script that creates the _processed.fst microdata file(s) must use labelled::set_variable_labels() to assign variable descriptions, ideally taken directly from the official survey codebook, for every column.
  • Dictionary Generation: Standardized metadata dictionaries are generated and saved by calling createDictionary() at the end of the _processed.R script. This function reads the assigned variable descriptions and data types from the processed microdata to build the _dictionary.rds file.

Ingestion Requirements

Processed survey microdata should meet these conditions:

  • Retain as many valid survey observations and informative columns as possible.
  • Use official codebook variable names where possible, coerced to lowercase alphanumeric characters with single underscores.
  • Replace raw numeric codes with descriptive labels from official codebooks, using factor variables (and ordered factors whenever ordinal scales exist).
  • Replace “valid blanks” or skips in the raw data with plausible values; NA’s are often actual zeros or some other knowable value based on the question structure.
  • Impute unintended missing values (NA) using fusionModel::imputeMissing().
  • Use hid for household IDs, pid for person-level IDs, weight for primary weights, and rep_1, rep_2, … for replicate weights.
  • Ensure location identifiers match keys defined in geo-processed/concordance/geo_concordance.fst.
  • Assign variable descriptions using labelled::set_variable_labels() before calling createDictionary().

Errors or omissions in the creation of the processed microdata almost invariably lead to problems downstream in the pipeline. For that reason, it is important to take the time to ensure that the processed microdata and dictionary files are as complete, accurate, and clean as possible.

Perhaps the best way to understand the requirements is to review ingestion scripts and outputs in /survey-processed.

Important: Whenever survey microdata assets are added or modified within the HPC, it is necessary to run fusionData::compileDictionary() to re-compile the dictionary.rda and surveys.rda metadata utilized by other functions in the fusionData package.

ACS-PUMS Microdata

This step is maintained by Kevin, and there is generally no reason for a typical user to modify the associated files.

Unlike donor surveys, processing the ACS Public Use Microdata Sample (PUMS) is automated via processACSmicrodata() and associated functions. The ACS-PUMS is critical to proper functioning of the entire fusionACS project, so a set of standardized ingestion functions are part of the fusionData package.

  • Raw Survey Microdata: Download nationwide 1-year files (csv_hus.zip for households and csv_pus.zip for persons) directly from the Census FTP site.
  • Data Dictionaries: Download from the Census PUMS Documentation page. Use the CSV data dictionaries (e.g., PUMS_Data_Dictionary_2019.csv), which are often represented on the Census website with an Excel icon.

Complete PUMS microdata going back to 2005 is already processed and stored in the Yale HPC, so the only update going forward is the addition of the latest annual release in the fall of each year.

See the processACSmicrodata() documentation for details.

See Annex 1: ACS-PUMS Ingestion Details for more information.

Step 3: Harmonize Predictor Variables

Once a donor survey has been successfully ingested and documented, it is possible to start thinking about how to fuse that survey to the ACS.

The statistical linchpin of the fusion process is the set of “harmonized” variables common to a donor survey and the ACS. Identifying conceptually similar variables across surveys and determining how they can be modified to measure similar concepts is one of the most important steps in the process. It is also potentially time-consuming and error-prone.

The “Survey Harmonization Tool” was created to make this process easier and safer. It is a Shiny app that makes it easier to detect, specify, and save “harmonies” constructed between variables in donor surveys and variables in the ACS. The app launches in a browser window with the following call:

# Launch interactive harmonization tool
harmony()

The harmony() app is used to modify on-disk harmonization file stored at: harmony/harmonies/[DONOR]_[VINTAGE]__ACS_[ACS_YEAR].R.

At present, it only allows specification of harmonies between a donor survey and a specific ACS vintage (e.g. harmonizing 2015 RECS to 2015 ACS).

Construction of a harmony generally follows these steps:

  1. Select a donor survey and vintage.
  2. Select the recipient ACS vintage.
  3. Select a “Donor variable” from the drop down list. The list is searchable to help locate variables associated with certain words.
  4. Select a “ACS variable” to use for the “other side” of the harmony.
  5. For factor variables, edit the “Group” columns in the spreadsheet objects to create the maximum-resolution harmony between the two variables. You can see the “live” outcome of the specified harmonization strategy in the table at the bottom of the page. For continuous variables, no additional modification is needed as long as the two variables measure similar concepts.
  6. Once the harmony is specified as you like, click “Submit harmony”. The button only becomes available to click if minimal safety checks are passed for a valid harmony.

When a user clicks “Submit harmony”, the currently-specified harmony (as defined by the selected variables and settings) is saved to disk. Specifically, the details of that particular harmony are added to the appropriate .R “harmony file” located at /harmony/harmonies. For example, the file describing how to harmonize RECS 2015 and ACS 2015 variables is: /harmony/harmonies/RECS_2015__ACS_2015.R.

You will receive a pop-up message indicating if the harmony was successfully added to the local .R harmony file (it will be created, if necessary). You can confirm the harmony was added by selecting the “View harmonies” panel.

Creating as many valid harmonies as possible increases the chances of high-quality fusion. That said, it is most important to harmonize the critical respondent characteristics like income, education, building type, age, etc. It is extremely helpful to be familiar with the ACS, since this makes it easy to look at the donor variables and identify potential overlap in concepts/variables.

It may be helpful to review existing harmonies that I’ve already constructed for the RECS and AHS. This will show you the settings used and give you a sense of how and why they were used.

To review the full suite of ACS and donor variables available in the HPC, you can use the universe() Shiny app:

# Launch interactive survey dictionary explorer
universe()

When harmonizing a new vintage of an existing donor survey or adding an additional ACS vintage, the process can be sped up by using conveyHarmony() to propagate existing harmonization rules to the new target. For example:

# Copy existing harmonization logic to a new vintage
conveyHarmony(
  from = "RECS_2015__ACS_2019.R",
  to   = "RECS_2015__ACS_2020.R"
)

conveyHarmony() is purposefully conservative; it only conveys harmonies that are strictly identical in both ‘from’ and ‘to’. But it can still save considerable time.

See Annex 2: Advanced Harmonization for more information.

Step 4: Generate Fusion Inputs

The fusionInput() function generates the “input” data files necessary to perform fusion in Step 5.

fusionInput() applies harmonization scripts to donor and target microdata via harmonize(), imputes the location (PUMA) of donor survey respondents via imputeLocation(), scales numeric predictor variables for consistency via scaleNumeric(), and automatically checks if the distribution of harmonized predictors are similar in both the donor and ACS recipients (i.e. quality-controls the predictor variables).

It is generally recommended that fusionInput() be run first with test_mode = TRUE to ensure all safety checks are passed before doing a production run.

fusionInput(
  donor      = "RECS_2015",
  acs_year   = 2015,
  respondent = "household",
  test_mode  = TRUE
)

If there is a problem with upstream data inputs, fusionInput() will fail – hopefully with a helpful error message. It sometimes takes a few tries to troubleshoot upstream issues, especially for a new or complicated donor survey.

Output files are stored in structured paths based on execution mode:

  • Test Mode: fusion_/[DONOR_NAME]/[DONOR_VINTAGE]/[ACS_YEAR]/input/[DATE]/
  • Production Mode: fusion/[DONOR_NAME]/[DONOR_VINTAGE]/[ACS_YEAR]/input/[DATE]/

Each run creates three files in the target directory:

  1. [DONOR]_[ACS_YEAR]_[TYPE]_donor.fst: Scaled, harmonized donor training microdata.
  2. [DONOR]_[ACS_YEAR]_[TYPE]_recipient.fst: Scaled, harmonized ACS prediction microdata.
  3. [DONOR]_[ACS_YEAR]_[TYPE]_inputlog.txt: Execution log containing system details, arguments, and predictor similarity scores.

See the fusionInput() documentation for details.

Step 5: Generate Fusion Outputs

The fusionOutput() function generates fusion result (“output”) data files, making use of the structured input files generated in Step 4.

Under the hood, fusionOutput() wraps core functions from the fusionModel package: prepXY() to pre-screen predictors and order the fusion variables; train() to fit LightGBM conditional expectation models; and fuse() to generate probabilistic implicates for the recipient ACS microdata. fusionOutput() is pre-configured to use sensible defaults suitable for production execution in the HPC.

Typically, a given donor survey vintage is fused to multiple ACS recipient vintages, in order to make the donor variables available for UrbanPop spatial downscaling (which requires fusion outputs for 5-year periods). So far, I have put the fusionOutput code in a .R file within the appropriate /fusion directory. For example: /fusion/AHS/2023/AHS 2023 fusion to 2019-2023 ACS.R

This file provides a good template for general fusion:

# Ensure latest version of fusionModel package is installed
devtools::install_github("ummel/fusionModel")
library(fusionModel)

# Ensure latest version of fusionData package is installed
devtools::install_github("ummel/fusionData")
library(fusionData)

#-----

# Fusion variables
fusion.vars <- c("acprimary", "adequacy", ...)  # Specify desired variables here

#----

fusionOutput_multiyear(
  acs_years   = 2019:2023,
  donor       = "AHS_2023",
  respondent  = "H",
  fusion_vars = fusion.vars,
  M           = 20,
  validation  = TRUE,
  test_mode   = FALSE,
  margin      = 4,
  nfolds      = 10,
  nquantiles  = 4
)

fusionOutput_multiyear() is a convenience wrapper around fusionOutput() to iteratively run data fusion models across sequential ACS recipient years. The function automatically passes the .fsn model file generated by the prior year’s run to accelerate variable selection and model training for subsequent years.

Important: For production runs, we use M = 20 implicates.

Depending on the size of the donor survey and number of variables being fused, fusionOutput() can require significant amounts of RAM. In practice, I have often found it necessary to use the HPC “Scavenge” partition to obtain a compute environment with a safe amount of RAM for typical runs. In most cases, I would try launching an RStudio Server instance with something like 12 CPU cores per node and 12-16 GiB per CPU core. You should assume that fusing to a single ACS year will take about 45-90 minutes.

Output files from fusionOutput() runs are stored in structured paths based on execution mode:

  • Test Mode: fusion_/[DONOR_NAME]/[DONOR_VINTAGE]/[ACS_YEAR]/output/[DATE]/
  • Production Mode: fusion/[DONOR_NAME]/[DONOR_VINTAGE]/[ACS_YEAR]/output/[DATE]/

Each run creates the following files in the target directory:

  1. [DONOR]_[ACS_YEAR]_[TYPE]_prep.rds: Saved feature selection object from prepXY mapping target variables to screened predictors.
  2. [DONOR]_[ACS_YEAR]_[TYPE]_model.fsn: Trained LightGBM model archive generated by train containing tree structures and feature metadata.
  3. [DONOR]_[ACS_YEAR]_[TYPE]_valid.fsd: (Optional, created if validation = TRUE) Fused implicates simulated back onto original donor observations for possible internal validation by the user.
  4. [DONOR]_[ACS_YEAR]_[TYPE]_fused.fsd: Fused microdata containing M simulated target implicates mapped onto the recipient ACS microdata.
  5. [DONOR]_[ACS_YEAR]_[TYPE]_outputlog.txt: Execution log containing system details, arguments, CPU/memory stats, and timing summaries.
  6. [DONOR]_[ACS_YEAR]_[TYPE]_outputlog0.txt: (Optional, created when reusing an existing .fsn) Retained console output from the original model training run.

See the fusionOutput() documentation for details.

Step 6: Compile Database and Github Public Release

As new fusion outputs are generated over time - either by adding new donor surveys and vintages, expanding the set of donor variables fused, or improving the underlying spatial predictors - the state of the fusionACS project and associated data in the HPC changes. The fusionData package includes a function, compileVersion(), that is designed to save a “snapshot” (version) of the entire state of the fusionACS database at the time it is run.

compileVersion(version_date = as.character(Sys.Date()), public_release = TRUE)

This saves unique, time-stamped “versions” of the database in fusionACS/versions (i.e. version control). For example, a researcher might do a suite of analysis using the fusionACS database as it existed on July 23, 2025 and then publish research findings using that data. That version (at fusionACS/versions/2025-07-23) is stored statically in the HPC and can be accessed at any future point to replicate results.

It is necessary to periodically run compileVersion() in response to non-trivial changes to the underlying fusion outputs. To speed up database compilation and reduce on-disk storage, compileVersion() automatically detects which data assets have changed since the most-recent version stored in the HPC. It then processes and saves only the modified or new assets and uses symbolic links to reference any unchanged assets. Because of this, it is critical that existing versions on-disk not be deleted (the symbolic links will break).

By default, compileVersion() also generates a “public release” of the current database version that is then uploaded to the fusionACS github repository. This allows users outside of the HPC to access a pseudo-sample of the underlying, complete database version in the HPC. A public release pseudo-sample is specifically designed for public, non-HPC users of the fusionACS package. Users outside the HPC can retrieve this public dataset using fusionACS::get_microdata().

Importantly, compileVersion() is also necessary to make the most recent state of the database available to fusionACS assemble() and analyze() functions within the HPC environment. If a user has performed a new round of fusion and wishes to work with that data within the HPC, they will need to run compileVersion() first to get access to the results of their new fusion outputs.

When compileVersion() is successfully executed, it automatically calls fusionACS::set_directory() to set the fusionACS package data directory to the new database version. That way, any subsequent data assembly or analysis via the fusionACS package in the HPC automatically uses the latest version of the fusionACS database.

See the compileVersion() documentation for details.

Step 7: Perform Analysis

The HPC environment is the only locale capable of analyzing the complete, multi-implicate fusionACS database.

When working on a new project or analysis using fusionACS data, the general recommendation is to use the fusionACS package outside of the HPC first (utilizing the public pseudo-sample) to do data exploration, analysis development, prototyping, and testing. Using the public pseudo-sample will not yield accurate estimates – especially for small effective sample sizes – but they are reasonable for prototyping most analyses. Data assembly and analysis is much faster using the pseudo-sample.

Note: Analyses relying exclusively on native ACS variables yield valid point estimates and margins of error using the public pseudo-sample. However, any analysis incorporating fused donor variables or sub-PUMA geographic units requires HPC access for accurate estimates.

Once you are ready to begin working with the complete fusionACS database, you can move your code inside the HPC. You can store it at /analysis if you want, but there is no requirement to do so.

Functionality and code execution is identical inside and outside of the HPC. So any code that executes properly using the pseudo-sample will also execute using the full database.

See the fusionACS package documentation for details and examples.

Annex 1: ACS-PUMS Ingestion Details

fusionACS includes code that allows it to consistently ingest and process ACS-PUMS microdata across survey years. This includes parsing of the raw data dictionaries, conversion of integer codes into text labels, correct data typing (including ordered factors), and descriptions of all variables.

As of early 2026, ACS-PUMS microdata has been processed and ingested for years 2005-2024.

The household microdata excludes vacant and group quarter housing units. The person microdata includes all individuals, including those in group quarters.

The processed microdata generally include native ACS-PUMS variables as-is, but exclude redundant/unnecessary variables and allocation flags. Some variable descriptions from the codebook are modified for clarity. A small number of erroneous or suppressed values are imputed, valid blanks are replaced with sensible values, and there is limited modification to make variables more consistent and/or usable across time. For example, the categorical property tax (pre-2018) and property value (pre-2008) variables are coerced to numerical equivalents for consistency with more recent survey years.

The ACS-PUMS original housing unit identifier (“serialno”) is replaced with a unique, integer household ID (“hid”) variable for use within fusionACS. The person-level microdata includes a person ID (“pid”) variable that identifies unique persons within each household, starting with the reference person (pid = 1).

The processed microdata (both household and person-level variables) are stored on-disk as .fst files for maximum efficiency and speed. The processed ACS-PUMS microdata .fst files are typically about half the size of the raw, zipped microdata provided by the Census Bureau.

The associated data dictionaries are stored as binary .rds R data files. For example, the processed microdata files and data dictionaries for the 2005 ACS-PUMS are:

  • fusionData/survey-processed/ACS/2005/ACS_2005_H_processed.fst
  • fusionData/survey-processed/ACS/2005/ACS_2005_P_processed.fst
  • fusionData/survey-processed/ACS/2005/ACS_2005_H_dictionary.rds
  • fusionData/survey-processed/ACS/2005/ACS_2005_P_dictionary.rds

These files contain the ACS-PUMS variables as found in the original Census Bureau data sources with limited edits and omissions, as noted above.

In addition, code is included to compute a limited number of “custom” variables. The custom variables are included to help facilitate common analyses and provide useful variables that can be derived from the original ACS-PUMS variables. As of early 2026, there are only household-level custom variables defined, and they are located at (for example): fusionData/survey-processed/ACS/2005/ACS_2005_H_custom.fst

The definitions of custom variables are included in the appropriate data dictionary (ACS_2005_H_dictionary.rds) alongside definitions of native ACS-PUMS variables. The “custom” column provides a flag to identify the custom variables.

Custom variables are defined using the code stored in the Yale HPC at: /gpfs/milgram/project/rao/shared/fusionACS/fusionData/survey-processed/ACS/custom

Of particular note are the five custom variables generated by the custom housing.R scripts. These are:

  • rentval: Annual rental value of dwelling, including utilities, imputed and adjusted for owner-occupied units. Provides a consistent measure of the consumption or “use value” of housing for both renters and homeowners. Reported gross rents (inclusive of utilities) for renter-occupied units (“grntp”) are used to impute rental values for owner-occupied units on the basis of observable characteristics. Then an “owner premium” is applied to the imputed rental values using the technique of BEA (2019).
  • propval: Property value reported by owner, imputed and adjusted for renter-occupied units. Measure of property value across both renters and homeowners. Reported property values for owner-occupied units are used to impute property values for renter-occupied units on the basis of observable characteristics. Then a “renter discount” is applied to the imputed property values using the technique of BEA (2019).
  • proptax: Typical annual property tax, imputed for renters and coerced to a monotonic relationship with property value within each PUMA. Provides a consistent measure of property value for both renters and homeowners. Reported property values for owner-occupied units (“valp”) are used to impute property value for renter-occupied units on the basis of observable characteristics. Then a “renter discount” is applied to the imputed property values using the technique of BEA (2019).
  • propins: Typical annual property insurance, imputed for renters and uninsured owners and coerced to a monotonic relationship with property value within each PUMA. About 22% of owner-occupied units without a mortgage report being uninsured. Some unknown percentage of renter-occupied units are also uninsured by the owner. Since insurance premiums are imputed for these housing units, propins measures the typical cost of insuring a property regardless of its actual insurance status.
  • mortgage: Annual mortgage payment, principal and interest. Self-reported household mortgage outlay, adjusted to exclude property tax and home insurance. Includes both primary and secondary mortgages on the property.

Annex 2: Advanced Harmonization

Additional details regarding “advanced” settings using the harmony() app.

Bin breakpoints

The Bin breakpoints field is used to specify how a continuous variable should be “binned” to turn it into a categorical variable - usually to allow for harmonization with a factor variable in the other survey. This is useful when an identical concept is measured on a continuous scale in one survey (e.g. income in dollars) and as a factor variable in the other survey (e.g. income range).

  • Example: Select “moneypy” for RECS 2015 to see how the ACS “hincp” is binned to create harmony.

Adjustment

The Adjustment field provides a powerful way to modify or adjust variables to accommodate non-standard harmonies. Text in the Adjustment field is passed as-is to a dplyr::mutate() call within harmonize() that modifies the associated variable prior to any other manipulation. The text passed to the mutate() call can utilize any other variables in the microdata. This allows for complicated harmonies to be accommodated.

  • Example: Select “vehq” for CEI 2015-2019. In this case, the “vehq” (owned vehicles) and “vehql” (leased vehicles) variables in the CEI - both continuous are added together by specifying "vehq + vehql" in the Adjustment field. The result is then binned to create harmony with the ACS “veh” variable, which is a factor variable referring to all available vehicles, whether owned or leased.

Household aggregator

Sometimes it is possible to create harmony between a household-level (“H”) donor variable and a person-level (“P”) ACS variable, provided that the latter is aggregated at the household level. In such cases, the Household aggregator field tells harmonize() how to aggregate or summarize the person-level ACS variable within each household. This field is only applicable when the donor variable is a household variable and the selected ACS variable is person-level (person-level donor variables can always be harmonized directly with person-level ACS variables).

  • Simple example: Select “hhage” for RECS 2015 (Respondent/head-of-household age). The ACS “agep” variable can be used to create harmony, but it is a person-level variable. By setting the Household aggregator field to "reference" we instruct harmonize() to use the “reference person” value for “agep” to create a household-level variable analogous to “hhage”.
  • Advanced example: Select “numchild” for RECS 2015 (Number of household members age 17 or younger). The ACS “agep” variable can be used to create harmony. Bin breakpoints are used to re-assign each household member “agep” value to 1 if less than 18 and 0 if 18 or more (see the “Group” column in the associated spreadsheet). The Household aggregator field is then set to "sum" to instruct harmonize() to sum these values at the household level, which creates harmony with “numchild”.
  • Very advanced example: Select “as_comp1” for CEI 2015-2019 (Number of males age 16 and over). Again, the ACS “agep” variable can be used to create harmony (as above for “numchild”), but we need to additionally restrict the harmony to males only. This is done by using the Adjustment field to first set “agep” to 0 for all females, then bin the result, then sum at the household level.

Comments

The Comments field is used to leave helpful information about why the harmony was constructed as it was. Any harmony that makes use of one of the “advanced” settings should probably have a comment explaining the rationale.

About

Data backend for fusionACS platform

ummel.github.io/fusionData/

Resources

Readme
GPL-3.0 license
Activity

Stars

9 stars

Watchers

3 watching

Forks

5 forks
Report repository

Releases

Packages

Used by

Contributors

Languages