聚集索引扫描97%

by Ben Weber

通过本·韦伯

聚集前1％：R中的资产分析 (Clustering the Top 1%: Asset Analysis in R)

The recent tax reform bill passed in the US has raised a lot of questions about wealth distribution in the country. While there’s been a lot of focus on how the tax plan will impact income, there’s been less attention focused on how this plan impacts the assets of wealthy households.

美国最近通过的税制改革法案引发了有关美国财富分配的许多问题。 尽管人们对税收计划将如何影响收入的关注很多，但对税收计划如何影响富裕家庭的资产的关注却很少。

The goal of this post is to show how the R programming language can be used to data mine publicly available sources to better understand the net worth of affluent households in the US. Using data from the 2016 Federal Reserve Survey of Consumer Finances, we investigate the following questions:

这篇文章的目的是展示如何使用R编程语言对可公开获取的数据进行数据挖掘，以更好地了解美国富裕家庭的净资产。 使用2016年美联储消费者财务调查的数据，我们调查了以下问题：

• How rich are the top 1% and top 0.1% of households?前1％和前0.1％的家庭有多少富翁？
• Are there different types of millionaires in the US?美国有不同类型的百万富翁吗？
• How do asset allocations differ across different net worth segments?不同资产净值部分的资产分配有何不同？

To answer these questions, we present descriptive statistics of this survey data and perform cluster analysis on affluent households, which we identify as households with a net worth of more than $1,000,000 USD.

为了回答这些问题，我们将提供此调查数据的描述性统计数据，并对富裕家庭进行聚类分析，我们将这些富裕家庭确定为净资产超过1,000,000美元的家庭。

Based on the survey data, our analysis shows that the net worth of the top 1% of households in the US is $10.4M and the net worth of the top 0.1% of households is $43.2M. This post presents an analysis of the different asset compositions of millionaires, and shows how asset allocations differ between the top 10%, 1%, and 0.1% of households in the US. The R source code used to produce all results and figures presented in this post is available as a Jupyter Notebook.

根据调查数据，我们的分析显示，美国收入最高的1％的家庭的净资产为1,040万美元，收入最高的0.1％的家庭的净资产为4,320万美元。 这篇文章对百万富翁的不同资产构成进行了分析，并展示了美国前10％，1％和0.1％的家庭之间的资产分配有何不同。 可用于生成本文中呈现的所有结果和图形的R源代码可作为Jupyter笔记本获得 。

Setting Up the Environment To perform cluster analysis on the affluent households in the US, we use several packages available from the CRAN library for R. For exploratory data analysis, we like to use the R kernel for the Jupyter notebook, since it enables data scientists to easily store notebooks on GitHub and share findings with other teams.

设置环境为了对美国富裕家庭进行聚类分析，我们使用了CRAN库中提供的多个R软件包。对于探索性数据分析，我们喜欢对Jupyter Notebook使用R内核，因为它可以使数据科学家轻松将笔记本存储在GitHub上并与其他团队共享发现。

Setting up this environment is outside the scope of this post, but I’ve previously detailed our motivation for this setup in this post, and additional details for setting up Jupyter with R support are available here.

设置此环境不在本文讨论范围之内，但是我之前已经在本文中详细介绍了进行此设置的动机，并且可以在此处获得有关设置具有R支持的Jupyter的其他详细信息。

We’re now ready to start digging into the survey data to better understand the assets of affluent households in the US. To begin, we’ll load several libraries that will help us analyze the survey data and perform clustering.

现在，我们准备开始调查调查数据，以更好地了解美国富裕家庭的资产。 首先，我们将加载几个库，这些库将帮助我们分析调查数据并执行聚类。

The code block below shows the libraries that need to be loaded for executing this notebook. The readxl library is needed to read the source data and convert it into a data frame, the reldist and ENmisc libraries are used for computing distributions with weighted data sets, and the remaining libraries are used for cluster analysis.

下面的代码块显示了执行此笔记本需要加载的库。 需要readxl库来读取源数据并将其转换为数据帧， reldist和ENmisc库用于计算具有加权数据集的分布，而其余的库用于聚类分析。

library(readxl)     # for reading xlsx files library(reldist)    # for computing weighted statisticslibrary(ENmisc)     # for weighted box plotslibrary(plotly)     # for interactive plotslibrary(factoextra) # for factor mapslibrary(FactoMineR) # Principal Component Analysis (PCA)library(cluster)    # Clustering algorithms (CLARA)library(class)      # for KNN

Getting the DataThe next step is to download the data from the Federal Reserve website. The survey data is available as a zipped xlsx file. To download the data and load it into R as a data frame for analysis, we use the code block below. Since this is a large file, we make sure that we do not download it multiple times. The resulting file that we unzip is about 40MB and takes some time to load into a data frame.

获取数据下一步是从美联储网站下载数据。 调查数据以压缩的xlsx文件形式提供。 要下载数据并将其作为数据帧加载到R中进行分析，我们使用下面的代码块。 由于这是一个大文件，因此请确保不要多次下载。 我们解压缩后得到的文件约为40MB，需要一些时间才能加载到数据框中。

if (!file.exists("SCFP2016.xlsx")) {          download.file(   "https://www.federalreserve.gov/econres/files/scfp2016excel.zip",   "SCFP2016.zip")  unzip("scfp2016.zip")}

df <- read_excel("SCFP2016.xlsx")

Summary Statistics Now that we have the survey data loaded into R, we can start to analyze the asset allocation of the richest households in the US. We’ll start with some summary stats: how many survey participants and how many households are represented by this survey? Counting the number of rows in the data frame answers the first questions (31.2k), and adding up the weights of all of the survey respondents answers the second question (126M).

摘要统计信息现在我们已经将调查数据加载到R中，我们可以开始分析美国最富裕家庭的资产分配。 我们将从一些摘要统计数据开始：此调查代表多少个调查参与者和多少个家庭？ 计算数据框中的行数可以回答第一个问题(31.2k)，将所有调查受访者的权重相加可以回答第二个问题(126M)。

# How many survey participants?nrow(df)

# How many households does the survey represent?floor(sum(df$WGT)))

# What is the weighted mean of household net worth?floor(sum(df$NETWORTH*df$WGT)/sum(df$WGT)))

# what is the median NW in US?  reldist::wtd.quantile(df$NETWORTH, q=0.5, weight = df$WGT)

# who is the 1%? reldist::wtd.quantile(df$NETWORTH, q=0.99, weight = df$WGT)

# top 0.1% Ultra-high net worth households  reldist::wtd.quantile(df$NETWORTH, q=0.999, weight = df$WGT)

To answer questions about averages, such as what is the mean household net worth, we need to use weighted statistics (since the weight of one survey respondent can be much larger than others). To compute the mean net worth, we can use built-in R functions, which returns a value of $690k. However, since net worth is much closer to a log normal than normal distribution, we should use other approaches.

要回答有关平均值的问题，例如家庭平均资产净值是多少，我们需要使用加权统计信息(因为一个调查对象的权重可能比其他人大得多)。 要计算平均净资产，我们可以使用内置的R函数，该函数返回的价值为69万美元。 但是，由于净资产比正态分布更接近对数正态，因此我们应该使用其他方法。

To compute the median value with weighted responses, we use the reldist library, which assigns more support to respondents with larger weights and less support to respondents with lower weights. When using this approach to compute the weighted median, we find that the 50th percentile for household net worth in the US is $97k.

要使用加权响应计算中位数，我们使用reldist库，该库为权重较大的受访者分配更多支持，而权重较低的受访者则分配较少的支持。 当使用这种方法计算加权中位数时，我们发现美国家庭净资产的第50个百分位数为9.7万美元。

The net worth of the top 1% is $10.4M and the net worth of the top 0.1% is $43.2M. We use the wtd.quantile function to compute these descriptive statistics, and the code sample above uses the fully quantified function names, due to collisions with the ENmisc package.

前1％的净资产为1,040万美元，前0.1％的净资产为4,320万美元。 我们使用wtd.quantile函数来计算这些描述性统计信息，并且由于与ENmisc包发生冲突，上述代码示例使用了完全量化的函数名。

Demographic Data The survey data provides a number of different demographic variables that can be used to analyze net worth by different factors. These variables include race, marital status, education level, employment status, and others. The goal of this post is to show how asset allocation varies by net worth segment, and analysis of how these demographic factors impact net worth is left as an exercise for the reader.

人口统计数据调查数据提供了许多不同的人口统计变量，可用于通过不同因素分析净资产。 这些变量包括种族，婚姻状况，受教育程度，就业状况等。 这篇文章的目的是展示资产分配如何随净资产细分而变化，并留待读者分析这些人口因素如何影响净资产。

# filter on ages 30 - 84, and group into 5-year bucketsdata <- df[df$AGE >= 30 & df$AGE < 85, ]ages <- floor(data$AGE/5.0)*5

# plot the weighted box plotwtd.boxplot(log10(1 + data$NETWORTH) ~ ages, data = data,   weights = data$WGT, main = 'Net Worth by Age',  xlab="Age", ylab="Net Worth (Log10)")

One demographic variable that we did explore is the impact of age on household net worth. As expected, the median net worth does increase as the head of household becomes older, with net worth plateauing around 60 years old.

我们所做的一项人口统计学变量是年龄对家庭净资产的影响。 正如预期的那样，随着户主年龄的增长，中位数净资产的确会增加，而其净资产却稳定在60岁左右。

The code above shows how to display a box plot of the survey data by age, using the ENmisc package to compute weighted distributions. The results of this plot show that the median net worth of households in the US are $114k at 40, $163k at 50, and $243k at 60.

上面的代码显示了如何使用ENmisc软件包按年龄显示调查数据的箱形图，以计算加权分布。 该图的结果显示，美国家庭的中值净资产在40岁时为11.4万美元，在50岁时为16.3万美元，在60岁时为24.3万美元。

Asset Allocation The next step in the notebook is to evaluate the asset allocation of different net worth segments. For this analysis, we define a segment based on the log10 value of the household net worth. That means all 5-figure households get grouped together, all 6-figure households get grouped together, as so on.

资产分配笔记本中的下一步是评估不同净资产段的资产分配。 对于此分析，我们根据家庭净资产的log10值定义一个细分。 这意味着将所有5位数字的家庭分组在一起，将所有6位数字的家庭分组在一起，依此类推。

# normalize assets by total financial and non-financial amountshouseholds <- data.frame(  LIQ = df$LIQ/assets,  ...  BUS = df$BUS/assets,    OTHNFIN = df$OTHNFIN/assets

# split into net worth segments, and compute mean distributions nw <- floor(log10(households$netWorth))segment <- ifelse(nw == 4, "     $10k",       ifelse(nw == 5, "    $100K",       ifelse(nw == 6, "   $1M",                     ifelse(nw == 7, "  $10M",                     ifelse(nw == 8, " $100M", "$1B+")))))

results <- as.data.frame((aggregate(households,list(segment),mean)))

# plot the resultsplot <- plot_ly(results, x = ~Group.1, y = ~100*LIQ, type = 'bar', name = 'Liquid') %>%  add_trace(y = ~100*CDS, name = 'Certificates of Deposit') %>%  add_trace(y = ~100*NMMF, name = 'Mutual Funds') %>%  ...  add_trace(y = ~100*BUS, name = 'Business Interests') %>%  layout(yaxis = list(title = '% of Assets',     ticksuffix = "%"), xaxis = list(title = "Net Worth"),    title = "Asset Allocation by Net Worth", barmode = 'stack')

The code above computes the asset allocation by household, by dividing the amount of assets for a household, such as business equity (BUS) over the total number of financial and non-financial assets, excluding debt. The “…” pattern is used to indicate that multiple rows have been excluded from the code snippet that are listed in the full notebook. The second code block groups the households into different net worth segments, and the third block plots the results as shown below.

上面的代码通过将家庭的资产数量(例如，企业股权( BUS ))除以金融和非金融资产的总数(不包括债务)来计算按家庭分配的资产。 “ …”模式用于指示从完整笔记本中列出的代码段中排除了多行。 第二个代码块将家庭划分为不同的净资产段，第三个代码块绘制结果，如下所示。

The results show that the wealthy, ultra wealthy, and billionaires have large amounts of assets in business equity (stock and options). The wealthy have only a small percentage of assets in retirement funds, and instead have assets in stock, mutual funds, and residential and commercial real estate.

结果表明，富人，超富人和亿万富翁拥有大量的企业股权资产(股票和期权)。 富人只拥有一小部分退休基金资产，而拥有股票，共同基金以及住宅和商业房地产资产。

Clustering MillionairesSo far, we’ve looked at aggregate statistics of wealthy households, but that doesn’t tell us about the different types of affluent households. To understand how assets vary across affluent homes, we can use cluster analysis. One of the most useful ways to visualize the difference between instances in a sample population is to use factor maps to visualize the variance in the population.

聚集百万富翁到目前为止，我们已经研究了富裕家庭的汇总统计数据，但这并不能告诉我们有关富裕家庭的不同类型。 要了解富裕家庭中资产的变化情况，我们可以使用聚类分析。 可视化样本总体中实例之间差异的最有用方法之一是使用因子图可视化总体中的方差。

# filter on affluent households, and print the total numberaffluent <- households[households$netWorth >= 1000000, ]cat(paste("Affluent Households: ", floor(sum(affluent$weight))))

# plot a Factor Map of assets fviz_pca_var(PCA(affluent,  graph = FALSE), col.var="contrib",      gradient.cols = c("#00AFBB", "#E7B800", "#FC4E07"), repel = TRUE)+  labs(title ="Affluent Households - Assets Factor Map")

The code above first filters survey respondents to affluent households, with a net worth of more than $1M USD, and then plots a factor map using principal component analysis (PCA). The figure below shows how the different assets impact the trajectory of plotting a household across the two principal components discovered via PCA.

上面的代码首先过滤了净资产超过100万美元的富裕家庭的调查对象，然后使用主成分分析(PCA)绘制了因子图。 下图显示了不同的资产如何影响通过PCA发现的两个主要组成部分上绘制家庭的轨迹。

The results plotted below show that there are a few different assets groups that vary across affluent net worth. The most significant factor is business equity. Some other groupings of factors include investment assets (STOCKS, BONDS) and real estate assets/retirement funds.

下面绘制的结果表明，有一些不同的资产组在富裕的净资产中有所不同。 最重要的因素是企业股权。 其他一些因素类别包括投资资产(STOCKS，BONDS)和房地产资产/退休基金。

How many clusters to use?We’ve now shown signs that there are different types of millionaires, and that assets vary based on net worth segments. To understand how asset allocation differs by net worth segment, we can use cluster analysis. We first identify clusters in the affluent survey respondents, and then apply these labels to the overall population of survey respondents.

要使用几个集群？ 现在我们已经显示出迹象，表明有不同类型的百万富翁，并且资产根据净资产细分而变化。 要了解资产分配随净值细分的差异，我们可以使用聚类分析。 我们首先在富裕的调查受访者中识别集群，然后将这些标签应用于调查受访者的总体。

k <- 7res.hc <- eclust(households[sample(nrow(households), 1000), ],  "hclust", k = k, graph = FALSE) fviz_dend(res.hc, rect = TRUE, show_labels = FALSE)

To determine how many clusters to use, we created a cluster dendrogram using the code snippet above, shown as the header image of this post. We also varied the number of clusters, k, until we had the largest number of distinctly identifiable clusters.

为了确定要使用多少个群集，我们使用上面的代码片段创建了群集树状图，如本文的标题图片所示。 我们还改变了簇的数量k ，直到我们拥有最大数量的可明显识别的簇。

If you’d prefer to take a quantitative approach, you can use the fviz_nbclust function, which computes the optimal number of clusters using a silhouette metric. For our analysis, we decided to use 7 clusters.

如果您想采用定量方法，则可以使用fviz_nbclust函数，该函数使用轮廓度量来计算最佳聚类数。 为了进行分析，我们决定使用7个群集。

clarax <- clara(affluent, k)fviz_cluster(clarax, stand = FALSE, geom = "point", ellipse = F)

To cluster the affluent households into unique groupings, we used the CLARA algorithm. A visualization of the different clusters is shown below. The results are similar to PCA and the factor map approach discussed above.

为了将富裕家庭聚类为唯一的分组，我们使用了CLARA算法 。 下面显示了不同群集的可视化。 结果类似于上面讨论的PCA和因子映射方法。

Cluster Descriptions Now that we’ve determined how many clusters to use, it’s useful to inspect the clusters and assign qualitative labels based on the feature sets. The code snippet below shows how to compute the average feature values for the 7 different clusters.

集群说明现在，我们确定了要使用的集群数量，检查集群并基于功能集分配定性标签非常有用。 下面的代码段显示了如何计算7个不同聚类的平均特征值。

groups <- clarax$clusteringresults <- as.data.frame(t(aggregate(affluent,list(groups),mean)))  results[2:18,]

The results of this code block are shown below. Based on these results, we came up with the following cluster descriptions:

该代码块的结果如下所示。 基于这些结果，我们提出了以下群集描述：

• V1: Stocks/Bonds — 31% of assets, followed by home and mutual fundsV1：股票/债券-占资产的31％，其次是房屋和共同基金
• V2: Diversified — 53% busequity, 10% home and 9% in other real estateV2：多元化-53％的股权，10％的住房和9％的其他房地产
• V3: Residential Real Estate — 48% of assetsV3：住宅房地产-资产的48％
• V4: Mutual Funds — 50% of assetsV4：共同基金-资产的50％
• V5: Retirement — 48% of assetsV5：退休-资产的48％
• V6: Business Equity — 85% of assetsV6：企业股权-资产的85％
• V7: Commercial Real Estate — 59% of assetsV7：商业房地产-资产的59％

With the exception of cluster V7, containing only 3% of the population, most of the clusters are relatively even in size. The second smallest cluster represents 12% of the population while the largest cluster represents 20%. You can use table(groups) to show the unweighted cluster population sizes.

除了仅包含3％人口的集群V7外，大多数集群的大小相对均等。 第二小的集群代表了12％的人口，而最大的集群代表了20％的人口。 您可以使用表(组)显示未加权的群集总数。

Cluster Populations by Net Worth SegmentsThe last step in this analysis is to apply the different cluster assignments to the overall population, and to group the populations by net worth segments. Since we trained the clusters on only affluent households, we need to use a classification algorithm to label the non-affluent households in the population. The code snippet below uses knn to accomplish this task.

按净值细分划分的聚类人口此分析的最后一步是将不同的聚类分配应用于总人口，并按净资产分类对人口进行分组。 由于我们仅在富裕家庭中培训了集群，因此我们需要使用分类算法来标记人口中的非富裕家庭。 下面的代码片段使用knn完成此任务。

The remaining code blocks compute the number of households that are classified as each cluster, for each of the net worth segments.

剩余的代码块针对每个净资产细分计算分类为每个集群的家庭数量。

# assign all of the households to a cluster groups <- knn(train = affluent, test = households,   cl = clarax$clustering, k = k, prob = T, use.all = T)

# figure out how many households are in each cluster clusters <- data.frame(  c1 = ifelse(groups == 1, weights, 0),   ...   c7 = ifelse(groups == 7, weights, 0) )

# assign each household to a net worth cluster nw <- floor(2*log10(nwHouseholds))/2results <- as.data.frame(t(aggregate(clusters,list(nw),sum)))

# compute the number of households that belong to each segmentresults$V1 <- results$V1/sum(ifelse(nw == 4, weights, 0))...results$V11 <- results$V11/sum(ifelse(nw == 9, weights, 0))

# plot the results plot <- plot_ly(results, x = ~10^Group.1, y = ~100*c1, type = 'scatter', mode = 'lines', name = "Stocks") %>%  add_trace(y = ~100*c2, name = "Diversified") %>%  ...  add_trace(y = ~100*c7, name = "Commercial R.E.") %>%  layout(yaxis = list(title = '% of Households', ticksuffix = "%"),     xaxis = list(title = "Net Worth ($)", type = "log"),      title  = "Cluster Populations by Net Worth")

The results of this process are shown in the figure below. The chart shows some obvious and some novel results: home ownership and retirement funds make up the majority of assets for non-affluent households, there is a relatively even mix of clusters around $2M (excluding commercial real estate and business equity), and business equity dominates net worth for the ultra-wealthy households, followed by other investment assets.

下图显示了该过程的结果。 图表显示了一些明显和新颖的结果：房屋所有权和退休基金占非富裕家庭的大部分资产，大约200万美元左右的集群组合(不包括商业房地产和商业股权)以及企业净资产占超富裕家庭净资产的主导，其次是其他投资资产。

SummaryIn this post we used R to download and analyze data from the 2016 Federal Reserve survey of consumer finances to understand how wealthy the top households are in the US, and to cluster affluent households by asset allocation. We identified 7 distinct groups of millionaires, and showed how the distribution of clusters varies based on net worth segment. Please keep in mind that the results presented are from weighted survey data, and may not be representative of the overall US population.

总结在本文中，我们使用R来下载和分析2016年美联储消费者财务调查的数据，以了解美国顶级家庭的富裕程度，并通过资产分配对富裕家庭进行聚类。 我们确定了7个不同的百万富翁群体，并展示了根据净资产细分群体的分布情况。 请记住，显示的结果来自加权调查数据，可能并不代表美国总体人口。

Ben Weber is a principal data scientist at Zynga. We are hiring!

Ben Weber是Zynga的首席数据科学家。 我们正在招聘 ！

翻译自: https://www.freecodecamp.org/news/clustering-the-top-1-asset-analysis-in-r-6c529b382b42/

聚集索引扫描97%