Spark-ml模型保存为PMML

spark版本2.1.3

maven设置

    <dependency><groupId>org.apache.spark</groupId><artifactId>spark-mllib_2.11</artifactId><version>2.1.3</version><exclusions><exclusion><groupId>org.jpmml</groupId><artifactId>pmml-model</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.jpmml</groupId><artifactId>jpmml-sparkml</artifactId><version>1.2.13</version></dependency>

spark-ml要去掉pmml-model依赖

模型训练

import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.spark.ml.Pipeline;
import org.apache.spark.ml.PipelineModel;
import org.apache.spark.ml.PipelineStage;
import org.apache.spark.ml.classification.RandomForestClassificationModel;
import org.apache.spark.ml.classification.RandomForestClassifier;
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator;
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator;
import org.apache.spark.ml.evaluation.RegressionEvaluator;
import org.apache.spark.ml.feature.VectorAssembler;
import org.apache.spark.ml.feature.VectorIndexer;
import org.apache.spark.ml.linalg.SparseVector;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.StructType;import org.dmg.pmml.PMML;
import org.jpmml.model.JAXBUtil;
import org.jpmml.sparkml.PMMLBuilder;
。。。String[] features=new String[]{"category", "future_day", "banner_min_time","banner_min_price","page_train", "page_flight", "page_bus", "page_transfer","start_end_distance", "total_transport", "high_railway_percent", "avg_time", "min_time","avg_price", "min_price","label_05060801", "label_05060701", "label_05060601", "label_02050601", "label_02050501", "label_02050401","is_match_category", "train_consumer_prefer", "flight_consumer_prefer", "bus_consumer_prefer"};VectorAssembler assembler = new VectorAssembler().setInputCols(features).setOutputCol("features");RandomForestClassifier rf = new RandomForestClassifier().setLabelCol("isclick").setFeaturesCol("features").setMaxDepth(7).setNumTrees(60).setSeed(2018).setMinInstancesPerNode(1);;Pipeline pipeline = new Pipeline().setStages(new PipelineStage[]{assembler, rf});PipelineModel pipelineModel = pipeline.fit(trainData);

保存PipelineModel模型

 pipelineModel.write().overwrite().save("D://model/random-forest");

hdfs保存方法也一样，换成hdfs路径就行了。

保存文件的PMML格式

        StructType schema = trainData.schema();PMML pmml = new PMMLBuilder(schema, pipelineModel).build();saveToLocalFile(pmml);
//        saveToHdfsFile(pmml);

    private void saveToLocalFile(PMML pmml) {String targetFile = "D://model/pmml/pipemodel";try (FileOutputStream fis = new FileOutputStream(targetFile)) {JAXBUtil.marshalPMML(pmml, new StreamResult(fis));} catch (JAXBException e) {e.printStackTrace();} catch (FileNotFoundException e) {e.printStackTrace();} catch (IOException e) {e.printStackTrace();}}

HDFS路径保存为PMML,fileSystem为HDFS文件系统

    private void saveToHdfsFile(PMML pmml) throws IOException {String targetFile = "/data/twms/traffichuixing/model/stage/pmml/rf-pipepmml";Path path = new Path(targetFile);try(FSDataOutputStream fos = fileSystem.create(path)) {JAXBUtil.marshalPMML(pmml, new StreamResult(fos));} catch (JAXBException e) {e.printStackTrace();logger.error(e.getMessage());}}

Spark-ml模型保存为PMML相关推荐

[PMML] LightGBM模型保存为PMML文件，通过Java或者Python调用
1.LightGBM 多分类 import os import lightgbm as lgb from sklearn import datasets from sklearn.model_sele ...
scala-MLlib官方文档---spark.ml package--ML Pipelines+Collaborative Filtering+Frequent Pattern Mining
三. ML Pipeline Main concepts in Pipelines(管道中的主要概念) MLlib对用于机器学习算法的API进行了标准化,从而使将多种算法组合到单个管道或工作流中变得更 ...
【Spark】Spark训练Lr模型，并保存为Pmml
scala版本spark构建的Lr模型: 一.问题背景需要构建一个Lr模型来进行物品的Ctr预测. 二.解决方案由于我们训练的数据量较多,所以首先考虑采用spark来构建模型并测试训练,这 ...
ML之nyoka：基于nyoka库利用LGBMClassifier模型实现对iris数据集训练、保存为pmml模型并重新载入pmml模型进而实现推理
ML之nyoka:基于nyoka库利用LGBMClassifier模型实现对iris数据集训练.保存为pmml模型并重新载入pmml模型进而实现推理目录基于nyoka库利用LGBMClassifi ...
ML之xgboost：基于xgboost(5f-CrVa)算法对HiggsBoson数据集(Kaggle竞赛)训练(模型保存+可视化)实现二分类预测
ML之xgboost:基于xgboost(5f-CrVa)算法对HiggsBoson数据集(Kaggle竞赛)训练(模型保存+可视化)实现二分类预测目录数据集简介输出结果设计思路核心代码数 ...
python环境下，XGBoost模型文件转pmml格式用于部署
一. python环境下生成PMML 1. sklearn接口的xgboost 需要用sklearn2pmml包来生成pmml文件,一共有两种方法: 方法一: 在管道pipeline中进行fit后保存 ...
基于Spark ML 聚类分析实战的KMeans
2019独角兽企业重金招聘Python工程师标准>>> 聚类分析是一个无监督学习 (Unsupervised Learning) 过程, 一般是用来对数据对象按照其特征属性进行分组, ...
使用spark ml pipeline进行机器学习
一.关于spark ml pipeline与机器学习一个典型的机器学习构建包含若干个过程 1.源数据ETL 2.数据预处理 3.特征选取 4.模型训练与验证以上四个步骤可以抽象为一个包括多个步骤的 ...
apache beam_Apache Beam ML模型部署
apache beam This blog post builds on the ideas started in three previous blog posts. 这篇博客文章基于之前三篇博 ...

Spark-ml模型保存为PMML

Spark-ml模型保存为PMML相关推荐

最新文章

热门文章