呀，要解析的xml文件有1G大，怎么办？

上个月工作中，遇到了一个超大的xml文件解析工作。然后按照以往的方法jdom试了一下，然后jvm内存溢出了。没得办法，只有上网寻找万能的网友们了。花了接近一上午的时间终于写好了一个实用符合编码习惯的工具给大家分享一下。

需要引入的jar包：

 <dependency><groupId>javax.persistence</groupId><artifactId>persistence-api</artifactId><version>1.0.2</version></dependency><dependency><groupId>org.apache.commons</groupId><artifactId>commons-lang3</artifactId><version>3.4</version></dependency><dependency><groupId>org.jdom</groupId><artifactId>jdom</artifactId><version>2.0.2</version></dependency>

第一段代码，

package com.kernel.util;import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;/*** 自定义xml解析工具* 更新时间：2019/9/6*/
public class MyXmlHandler extends DefaultHandler {private Class<?> clazz;             //反射对象private List<Object> list;          //解析结果private String element;             //获取的节点的名称private List<String> fieldList;     //反射对象的字段的名称private Object entity;              //实例化的反射对象private String rootName;            //xml中相对根节点private StringBuffer value;         //现有标签里的值，预防ch[]数组里面获取到的不是一个完整的xmlpublic MyXmlHandler(Object entity) {initHandler(entity.getClass());}public MyXmlHandler(Class<?> clazz) {initHandler(clazz);}public MyXmlHandler(Object entity, String rootName) {initHandler(entity.getClass(), rootName);}public MyXmlHandler(Class<?> clazz, String rootName) {this.initHandler(clazz, rootName);}private void initHandler(Class<?> clazz) {initHandler(clazz, clazz.getSimpleName());}private void initHandler(Class<?> clazz, String rootName) {this.clazz = clazz;this.rootName = rootName;fieldList = new ArrayList<>();Field[] fields = clazz.getDeclaredFields();for (Field f : fields) {fieldList.add(f.getName());}}@Overridepublic void startDocument() throws SAXException {super.startDocument();list = new ArrayList<>();}@Overridepublic void endDocument() throws SAXException {super.endDocument();fieldList.clear();}@Overridepublic void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {super.startElement(uri, localName, qName, attributes);if (qName.equalsIgnoreCase(rootName)) {try {entity = clazz.newInstance();} catch (Exception e) {System.out.println("init new entity error：" + clazz.getName() + "\t" + qName + "\t" + e.getMessage());}}value = new StringBuffer();element = qName;}@Overridepublic void endElement(String uri, String localName, String qName) {if (element != null && entity != null) {initEntityField(value.toString());}if (qName.equalsIgnoreCase(rootName)) {list.add(entity);}element = null;}@Overridepublic void characters(char[] ch, int start, int length) {String content = new String(ch, start, length);value.append(content);}/*** 给对象的属性赋值** @param elementValue 现有字段的值*/private void initEntityField(String elementValue) {try {if (fieldList.stream().anyMatch(s -> s.equalsIgnoreCase(element))) {String name = fieldList.stream().filter(s -> s.equalsIgnoreCase(element)).findFirst().get();Field field = clazz.getDeclaredField(name);field.setAccessible(true);Object value = getValidValue(field, elementValue);field.set(entity, value);}} catch (Exception e) {System.out.println("set entity`s value error：" + clazz.getName() + "\t" + element + "\t" + e.getMessage());}}/*** 获取实例对象的属性的有效值,可根据需要继续增加其他基本类型*/private static Object getValidValue(Field field, String content) {Object value = null;String flag = field.getGenericType().getTypeName();switch (flag) {case "java.lang.String":value = content;break;case "int":value = Integer.parseInt(content);break;case "java.util.Date":value = DateUtil.parseDate(content);break;default:break;}return value;}public List<Object> getList() {return list;}}

第二处工具类封装上述工具的代码，注意是readXml方法封装的上述代码。

package com.kernel.util;import com.kernel.entity.User;
import org.apache.commons.lang3.time.DateFormatUtils;
import org.jdom2.Document;
import org.jdom2.Element;
import org.jdom2.input.SAXBuilder;import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;public class XmlUtil {/*** 生成xml** @param element 数据区分节点* @param obj     对象的实例*/public static void initCreateXML(Element element, Object obj) {//反射对象的属性，获取所有字段Field[] fields = obj.getClass().getDeclaredFields();for (Field f : fields) {try {f.setAccessible(true);Element e = new Element(f.getName().toUpperCase());Object o = f.get(obj);if (o instanceof Number) {e.setText((Number) o + "");//装箱后建议使该方法} else if (o instanceof Date) {//commons-lang3.jar中的方法String d = DateFormatUtils.format((Date) o, "yyyy-MM-dd HH:mm:ss");e.setText(d);} else {e.setText(String.valueOf(o));}element.addContent(e);} catch (IllegalAccessException e1) {e1.printStackTrace();}}}/*** 解析xml文件*/public static List<Object> readXml(File file, Object entity) {return readXml(file, entity, null);}/*** 解析xml文件，反射对象的字段名同xml的节点名称。* 便于减少内存占用，快速解析xml文件，比jdom2的全文加载更快** @param file     xml文件* @param entity   xml反射的对象* @param rootName 相对节点名称*/public static List<Object> readXml(File file, Object entity, String rootName) {List<Object> list = new ArrayList<>();try {SAXParserFactory factory = SAXParserFactory.newInstance();factory.setNamespaceAware(true);factory.setValidating(true);SAXParser parser = factory.newSAXParser();MyXmlHandler handler;if (rootName == null) {handler = new MyXmlHandler(entity);} else {handler = new MyXmlHandler(entity, rootName);}parser.parse(file, handler);list.addAll(handler.getList());} catch (Exception e) {e.printStackTrace();}return list;}public static void readXmlByJDOM(File file){try {SAXBuilder saxBuilder=new SAXBuilder();InputStream in=new FileInputStream(file);Document doc=saxBuilder.build(in);Element root=doc.getRootElement();List<Element>elementList=root.getChildren();System.out.println(elementList.size());} catch (Exception e) {e.printStackTrace();}}public static void main(String[] args) {File file=new File("F:\\test\\fileTest\\user01.xml");
//        readXmlByJDOM(file);List<Object> users=readXml(file,new User());System.out.println(users.size());}}

第三处代码：user实体类，字段上的一些注解可以先不管

package com.kernel.entity;import com.fasterxml.jackson.annotation.JsonFormat;
import org.springframework.context.annotation.PropertySource;
import org.springframework.format.annotation.DateTimeFormat;import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;
import java.io.Serializable;
import java.util.Date;
//persistence-api
@Entity
@Table(name = "user")
public class User implements Serializable {@Id@Column(name = "id")private String id;@Column(name = "username")private String username;@Column(name = "gender")private String gender;@Column(name = "age")private int age;@Column(name = "phone")private String phone;@Column(name = "email")private String email;@Column(name = "birth")@DateTimeFormat(pattern = "yyyy-MM-dd")//前台到后端的转化private Date birth;@Column(name = "addtime")private Date addtime;@Column(name = "role")private String role;public String aq;private static String ad;
//    private static final String ad1 = "123tttvvv";public User() {}public User(String id, String username, String gender, int age, String phone, String email, Date birth, Date addtime, String role) {this.id = id;this.username = username;this.gender = gender;this.age = age;this.phone = phone;this.email = email;this.birth = birth;this.addtime = addtime;this.role = role;}public String getId() {return id;}public void setId(String id) {this.id = id;}public String getUsername() {return username;}public void setUsername(String username) {this.username = username;}public String getGender() {return gender;}public void setGender(String gender) {this.gender = gender;}public int getAge() {return age;}public void setAge(int age) {this.age = age;}public String getPhone() {return phone;}public void setPhone(String phone) {this.phone = phone;}public String getEmail() {return email;}public void setEmail(String email) {this.email = email;}@JsonFormat(timezone = "GMT+8",pattern = "yyyy-MM-dd")//后端到前端的转化public Date getBirth() {return birth;}public void setBirth(Date birth) {this.birth = birth;}public Date getAddtime() {return addtime;}public void setAddtime(Date addtime) {this.addtime = addtime;}public String getRole() {return role;}public void setRole(String role) {this.role = role;}
}

测试的xml文件模板，为减少排版，这里就少显示点

<?xml version="1.0" encoding="utf-8"?>
<USERS version="1.0"><user><ID>a0</ID><USERNAME>jack0c5e5e8e-64f5-4346-9c3f-83aad186c7d3</USERNAME><GENDER>m</GENDER><AGE>12</AGE><PHONE>110</PHONE><EMAIL>86da9724-70a5-443f-a9f8-5cf86f7c4cec110@com</EMAIL><BIRTH>2019-06-02 14:51:16</BIRTH><ADDTIME>2019-06-02 14:51:16</ADDTIME><ROLE>员工0</ROLE><AQ>null</AQ><AD>null</AD></user><user><ID>a1</ID><USERNAME>jack9bdc3093-efa6-46ad-b877-da3c256f5b6a</USERNAME><GENDER>m</GENDER><AGE>12</AGE><PHONE>110</PHONE><EMAIL>c0d1e737-2cec-41d5-8629-b5db44ccc640110@com</EMAIL><BIRTH>2019-06-02 14:51:17</BIRTH><ADDTIME>2019-06-02 14:51:17</ADDTIME><ROLE>员工1</ROLE><AQ>null</AQ><AD>null</AD></user>
</USERS>

呀，要解析的xml文件有1G大，怎么办？相关推荐

PMS解析AndroidManifest.xml文件的过程
一.前言前段时间在看当下主流的安卓插件化技术原理的时候,发现目前插件化技术对于四大组件的处理基本都是通过代理来实现的(动态注册的广播接收器除外).简单说就是需要先在主包里预埋一个注册在 ...
xml的学习和使用python解析读取xml文件
1. XML的介绍 XML 指可扩展标记语言(EXtensible Markup Language). 和json类似也是用于存储和传输数据,还可以用作配置文件. 类似于HTML超文本标记语言,但是H ...
dom4j解析RDF/XML文件
dom4j解析RDF/XML文件 xml <?xml version="1.0" encoding="UTF-8" ?> <rdf:RDF x ...
用Python解析AndroidManifest.xml文件找MainActivity
写在前面的话这个是一篇水博客,因为我知道只要是个大二的小朋友就可以搞定,我大二就一天在搞这些事情,今天纯属吃饱了撑着,想敲键盘,等实验结果然后随便写写.我知道大家都会的,不会的小朋友随便看看然后自己 ...
android xml解析demo,Android解析自定义xml文件--Sax解析xml文件，测试demo（方案二）...
转载请注明出处:http://blog.csdn.net/droyon/article/details/9346657 Sax解析xml 以下是测试Demo 运行程序类 public class Te ...
java解析xml文件失败,在Java中解析大型XML文件时找不到文件异常
我使用SAX(Simple API for XML)来解析XML文档.该文档是一个巨大的XML文件(dblp.xml - 1.46 GB),我写了几行解析器并在小文件上测试它,它的工作原理. Samp ...
c#语言用XmlReader解析超大XML文件
用XmlReader解析xml文件不用提前将内容加载到内存,而是逐行读取,减少内存占用,其高效率,能够有效读取超大xml文件.废话少说,直接上代码! xml文件格式样例: <?xml versi ...
解析一个xml文件并将数据存入数据库
解析xml文件并将文件存入数据库一.使用dom4j对xml文件进行解析 1.首先要有一个xml文件 2.需求 3.创建Books实例类 4.开始解析xml文件 5.测试代码: 6.结果: 二.进行数 ...
vtk实战(十四)——解析vtk XML 文件的内容
对于vtk XML格式的文件: .vtu, .vtp, .vts, .vtr, .vti, .vto, 解析其存储单元.单元数据. #include <vtkSmartPointer.h> ...

呀，要解析的xml文件有1G大，怎么办？

呀，要解析的xml文件有1G大，怎么办？相关推荐

最新文章

热门文章