java 自己实现解析处理user-agent 获取设备信息 ip-ua转化归因

前景提要

最近在做app投放的转化归因，几个搜索平台并不一定能获取到muid，于是需要用到ip-ua归因模式

网上粗略搜了一下，发现许多文章ua处理用的uaparser又或者user-agent-utils，随遂找来源码看了看，对手机设备的划分太粗糙了，不符合要求。

准备自己写一个。

公司内已经上线使用，目前没发现问题

User-agent格式

写之前首先要了解ua的格式
鉴于不同平台的格式也会有不同，所以对主流平台进行统计并观察规律也是必要的。

User-Agent通常格式：

自定义标识 (平台) 引擎版本浏览器版本号

eg:Mozilla/5.0 (iPhone; CPU iPhone OS 14_8_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1

实际上需要用到的就是第一个括号内的平台信息，内部信息用英文半角分号分开

常见格式：

(iPhone; CPU iPhone OS {os version})
(Linux; {os version}; {lang};{device name} Build/{core version})

截取ua中第一个括号的内容，依次分析内容。

这里和前端统计了常见流量来源的user-agent：

常见渠道user-agent

来源	预估占比	eg
safari	20%	Mozilla/5.0 (iPhone; CPU iPhone OS 14_8_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1
百度	17%	Mozilla/5.0 (Linux; Android 9; V1901A Build/P00610; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.27 SP-engine/2.37.0 baiduboxapp/12.28.5.10 (Baidu; P1 9) NABar/1.0
QQ	13%	Mozilla/5.0 (Linux; U; Android 9; zh-cn; LON-AL00 Build/HUAWEILON-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.72 MQQBrowser/12.1 Mobile Safari/537.36 COVC/045825
华为	13%	Mozilla/5.0 (Linux; Android 10; HarmonyOS; ELS-AN00; HMSCore 6.2.0.302) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.105 HuaweiBrowser/12.0.1.300 Mobile Safari/537.36 -----------------f分割-------------Mozilla/5.0 (Linux; Android 10; SEA-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.64 HuaweiBrowser/10.0.3.311 Mobile Safari/537.36
小米	10%	Mozilla/5.0 (Linux; U; Android 11; zh-cn; Redmi K30 Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.116 Mobile Safari/537.36 XiaoMi/MiuiBrowser/15.6.8-------------------------Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; en-us) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3 XiaoMi/MiuiBrowser/15.6.8
uc	10%	Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X; zh-CN) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/19B74 UCBrowser/13.6.4.1594 Mobile AliApp(TUnionSDK/0.1.20.4)------------------------------------------Mozilla/5.0 (Linux; U; Android 11; zh-CN; Mi 10 Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.108 UCBrowser/13.6.7.1148 Mobile Safari/537.36
其他	其他	随缘

可以看到，不同渠道还是有区别的。

代码也就完成了：

代码

/*** User-Agent处理类* 用于广告回传时处理ua从而获取设备信息** User-Agent通常格式：* 自定义标识 (平台) 引擎版本 浏览器版本号** eg:Mozilla/5.0 (iPhone; CPU iPhone OS 14_8_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1** 实际上用到的就是第一个括号内的平台信息，内部信息用英文半角分号分开**/
@Slf4j
public class UserAgentUtils {/*** 识别操作系统的正则，只配了ios，鸿蒙，android* 配置正则存在顺序（短路）*/private  List<StrategyRule> systemInfoStrategy;/*** 识别设备型号的正则* 配置正则存在顺序（短路）*/private  List<StrategyRule> deviceInfoStrategy;private static String OS_TYPE = "osType";private static String OS_VERSION = "osVersion";private static String OS_DEVICE = "mobileModel";private static String IOS_OS = "ios";public UserAgentUtils(List<StrategyRule> systemInfoStrategy, List<StrategyRule> deviceInfoStrategy){this.systemInfoStrategy = systemInfoStrategy;this.deviceInfoStrategy = deviceInfoStrategy;}public boolean compareByUserAgent(String ua1, String ua2, String ip1, String ip2){return analysisUserAgent(ua1, ip1).equalWith(analysisUserAgent(ua2, ip2));}/*** 先获取操作系统的类别（android x，ios xx.x.x，HarmonyOS）* 再去截取手机型号* @param ua* @param ip* @return*/public UserAgentDevice analysisUserAgent(String ua, String ip){UserAgentDevice agentDevice = new UserAgentDevice();agentDevice.setIp(ip);Map<String, String> systemInfo = handleByStrategy(ua, systemInfoStrategy);if(systemInfo == null){log.warn("不可识别的系统，可能是pc端之类没有做正则的类别，ua:{}",ua);} else {agentDevice.setMobileSystem(systemInfo.get(OS_TYPE));agentDevice.setSystemVersion(systemInfo.get(OS_VERSION));}if(IOS_OS.equals(agentDevice.getMobileSystem())){//ios 型号用ios+版本号拼接agentDevice.setMobileModel(agentDevice.getMobileSystem() + agentDevice.getSystemVersion().split(".")[0]);return agentDevice;}Map<String, String> osInfo = handleByStrategy(ua, deviceInfoStrategy);if(osInfo == null){log.warn("未获取到手机型号，可能是未配置的渠道样式，ua:{}",ua);} else {agentDevice.setMobileModel(osInfo.get(OS_DEVICE));}return agentDevice;}/*** 针对不同功能，ua处理方法不同* 这是一个统一处理方法* @param ua* @param strategyList* @return*/public static Map<String, String> handleByStrategy(String ua,List<StrategyRule> strategyList){Map<String, String> result = new HashMap<>();for(StrategyRule rule : strategyList){for(String regular : rule.getRegularList()){//匹配正则Matcher matcher = Pattern.compile(regular,Pattern.CASE_INSENSITIVE).matcher(ua);if(matcher.find()){int k = 1;//对捕获组做处理以适应不同返回for(StrategyFun f : rule.getFunList()){result.putAll(f.doAction(matcher.group(k++)));}return result;}}}return null;}@Datapublic static class StrategyRule{/*** 正则list*/private List<String> regularList;private List<StrategyFun> funList;}@Datapublic static class StrategyFun{/*** 对应直接返回键值对情况*/String name;String value;boolean valueFlag = false;/*** 对应replace的情况*/String r1;String r2;/*** 对不同策略对处理函数* @param* @return*/public Map<String, String> doAction(String str){if(valueFlag){return ImmutableMap.of(name, str);}if(value != null){return ImmutableMap.of(name, value);}if(r1 != null){return ImmutableMap.of(name, str.replaceAll(r1, r2));}return null;}}}

通过正则匹配设备，正则的测试在后面有测试类。

handleByStrategy，一个统一的处理函数，处理的方式放到funList中，可以扩展各种处理功能。实际上就是根据funList中的配置，返回对应的键值对，可以根据不同类别配置不同策略，比如ios的版本号做replace，鸿蒙没有版本号使用手机型号代替之类的。具体配置可以看后面。

这里只做了这次比较关心的手机系统和型号。

然后返回，对比

@Data
public class UserAgentDevice {/*** android或ios*/private String mobileSystem;private String systemVersion;/*** 手机型号，eg：Mi，HUAWEI, iphone*/private String mobileModel;private String ip;/*** ip必须相同，操作系统、操作系统型号、手机型号至少有两个相同(某些获取缺漏的情况)* @param* @return*/public boolean equalWith(UserAgentDevice x) {if(ip == null || !ip.equals(x.getIp())){return false;}int count = 0;if(mobileModel != null && mobileModel.equals(x.getMobileModel())){++count;}if(mobileSystem != null && mobileSystem.equals(x.getMobileSystem())){++count;}if(systemVersion != null && systemVersion.equals(x.getSystemVersion())){++count;}return count > 1;}
}

一个测试类，包括写好的正则

/*** 正则测试，正则真是写吐了* @author tianyangrui* @date 2021/12/14*/
public class UserAgentUtilsTest {private static List<String> uaCase = new ArrayList<>();private static void uadd(String s){ uaCase.add(s);}static void init(){uadd("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)");uadd("Mozilla/5.0 (iPhone; CPU iPhone OS 14_4_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 swan/2.26.0 swan-baiduboxapp/12.6.5.10 baiduboxapp/12.6.5.10 (Baidu; P2 14.4.2) ");uadd("Mozilla/5.0 (Linux; Android 8.1.0; DUB-AL00 Build/HUAWEIDUB-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.9 SP-engine/2.21.0 matrixstyle/0 lite baiduboxapp/5.4.0.10 (Baidu; P1 8.1.0) NABar/1.0");uadd("Mozilla/5.0 (Linux; Android 9.1.0) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.9 SP-engine/2.21.0 matrixstyle/0 lite baiduboxapp/5.4.0.10 (Baidu; P1 8.1.0) NABar/1.0");uadd("Mozilla/5.0 (Linux; Android 10.1.0 ;) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.9 SP-engine/2.21.0 matrixstyle/0 lite baiduboxapp/5.4.0.10 (Baidu; P1 8.1.0) NABar/1.0");uadd("Mozilla/5.0 (Linux; U; Android 10; zh-CN; HLK-AL00 Build/HONORHLK-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/69.0.3497.100 UWS/3.22.2.28 Mobile Safari/537.36 UCBS/3.22.2.28_210922181100 ChannelId(0) NebulaSDK/1.8.100112 Nebula AlipayDefined(nt:WIFI,ws:360|0|3.0) AliApp(AP/10.2.36.8000) AlipayClient/10.2.36.8000 Language/zh-Hans useStatusBar/true isConcaveScreen/false Region/CNAriver/1.0.0");uadd("Mozilla/5.0 (Linux; Android 11; V2055A; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4280.141 Mobile Safari/537.36 VivoBrowser/10.3.18.0");uadd("Mozilla/5.0 (Linux; U; Android 11; zh-cn; PEGM00 Build/RKQ1.200903.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/70.0.3538.80 Mobile Safari/537.36 HeyTapBrowser/40.7.31.1");uadd("Mozilla/5.0 (Linux; U; Android 11; zh-CN; Mi 10 Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.108 UCBrowser/13.6.7.1148 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; U; Android 11; zh-cn; Redmi K30 Build/RKQ1.200826.002) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.116 Mobile Safari/537.36 XiaoMi/MiuiBrowser/15.6.8");uadd("Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; en-us) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3 XiaoMi/MiuiBrowser/15.6.8");uadd("Mozilla/5.0 (Linux; Android 10; HarmonyOS; ELS-AN00; HMSCore 6.2.0.302) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.105 HuaweiBrowser/12.0.1.300 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; Android 10; HarmonyOS; ELS-AN01) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.105 HuaweiBrowser/12.0.1.300 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; Android 11; NTH-AN00; HMSCore 6.2.0.302) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.105 HuaweiBrowser/12.0.1.300 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; Android 10; SEA-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.105 HuaweiBrowser/12.0.1.300 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; Android; SEA-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.64 HuaweiBrowser/10.0.3.311 Mobile Safari/537.36");uadd("Mozilla/5.0 (Linux; U; Android ; zh-cn; LON-AL00 Build/HUAWEILON-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.72 MQQBrowser/12.1 Mobile Safari/537.36 COVC/045825");uadd("Mozilla/5.0 (Linux; U; Android ) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.72 MQQBrowser/12.1 Mobile Safari/537.36 COVC/045825");uadd("Mozilla/5.0 (Linux; U; Android 9; zh-cn; KB2000 Build/RP1A.201005.001) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.72 MQQBrowser/12.1 Mobile Safari/537.36 COVC/045825");uadd("Mozilla/5.0 (Linux; Android 9; V1901A Build/P00610; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.27 SP-engine/2.37.0 baiduboxapp/12.28.5.10 (Baidu; P1 9) NABar/1.0");uadd("Mozilla/5.0 (iPad; U; iPad OS 5_1_1 like Mac OS X; en-us) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3 XiaoMi/MiuiBrowser/15.6.8");uadd("Mozilla/5.0 (iPhone; CPU iPhone OS 14,4,2 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3 XiaoMi/MiuiBrowser/15.6.8");uadd("Mozilla/5.0 (iPhone; CPU iPhone OS 11-6 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3 XiaoMi/MiuiBrowser/15.6.8");uadd("dadashop/dadashop_version (iPhone; CPU iPhone OS 20_8_1)");uadd("dadashop/dadashop_version (iPhone; CPU iPhone OS 19_8_1 ;u)");uadd("dadashop/dadashop_version (iPhone; CPU iPhone OS 17_8_1;u)");uadd("dadaShop/8.22.0 (com.dada.store; build:615; iOS 18.8.1) Alamofire/8.22.0");}@Testpublic void iosTest(){init();String ppp = "[^a-zA-Z0-9 ]";List<String> pList = Arrays.asList("(ip[honead]+)(?:.*os\\s([\\w.,/\\-]+)\\slike|;\\sopera)","(ip[honead]+).*os\\s([\\w.,/\\-]+)[);]","(ios)\\s([\\w.,/\\-]+)[);]","(ios|ip[honead]+)\\s*([;)])");for(String rp : pList){int i=0;Pattern r = Pattern.compile(rp,Pattern.CASE_INSENSITIVE);for(String s: uaCase){Matcher m = r.matcher(s);if(m.find()){System.out.println(i+" : "+m.group(0));System.out.println("name : "+m.group(1));System.out.println("version : "+m.group(2).replaceAll(ppp,".")+"\n");i++;}}System.out.println("-------------------\n");}}@Testpublic void harmonyTest(){init();String pattern = "linux;.*android.*(harmony).*;\\s*([\\w.,/\\-]+)\\s*[;)]";int i=0;Pattern r = Pattern.compile(pattern,Pattern.CASE_INSENSITIVE);for(String s: uaCase){Matcher m = r.matcher(s);if(m.find()){System.out.println(i+" : "+m.group(0));System.out.println("name : "+m.group(1));System.out.println("version : "+m.group(2)+"\n");i++;}}}@Testpublic void androidTest(){init();String ppp = "[^a-zA-Z0-9 ]";List<String> pList = Arrays.asList("linux;.*(android)\\s([\\w.,/\\-]+)\\s*[;)]","linux;.*(android)\\s*([;)])");for(String rp : pList){int i=0;Pattern r = Pattern.compile(rp,Pattern.CASE_INSENSITIVE);for(String s: uaCase){Matcher m = r.matcher(s);if(m.find()){System.out.println(i+" : "+m.group(0));System.out.println("name : "+m.group(1));System.out.println("version : "+m.group(2).replaceAll(ppp,".")+"\n");i++;}}System.out.println("-------------------\n");}}@Testpublic void deviceTest(){init();String ppp = "[^a-zA-Z0-9 ]";List<String> pList = Arrays.asList(";\\s*([\\w.,/\\- ]+)\\sbuild/","linux;.*android.*harmony.*;\\s*([\\w.,/\\- ]+);.*HuaWeiBrowser","linux;.*android.*harmony.*;\\s*([\\w.,/\\- ]+)[)].*HuaWeiBrowser","linux;.*android.*;\\s*([\\w.,/\\- ]+);.*HuaWeiBrowser","linux;.*android.*;\\s*([\\w.,/\\- ]+)[)].*HuaWeiBrowser",".*android.*;\\s*([\\w.,/\\- ]+); wv[)].*VivoBrowser");for(String rp : pList){int i=0;Pattern r = Pattern.compile(rp,Pattern.CASE_INSENSITIVE);for(String s: uaCase){Matcher m = r.matcher(s);if(m.find()){System.out.println(i+" : "+m.group(0));System.out.println("name : "+m.group(1)+"\n");i++;}}System.out.println("-------------------\n");}}}

然后是systemInfoStrategy的json配置

[{"regularList": ["(ip[honead]+)(?:.*os\\s([\\w.,/\\-]+)\\slike|;\\sopera)","(ip[honead]+).*os\\s([\\w.,/\\-]+)[);]","(ios)\\s([\\w.,/\\-]+)\\s*[);]"],"funList": [{"name": "osType","value": "ios"},{"name": "osVersion","r1": "[^a-zA-Z0-9 ]","r2": "."}]},{"regularList": ["(ios|ip[honead]+)\\s*([;)])"],"funList": [{"name": "osType","value": "ios"},{"name": "osVersion","value": "unknown"}]},{"regularList": ["linux;.*android.*(harmony).*;\\s*([\\w.,/\\-]+)\\s*[;)]"],"funList": [{"name": "osType","value": "HarmonyOS"},{"name": "osVersion","valueFlag": true}]},{"regularList": ["linux;.*(android)\\s([\\w.,/\\-]+)\\s*[;)]"],"funList": [{"name": "osType","value": "android"},{"name": "osVersion","r1": "[^a-zA-Z0-9 ]","r2": "."}]},{"regularList": ["linux;.*(android)\\s*([;)])"],"funList": [{"name": "osType","value": "android"},{"name": "osVersion","value": "unknown"}]}
]

公司内已经上线使用，目前没发现问题

有发现任何问题可以在评论区提出，感激不尽

————————————————————————————————————————
上线跑了一个半月，只有ios出了点问题，
我没想到ip相同的用户会这么这么多。
对android系统的匹配到是影响不大
ios就比较夸张，人超多，而在匹配时，ios只精确到大版本
同一ip下多个ios用户容易被视为一个，比如这里，我

两个小版本不同的ios被匹配成了同一个ios15，大量的非广告ios用户被错误匹配（占比70%）
这也是ios的数据异常的好，排查后才发现的
同一ip下同一个ios很多。
后来老板拉人聊了下，觉得ios就不要ip-ua了，数据水分太差