Linux上用于Json数据处理并导入Mysql的几个有用Python脚本
2024-05-27 18:41:56
1、环境目录结构
[root@localhost python_dir]# pwd
/root/python_dir
[root@localhost python_dir]# ls -lGR
.:
total 5148
-rw-r--r--. 1 root 604 Dec 17 11:24 db.py
-rw-r--r--. 1 root 886 Dec 17 11:54 db.pyc
-rw-r--r--. 1 root 137 Dec 18 03:26 dirfilefilter_print.py
-rwxr-xr-x. 1 root 1123 Dec 18 06:54 json2mysql_python_arrayastext.py
-rwxr-xr-x. 1 root 925 Dec 18 03:39 json2mysql_python_dirfilefilter.py
-rwxrwxrwx. 1 root 939 Dec 18 02:21 json2mysql_python_recordasarray.py
-rw-r--r--. 1 root 2100077 Dec 18 01:54 json_array.json
-rw-r--r--. 1 root 1998167 Dec 18 02:01 json_array.txt
-rw-r--r--. 1 root 5150 Dec 18 06:05 test1.txt
-rw-r--r--. 1 root 7708 Dec 18 02:39 test2.txt
-rw-r--r--. 1 root 7708 Dec 17 11:11 test3.txt
-rw-r--r--. 1 root 1120389 Dec 17 13:22 test_big.json
2、MySQL-python的安装
安装过程：
[root@Master ~]# yum install MySQL-python
Loaded plugins: refresh-packagekit, security
base | 3.7 kB 00:00
epel | 4.3 kB 00:00
epel/primary_db | 5.7 MB 00:05
extras | 3.4 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 3.2 MB 00:03
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6 will be installed
--> Finished Dependency ResolutionDependencies Resolved==============================================================================================================================================================================================================================================================================Package Arch Version Repository Size
==============================================================================================================================================================================================================================================================================
Installing:MySQL-python x86_64 1.2.3-0.3.c1.1.el6 base 86 kTransaction Summary
==============================================================================================================================================================================================================================================================================
Install 1 Package(s)Total download size: 86 k
Installed size: 246 k
Is this ok [y/N]: y
Downloading Packages:
MySQL-python-1.2.3-0.3.c1.1.el6.x86_64.rpm | 86 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running TransactionInstalling : MySQL-python-1.2.3-0.3.c1.1.el6.x86_64 1/1
Unable to connect to dbusVerifying : MySQL-python-1.2.3-0.3.c1.1.el6.x86_64 1/1 Installed:MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6 Complete!
测试成功与否：
[root@Master ~]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>
3、数据库连接配置的脚本
[root@localhost python_dir]# cat db.py
import MySQLdbdb_config = {'host': '120.55.189.188','user': 'datahouse','passwd': 'DTHS2016','port': 3306,'db': 'logdata','charset': 'utf8'
}def getDB():try:conn = MySQLdb.connect(host=db_config['host'],user=db_config['user'],passwd=db_config['passwd'],port=db_config['port'],charset=db_config['charset'])conn.autocommit(True)curr = conn.cursor()curr.execute("SET NAMES utf8");curr.execute("USE %s" % db_config['db']);return conn, currexcept MySQLdb.Error,e:print "Mysql Error %d: %s" % (e.args[0], e.args[1])return None, None
4、检索目录下的所有文件并根据需要过滤的脚本
[root@localhost python_dir]# cat dirfilefilter_print.py
import glob, os, datetime
filesRead = r"/root/python_dir/test*.txt"
list = glob.glob(filesRead)
#print list
for f in list: print(f)
5、将Json的一个array作为text插入Mysql表中字段
[root@localhost python_dir]# cat json2mysql_python_arrayastext.py
# -*- encoding:utf-8 -*-
from db import getDB
import json
import warnings
warnings.filterwarnings("ignore")conn,curr = getDB()i=0for x in open('/root/python_dir/test1.txt'):yemao = json.loads(x)if not yemao.has_key('_reason'):if yemao.has_key('id'):id = yemao['id']else:id = '0'if yemao.has_key('browser'):browser = yemao['browser']else:browser = '0'if yemao.has_key('data'): data = yemao['data']if data.has_key('$items'):items = items['$items']else:items = '0'print itemselse:data = '0'print datadata = '"'+str(data)+'"'#items = yemao['items'] url_current = yemao['url_current']time = yemao['time']ip = yemao['ip']sql = "insert into log_yemao_python(id,browser,url_current,time,ip,items) values ('%s', '%s', '%s', '%s', '%s', %s)" % (id,browser,url_current,time,ip,data)print sqlcurr.execute(sql)print ii += 1else:print ii += 1
print icurr.close()
conn.close()print 'yemao_python done'
6、过滤目录下的文件并导入到Mysql数据库
[root@localhost python_dir]# cat json2mysql_python_dirfilefilter.py
# -*- encoding:utf-8 -*-
from db import getDB
import json
import warnings
import glob, os, datetime
warnings.filterwarnings("ignore")conn,curr = getDB()filesRead = r"/root/python_dir/test*.txt"
list = glob.glob(filesRead)
for f in list: print(f) for json_per in open(f):yemao = json.loads(json_per)if not yemao.has_key('_reason'):if yemao.has_key('id'):id = yemao['id']else:id = '0'if yemao.has_key('browser'):browser = yemao['browser']else:browser = '0'items = yemao['url_from']url_current = yemao['url_current']time = yemao['time']ip = yemao['ip']sql = "insert into log_yemao_python(id,browser,url_current,time,ip,items) values ('%s', '%s', '%s', '%s', '%s', '%s')" % (id,browser,url_current,time,ip,items)print sqlcurr.execute(sql)curr.close()
conn.close()print 'yemao_python done'
7、对于Json以数组出现的数据导入Mysql
[root@localhost python_dir]# cat json2mysql_python_recordasarray.py
# -*- encoding:utf-8 -*-
from db import getDB
import json
import warnings
warnings.filterwarnings("ignore")conn,curr = getDB()i=0for json_array in open('/root/python_dir/json_array.txt'):yemao_array = json.loads(json_array)for yemao in yemao_array: if not yemao.has_key('_reason'):if yemao.has_key('id'):id = yemao['id']else:id = '0'if yemao.has_key('browser'):browser = yemao['browser']else:browser = '0'items = yemao['url_from']url_current = yemao['url_current']time = yemao['time']ip = yemao['ip']sql = "insert into log_yemao_python(id,browser,url_current,time,ip,items) values ('%s', '%s', '%s', '%s', '%s', '%s')" % (id,browser,url_current,time,ip,items)print sqlcurr.execute(sql)print ii += 1else:print ii += 1
print icurr.close()
conn.close()print 'yemao_array_python done'
8、所用到的测试数据示例
[root@localhost python_dir]# more json_array.txt
[{"id":0,"time":1447083929,"url_from":"http:\/\/baimao.60kezhan.com\/dorm\/207122","url_current":"http:\/\/baimao.60kezhan.com\/cart\/dormUpdate","url_to":"","options":"","ip":"61.178.103.132, 10.165.98.189","uid":0,"new_visitor":0,"province":"","city":"","site":"","device"
:"","browser":"Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/38.0.2125.122 Safari\/537.36 SE 2.X MetaSr 1.0","phone":"","token":"46d75506e586fca1015bbcc835ea0e3b","dorm":"","order_phone":"","order_dormitory":"","order_amount":0,"or
der_id":0,"uname":"","site_id":0,"address":"","dorm_id":46244,"dormentry_id":0,"tag":"\u8d2d\u7269\u8f66\u64cd\u4f5c","rid":1623,"cart_quantity":1,"response":"{\"item_num\":3,\"item_amount\":10.2,\"items\":[{\"rid\":1567,\"quantity\":2,\"price\":4,\"purchase_price\":4,\
"amount\":8,\"name\":\"\\u5eb7\\u5e08\\u5085 \\u9999\\u8fa3\\u725b\\u8089\\u9762 \\u65b9\\u4fbf\\u9762 108g\\\/\\u6876\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg!
medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg\"},{\"rid\":1623,\"quantity\":1,\"price\":2.2,\"purchase_price\":2.2,\"amount\":2.2,\"name\":\"\\u96ea\\u78a7 330ml\\\/\\u7f50\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r162
3.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1623.jpg!medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1623.jpg\"}]}","paytype":0,"data":{"item_num":3,"item_amount":10.2,"items":[{"rid":1567,"quantity":2,"price":4,"purch
ase_price":4,"amount":8,"name":"\u5eb7\u5e08\u5085 \u9999\u8fa3\u725b\u8089\u9762 \u65b9\u4fbf\u9762 108g\/\u6876","image_small":"http:\/\/image.60kezhan.com\/Food\/r1567.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1567.jpg!medium","image_big":"http:\/\
/image.60kezhan.com\/Food\/r1567.jpg"},{"rid":1623,"quantity":1,"price":2.2,"purchase_price":2.2,"amount":2.2,"name":"\u96ea\u78a7 330ml\/\u7f50","image_small":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg
!medium","image_big":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg"}]},"info":"\u8d2d\u7269\u8f66\u4fe1\u606f","status":1}]
[{"id":0,"time":1447083930,"url_from":"http:\/\/baimao.60kezhan.com\/dorm\/207122","url_current":"http:\/\/baimao.60kezhan.com\/cart\/dormUpdate","url_to":"","options":"","ip":"61.178.103.132, 10.165.98.189","uid":0,"new_visitor":0,"province":"","city":"","site":"","device"
:"","browser":"Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/38.0.2125.122 Safari\/537.36 SE 2.X MetaSr 1.0","phone":"","token":"46d75506e586fca1015bbcc835ea0e3b","dorm":"","order_phone":"","order_dormitory":"","order_amount":0,"or
der_id":0,"uname":"","site_id":0,"address":"","dorm_id":46244,"dormentry_id":0,"tag":"\u8d2d\u7269\u8f66\u64cd\u4f5c","rid":1623,"cart_quantity":2,"response":"{\"item_num\":4,\"item_amount\":12.4,\"items\":[{\"rid\":1567,\"quantity\":2,\"price\":4,\"purchase_price\":4,\
"amount\":8,\"name\":\"\\u5eb7\\u5e08\\u5085 \\u9999\\u8fa3\\u725b\\u8089\\u9762 \\u65b9\\u4fbf\\u9762 108g\\\/\\u6876\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg!
medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1567.jpg\"},{\"rid\":1623,\"quantity\":2,\"price\":2.2,\"purchase_price\":2.2,\"amount\":4.4,\"name\":\"\\u96ea\\u78a7 330ml\\\/\\u7f50\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r162
3.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1623.jpg!medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1623.jpg\"}]}","paytype":0,"data":{"item_num":4,"item_amount":12.4,"items":[{"rid":1567,"quantity":2,"price":4,"purch
ase_price":4,"amount":8,"name":"\u5eb7\u5e08\u5085 \u9999\u8fa3\u725b\u8089\u9762 \u65b9\u4fbf\u9762 108g\/\u6876","image_small":"http:\/\/image.60kezhan.com\/Food\/r1567.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1567.jpg!medium","image_big":"http:\/\
/image.60kezhan.com\/Food\/r1567.jpg"},{"rid":1623,"quantity":2,"price":2.2,"purchase_price":2.2,"amount":4.4,"name":"\u96ea\u78a7 330ml\/\u7f50","image_small":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg
!medium","image_big":"http:\/\/image.60kezhan.com\/Food\/r1623.jpg"}]},"info":"\u8d2d\u7269\u8f66\u4fe1\u606f","status":1}]
[{"id":0,"time":1447083930,"url_from":"http:\/\/baimao.60kezhan.com\/entry\/select","url_current":"http:\/\/baimao.60kezhan.com\/dorm","url_to":"http:\/\/baimao.60kezhan.com\/dorm\/47018","options":"","ip":"14.147.223.224, 10.165.98.189","uid":0,"new_visitor":0,"province":"",
"city":"","site":"","device":"","browser":"Mozilla\/5.0 (iPhone; CPU iPhone OS 9_0_2 like Mac OS X) AppleWebKit\/601.1.46 (KHTML, like Gecko) Mobile\/13A452 MicroMessenger\/6.3.6 NetType\/WIFI Language\/zh_CN","phone":"","token":"2b8021b486cf117377d1ecae194ad0f0","dorm"
:"","order_phone":"","order_dormitory":"","order_amount":0,"order_id":0,"uname":"","site_id":0,"address":"","dorm_id":0,"dormentry_id":0,"tag":"","rid":0,"cart_quantity":0,"response":"","paytype":0}]
......test1,2,3数据内容相同
[root@localhost python_dir]# cat test2.txt
{"id" : 0, "time" : 1450195240, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/2319", "url_to" : "", "options" : "", "ip" : "113.246.113.224, 10.165.98.189", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone8,1 iPhone_OS/9.2 Eleme/5.5.1 ID/5E592B87-5CCB-4B65-81D1-212155B41580; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75", "phone" : "", "token" : "6c5d9cf52c45a586e9e26963db03fbe4", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195242, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/3234", "url_to" : "", "options" : "", "ip" : "39.78.28.155, 10.252.117.252", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 vivo_X710L/msm8974 Android/4.4.2 Display/KVT49L_dev-keys Eleme/5.5.1 ID/e507bbf0-0159-33c5-8c2e-1f44e4c736c2; KERNEL_VERSION:3.4.0-perf API_Level:19 Mozilla/5.0 (Linux; Android 4.4.2; vivo X710L Build/KVT49L) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36", "phone" : "", "token" : "aeeece703af7afe83df6de0da2e502ee", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195244, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/1367", "url_to" : "", "options" : "", "ip" : "58.56.139.202, 10.165.98.189", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone7,2 iPhone_OS/9.2 Eleme/5.5.1 ID/DCEBFAAC-C10B-4547-8477-D1FD0942DE2B; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75", "phone" : "", "token" : "89bf0dcc6df4616f1a1330a027817dab", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195244, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/98", "url_to" : "", "options" : "", "ip" : "223.104.176.215, 10.252.117.252", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone7,2 iPhone_OS/9.1 Eleme/5.5.1 ID/0A981693-FD51-4DC5-9FAF-E0A0D5080398; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13B143", "phone" : "", "token" : "6f144736315bf84a5cfe5c1b2dae7393", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195247, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/1693", "url_to" : "", "options" : "", "ip" : "125.46.187.1, 10.252.117.252", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone5,2 iPhone_OS/9.2 Eleme/5.5.1 ID/F0C92DA4-09C3-4ED4-A2D9-7087FD92AFA0; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75", "phone" : "", "token" : "570bea3fc168419929ee1a13885c5d7f", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195248, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/2336", "url_to" : "", "options" : "", "ip" : "113.246.113.224, 10.165.98.189", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone8,1 iPhone_OS/9.2 Eleme/5.5.1 ID/5E592B87-5CCB-4B65-81D1-212155B41580; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75", "phone" : "", "token" : "6c5d9cf52c45a586e9e26963db03fbe4", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195249, "url_from" : "http://baimao.60kezhan.com/entry/nearby", "url_current" : "http://baimao.60kezhan.com/entry/index/site_id/450", "url_to" : "", "options" : "", "ip" : "117.176.255.144, 10.165.98.189", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Rajax/1 Apple/iPhone8,1 iPhone_OS/9.2 Eleme/5.5.1 ID/8E4436BE-5F7C-4A01-AD97-A3A395D15A2A; IsJailbroken/0 Mozilla/5.0 (iPhone; CPU iPhone OS 9_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13C75", "phone" : "", "token" : "ce1902b7339e42169e949efee15969f7", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 0, "dormentry_id" : 0, "tag" : "", "rid" : 0, "cart_quantity" : 0, "response" : "", "paytype" : 0 }
{"id" : 0, "time" : 1450195252, "url_from" : "http://baimao.60kezhan.com/dorm/30864", "url_current" : "http://baimao.60kezhan.com/cart/dormUpdate", "url_to" : "", "options" : "", "ip" : "49.52.96.214, 10.165.98.189", "uid" : 0, "new_visitor" : 0, "province" : "", "city" : "", "site" : "", "device" : "", "browser" : "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36", "phone" : "", "token" : "8db199682a0b175908be1058263813ef", "dorm" : "", "order_phone" : "", "order_dormitory" : "", "order_amount" : 0, "order_id" : 0, "uname" : "", "site_id" : 0, "address" : "", "dorm_id" : 25721, "dormentry_id" : 0, "tag" : "购物车操作", "rid" : 3026, "cart_quantity" : 3, "response" : "{\"item_num\":3,\"item_amount\":8.7,\"items\":[{\"rid\":3026,\"quantity\":3,\"price\":2.9,\"purchase_price\":2.9,\"amount\":8.7,\"name\":\"PET330ml \\u771f\\u70ab\\u6a31\\u6843\\u5473\\u9e21\\u5c3e\\u9152\\u3010\\u5e02\\u573a\\u4ef76.5\\u5143\\u3011\",\"image_small\":\"http:\\/\\/image.60kezhan.com\\/Food\\/r3026.jpg\",\"image_medium\":\"http:\\/\\/image.60kezhan.com\\/Food\\/r3026.jpg\",\"image_big\":\"http:\\/\\/image.60kezhan.com\\/Food\\/r3026.jpg\"}]}", "paytype" : 0, "data" : { "item_num" : 3, "item_amount" : 8.7, "items" : [ { "rid" : 3026, "quantity" : 3, "price" : 2.9, "purchase_price" : 2.9, "amount" : 8.7, "name" : "PET330ml 真炫樱桃味鸡尾酒【市场价6.5元】", "image_small" : "http://image.60kezhan.com/Food/r3026.jpg", "image_medium" : "http://image.60kezhan.com/Food/r3026.jpg", "image_big" : "http://image.60kezhan.com/Food/r3026.jpg" } ] }, "info" : "购物车信息", "status" : 1 }[root@localhost python_dir]# more test_big.json
{"id":0,"time":1447084557,"url_from":"http:\/\/baimao.60kezhan.com\/dorm\/213402","url_current":"http:\/\/baimao.60kezhan.com\/cart\/dormUpdate","url_to":"","options":"","ip":"121.31.251.1, 10.252.117.252","uid":0,"new_visitor":0,"province":"","city":"","site":"","device":"
","browser":"Mozilla\/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit\/601.1.46 (KHTML, like Gecko) Mobile\/13B143 MicroMessenger\/6.3.6 NetType\/3G+ Language\/zh_CN","phone":"","token":"9eea24a17e7ff7b8fd3a6892a9b852e2","dorm":"","order_phone":"","order_dormi
tory":"","order_amount":0,"order_id":0,"uname":"","site_id":0,"address":"","dorm_id":26897,"dormentry_id":0,"tag":"\u8d2d\u7269\u8f66\u64cd\u4f5c","rid":1357,"cart_quantity":1,"response":"{\"item_num\":25,\"item_amount\":41.4,\"items\":[{\"rid\":79,\"quantity\":10,\"pri
ce\":1.3,\"purchase_price\":1.3,\"amount\":13,\"name\":\"\\u53cb\\u81e3 \\u91d1\\u4e1d\\u8089\\u677e\\u997c \\u968f\\u624b\\u5305 36g\\\/\\u5305\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r79.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.c
om\\\/Food\\\/r79.jpg!medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r79.jpg\"},{\"rid\":492,\"quantity\":5,\"price\":0.9,\"purchase_price\":0.9,\"amount\":4.5,\"name\":\"\\u708e\\u4ead\\u6e14\\u592b \\u53f0\\u6e7e\\u9c7c\\u8c46\\u8150\\uff08\\u70e7\
\u70e4\\u5473\\uff09\\u72ec\\u7acb\\u5305 \\u6e29\\u5dde\\u540d\\u5403 17g\\\/\\u5305\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r492.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r492.jpg!medium\",\"image_big\":\"http:\\\/\
\\/image.60kezhan.com\\\/Food\\\/r492.jpg\"},{\"rid\":2102,\"quantity\":3,\"price\":1,\"purchase_price\":1,\"amount\":3,\"name\":\"\\u53cc\\u6c47 \\u6ce1\\u9762\\u62cd\\u6863\\u9999\\u80a0 30g\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2102.jpg!small\
",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2102.jpg!medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2102.jpg\"},{\"rid\":2130,\"quantity\":5,\"price\":1.3,\"purchase_price\":1.3,\"amount\":6.5,\"name\":\"\\u4e61\\u5df4\\u4f6c \\u
5364\\u86cb 30g\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2130.jpg!small\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2130.jpg!medium\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r2130.jpg\"},{\"rid\":1333,\"quantit
y\":1,\"price\":5.5,\"purchase_price\":5.5,\"amount\":5.5,\"name\":\"\\u5361\\u592b\\u8da3\\u591a\\u591a \\u7ecf\\u5178\\u5de7\\u514b\\u529b\\u5473 95g\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1333.jpg\",\"image_medium\":\"http:\\\/\\\/image.59stor
e.com\\\/Food\\\/r1333.jpg\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1333.jpg\"},{\"rid\":1357,\"quantity\":1,\"price\":8.9,\"purchase_price\":8.9,\"amount\":8.9,\"name\":\"\\u8fbe\\u5229\\u56ed \\u6cd5\\u5f0f\\u8f6f\\u9762\\u5305 \\u9999\\u6a59\\u547
3 360g\\\/\\u888b\",\"image_small\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1357.jpg\",\"image_medium\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1357.jpg\",\"image_big\":\"http:\\\/\\\/image.60kezhan.com\\\/Food\\\/r1357.jpg\"}]}","paytype":0,"data":{"item_num":
25,"item_amount":41.4,"items":[{"rid":79,"quantity":10,"price":1.3,"purchase_price":1.3,"amount":13,"name":"\u53cb\u81e3 \u91d1\u4e1d\u8089\u677e\u997c \u968f\u624b\u5305 36g\/\u5305","image_small":"http:\/\/image.60kezhan.com\/Food\/r79.jpg!small","image_medium":"http:\
/\/image.60kezhan.com\/Food\/r79.jpg!medium","image_big":"http:\/\/image.60kezhan.com\/Food\/r79.jpg"},{"rid":492,"quantity":5,"price":0.9,"purchase_price":0.9,"amount":4.5,"name":"\u708e\u4ead\u6e14\u592b \u53f0\u6e7e\u9c7c\u8c46\u8150\uff08\u70e7\u70e4\u5473\uff09\u72ec
\u7acb\u5305 \u6e29\u5dde\u540d\u5403 17g\/\u5305","image_small":"http:\/\/image.60kezhan.com\/Food\/r492.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r492.jpg!medium","image_big":"http:\/\/image.60kezhan.com\/Food\/r492.jpg"},{"rid":2102,"quantity":3,"pri
ce":1,"purchase_price":1,"amount":3,"name":"\u53cc\u6c47 \u6ce1\u9762\u62cd\u6863\u9999\u80a0 30g","image_small":"http:\/\/image.60kezhan.com\/Food\/r2102.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r2102.jpg!medium","image_big":"http:\/\/image.60kezhan.c
om\/Food\/r2102.jpg"},{"rid":2130,"quantity":5,"price":1.3,"purchase_price":1.3,"amount":6.5,"name":"\u4e61\u5df4\u4f6c \u5364\u86cb 30g","image_small":"http:\/\/image.60kezhan.com\/Food\/r2130.jpg!small","image_medium":"http:\/\/image.60kezhan.com\/Food\/r2130.jpg!medium
","image_big":"http:\/\/image.60kezhan.com\/Food\/r2130.jpg"},{"rid":1333,"quantity":1,"price":5.5,"purchase_price":5.5,"amount":5.5,"name":"\u5361\u592b\u8da3\u591a\u591a \u7ecf\u5178\u5de7\u514b\u529b\u5473 95g","image_small":"http:\/\/image.60kezhan.com\/Food\/r1333.jp
g","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1333.jpg","image_big":"http:\/\/image.60kezhan.com\/Food\/r1333.jpg"},{"rid":1357,"quantity":1,"price":8.9,"purchase_price":8.9,"amount":8.9,"name":"\u8fbe\u5229\u56ed \u6cd5\u5f0f\u8f6f\u9762\u5305 \u9999\u6a59\u5473360g\/\u888b","image_small":"http:\/\/image.60kezhan.com\/Food\/r1357.jpg","image_medium":"http:\/\/image.60kezhan.com\/Food\/r1357.jpg","image_big":"http:\/\/image.60kezhan.com\/Food\/r1357.jpg"}]},"info":"\u8d2d\u7269\u8f66\u4fe1\u606f","status":1}9、其他说明
Python环境一般Linux系统默认安装，解析JSON的组件默认也有；但连接MySQL数据库的组件要自行安装。
Python脚本写好以后，在linux环境下，按“[root@localhost python_dir]# python dirfilefilter_print.py”方式调用执行。
同事写的JSON与数据库交互并进行解析的代码，供参考：
1_box_item.py
# -*- coding:utf-8 -*-
# adjust dormitem every day
from db import getDB
import json
import warnings
warnings.filterwarnings("ignore")conn, curr = getDB()i = 0sql = "SELECT max(order_id) FROM h_b_order_item"
curr.execute(sql)
max_order_ids = curr.fetchall()
for item in max_order_ids:max_order_id = item[0]sql = "SELECT order_id,detail_json FROM h_b_order where order_id > %d" % (max_order_id)
curr.execute(sql)
box_item_list = curr.fetchall()
for box_item in box_item_list:order_id = box_item[0]box_item1 = box_item[1]items = json.loads(box_item1)for x in items:rid = x['rid']quantity = x['quantity']price = x['price']amount = x['amount']sql = "insert into h_b_order_item(order_id,rid,quantity,price,amount) values (%d,%d,%d,%.2f,%.2f)" % (order_id,rid,quantity,price,amount)print sqlcurr.execute(sql)i += 1curr.close()
conn.close()print 'done'
print 'total rows:' + str(i)
Linux上用于Json数据处理并导入Mysql的几个有用Python脚本相关推荐
- Swift - SwiftyJSON的使用详解（附样例，用于JSON数据处理）
转自:http://www.hangge.com/blog/cache/detail_968.html Swift - SwiftyJSON的使用详解(附样例,用于JSON数据处理) 2016-01- ...
- Linux上搭建java环境（Tomact mysql jdk）
Linux简介 Linux是一套免费使用和自由传播的类Unix操作系统,是一个基于POSIX和Unix的多用户.多任务.支持多线程和多CPU的操作系统.它能运行主要的Unix工具软件.应用程序和网络协 ...
- linux上离线安装mysql_Linux下安装mysql（离线安装和在线安装）
一:在线安装mysql 1.首先检测一下,mysql之前有没有被安装 命令:rpm -qa | grep mysql 2.删除mysql的命令: rpm -e --nodeps `rpm -qa | ...
- mysql导入亿级数据_如何将上亿条大容量数据导入MYSQL
我有一份数据量达到1,0015,0806条的csv文件,现在需要导入Mysql中. 在我电脑中安装的版本是MYSQL8.0.12,常见的管理工具有: 已知的有三种办法: 1.用图形管理工具Navica ...
- Linux服务器CPU、内存、磁盘空间、负载情况查看python脚本
[本文出自天外归云的博客园] 网上搜,东拼西凑,组装了一个可以查Linux服务器CPU使用率.内存使用率.磁盘空间占用率.负载情况的python脚本. 脚本内容如下: # -*- coding:utf ...
- 在Linux上安装java、tomcat、mysql
连接服务器软件 Xshell/ssh:远程控制服务器 filezilla/xftp:管理服务器里面的文件的工具 连接服务器 IP.端口号(22).账号.密码 工具使用: 1.使用 Xshell工具,连 ...
- [绍棠_Swift] SwiftyJSON的使用详解（附样例，用于JSON数据处理）
1,SwiftyJSON介绍与配置 SwiftyJSON是个使用Swift语言编写的开源库,可以让我们很方便地处理JSON数据(解析数据.生成数据). GitHub地址:https://github. ...
- Linux 上如何让任意普通用户执行拥有root权限的特定脚本或者程序
问题描述: 有一个脚本,只能是具有root执行权限的用户来执行.这意味着只有root用户和拥有sudo权限的用户才能执行,但我们的需求是让普通用户在没有取得root权限的情况下也能执行,那么就开发另一 ...
- python 数字证书模拟登录_用于生成WebService使用的数字证书及签署证书.python脚本...
#! /bin/env python # -*- encoding:gbk -*- """ 用于生成 WebService 使用的 CA 及并且签署证书 "&q ...
- 如何在Red Hat Enterprise Linux上安装Python 3
如何在Red Hat Enterprise Linux上安装Python 3. 本文介绍如何安装Python 3, ,pip,venv,virtualenv和pipenv在企业Linux 7.按照本文 ...
最新文章
- jsf登录注册页面_您将在下一个项目中使用JSF吗？
- 存储过程—导出table数据为inser sqlt语句
- power designer 使用vba实现自动化操作学习
- java 中对对象的调用
- 咨询报告生成：使用python生成pptx格式的报告
- Android：Activity统一堆栈管理（实现随时finish特定或是所有Activty）
- mysql truncate很慢_mysql truncate 的问题
- mysql carnation_RDS mysql5.6 数据库还原到本地
- vim粘贴代码格式变乱
- 卡尔曼滤波-卡尔曼滤波全篇讲解
- 教你如何看headers
- 在win服务器查看系统报错日志文件,win2003查看及管理系统日志的方法
- 计算机公共基础知识论文,公共基础知识学术论文.docx
- 同时支持手机号、用户名、邮箱登录
- 小时候 觉得爸爸就是天 无所不能～
- BT源代码学习心得(九)：客户端源代码分析(图形界面浅析) -- 转贴自 wolfenstein (NeverSayNever)
- IOS 自动订阅以及恢复购买
- CSS3回炉计划-编码技巧
- 基于51单片机的自动红外感应洗手器proteus仿真程序设计
- 想想都脊背一凉，细思极恐，大家提早醒悟