由frankmocap得到的.pkl文件转为.bvh或者.fbx

需求

由Frankmocap所得到的.pkl文件转为blender里的.bvh或者Maya里的.fbx
Frankmocap_github项目地址
2D转3D转.bvh可以看VideoTo3dPoseAndBvh，.bvh转3D相对来说好转一点毕竟.bvh中的关节点与关节点组成的向量有这6自由度信息，但是3D点只有3自由度要转.bvh就要限制自由度。

问题

1.到2022年5月9号为止项目原作者还没有更新这一功能，即提供一个blender的addon;
2.第三方作者魔改了很多相关的插件，可以为EasyMocap，Vibe，Frankmocap等提供blender接口；然而很多可能满足smpl需求却不满足smplx需求；
3.blender里的python环境，package的安装相关教程较少，官方文档又不想看。

相关 2

第二个忘记是哪找的了，是一个python文件fbx_output_FRANKMOCAP.py，这个程序里面导入模块import bpy，很明显是要在blender里面运行的；blender用python是真的不友好，要去看官方手册，里面的python控制台我不知道有什么用。。。。而且不同版本的blender也不一样会报很多奇怪的错误。

环境

ubuntu + blender3.1.2稳定版；

下载文件

需要一个SMPL_unity_v.1.0.0.zip文件（可以去SMPL作者官网那里下载），解压后放到VIBE项目下，把fbx_output_FRANKMOCAP.py插件放到VIBE项目下的lib/utils下面，原项目的环境不用配，只需要配置插件里的环境就好了，配置方法如下；

安装package

只要把要的包放到blender目录下的site-packages里就好了，我的操作是自己建一个和blender的python版本一样的conda虚拟环境，然后进到环境里用pip install packages -t directory命令装到blender目录下的site-packages里；

运行

运行脚本或者.py文件可以输入以下命令：./blender --python ~/Desktop/VIBE_pkl2fbx/lib/utils/fbx_output_FRANKMOCAP.py，原文件是有argparse命令行参数的，我不知道通过blender环境怎么输入，就直接改了原代码，把要输入的参数都直接修改好了，如果有人要用代码的话得自己改一下里面的路径和参数。最终可以在blender里导出.bvh或者.fbx文件。
Blender Python API中文介绍

blender的用法（ubuntu）

1.可以直接命令行下载安装，也可以去官网下载压缩包解压，我是解压的；
2.Blender左下角可以调出时间线功能，一般.bvh和.fbx都是一段骨骼或者蒙皮的动画，时间线可以播放动作；
3.找到blender安装目录下的bin下面的python解释器，用ubuntu终端输入命令行./python3.10可以进到blender解释器里，虽然好像没什么用；
4.blender软件里的python控制台是已经进到他那个python环境里了，在这里也运行不了啥程序，除了bpy模块下的函数我不知有什么其他作用。这时候有两种方法：第一种就是用scripting去打开.py文件再运行；第二种就是使用Ubuntu的终端，首先来到blender安装目录，运行./blender就可以打开这个软件（双击图标也可以），运行脚本或者.py文件可以输入以下命令：./blender --python directory.py。
5.blender肯定可以通过IDE或者其他API来使用python，但是作者只想用一个工具就没有深入。
6.所有效果图其实是一段会动的视频和frankmocap对应的输入视频是一样的动作，缺点是在这个插件里手部点没有输出。

修改过后的代码

# -*- coding: utf-8 -*-# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
#
# Author: Joachim Tesch, Max Planck Institute for Intelligent Systems, Perceiving Systems
#
# Create keyframed animated skinned SMPL mesh from .pkl pose description
#
# Generated mesh will be exported in FBX or glTF format
#
# Notes:
#  + Male and female gender models only
#  + Script can be run from command line or in Blender Editor (Text Editor>Run Script)
#  + Command line: Install mathutils module in your bpy virtualenv with 'pip install mathutils==2.81.2'import os
import sys
import bpy
import time
import joblib
import argparse
import numpy as np
import addon_utils
from math import radians
from mathutils import Matrix, Vector, Quaternion, Euler
import pickle
import os.path
import pandas as pd# Globals
male_model_path = '/home/dms/Desktop/VIBE_pkl2fbx/data/SMPL_unity_v.1.0.0/smpl/Models/SMPL_m_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx'
female_model_path = '/home/dms/Desktop/VIBE_pkl2fbx/data/SMPL_unity_v.1.0.0/smpl/Models/SMPL_f_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx'fps_source = 30
fps_target = 30gender = 'male'start_origin = 1bone_name_from_index = {0 : 'Pelvis',1 : 'L_Hip',2 : 'R_Hip',3 : 'Spine1',4 : 'L_Knee',5 : 'R_Knee',6 : 'Spine2',7 : 'L_Ankle',8: 'R_Ankle',9: 'Spine3',10: 'L_Foot',11: 'R_Foot',12: 'Neck',13: 'L_Collar',14: 'R_Collar',15: 'Head',16: 'L_Shoulder',17: 'R_Shoulder',18: 'L_Elbow',19: 'R_Elbow',20: 'L_Wrist',21: 'R_Wrist',22: 'L_Hand',23: 'R_Hand'
}# Helper functions# Computes rotation matrix through Rodrigues formula as in cv2.Rodrigues
# Source: smpl/plugins/blender/corrective_bpy_sh.py
def Rodrigues(rotvec):theta = np.linalg.norm(rotvec)r = (rotvec/theta).reshape(3, 1) if theta > 0. else rotveccost = np.cos(theta)mat = np.asarray([[0, -r[2], r[1]],[r[2], 0, -r[0]],[-r[1], r[0], 0]])return(cost*np.eye(3) + (1-cost)*r.dot(r.T) + np.sin(theta)*mat)# Setup scene
def setup_scene(model_path, fps_target):scene = bpy.data.scenes['Scene']############################ Engine independent setup###########################scene.render.fps = fps_target# Remove default cubeif 'Cube' in bpy.data.objects:bpy.data.objects['Cube'].select_set(True)bpy.ops.object.delete()# Import gender specific .fbx template filebpy.ops.import_scene.fbx(filepath=model_path)# Process single pose into keyframed bone orientations
def process_pose(current_frame, pose, trans, pelvis_position):if pose.shape[0] == 72:rod_rots = pose.reshape(24, 3)else:rod_rots = pose.reshape(26, 3)mat_rots = [Rodrigues(rod_rot) for rod_rot in rod_rots]# Set the location of the Pelvis bone to the translation parameterarmature = bpy.data.objects['Armature']bones = armature.pose.bones# Pelvis: X-Right, Y-Up, Z-Forward (Blender -Y)# Set absolute pelvis location relative to Pelvis bone headbones[bone_name_from_index[0]].location = Vector((100*trans[1], 100*trans[2], 100*trans[0])) - pelvis_position# bones['Root'].location = Vector(trans)bones[bone_name_from_index[0]].keyframe_insert('location', frame=current_frame)for index, mat_rot in enumerate(mat_rots, 0):if index >= 24:continuebone = bones[bone_name_from_index[index]]bone_rotation = Matrix(mat_rot).to_quaternion()quat_x_90_cw = Quaternion((1.0, 0.0, 0.0), radians(-90))quat_z_90_cw = Quaternion((0.0, 0.0, 1.0), radians(-90))if index == 0:# Rotate pelvis so that avatar stands upright and looks along negative Y avisbone.rotation_quaternion = (quat_x_90_cw @ quat_z_90_cw) @ bone_rotationelse:bone.rotation_quaternion = bone_rotationbone.keyframe_insert('rotation_quaternion', frame=current_frame)return# Process all the poses from the pose file
def process_poses(input_path,gender,fps_source,fps_target,start_origin,person_id=1,
):print('Processing: ' + input_path)# comeco da alteracao que fizpkl_path = '/home/dms/Desktop/VIBE_pkl2fbx/data/mocap/'list_dir = os.listdir(pkl_path)s_list = sorted(list_dir)append_data = (pd.DataFrame(pd.DataFrame.from_dict(pickle.load(open(pkl_path+s_list[0],'rb'))).pred_output_list[0]['pred_body_pose']))for i in s_list:if i == s_list[0]:print('0')nextelse:print(i)frank_data = pickle.load(open(pkl_path+i,'rb'))frank = pd.DataFrame.from_dict(frank_data)if 'pred_body_pose' not in frank.columns:nextfrank_pose = pd.DataFrame(frank.pred_output_list[0]['pred_body_pose'])append_data = append_data.append(frank_pose)# print(append_data)poses = append_data.to_numpy()########## Fim da alteracao que fiz# data = joblib.load(input_path)# poses = data[person_id]['pose']trans = np.zeros((poses.shape[0], 3))if gender == 'female':model_path = female_model_pathfor k,v in bone_name_from_index.items():bone_name_from_index[k] = 'f_avg_' + velif gender == 'male':model_path = male_model_pathfor k,v in bone_name_from_index.items():bone_name_from_index[k] = 'm_avg_' + velse:print('ERROR: Unsupported gender: ' + gender)sys.exit(1)# Limit target fps to source fpsif fps_target > fps_source:fps_target = fps_sourceprint(f'Gender: {gender}')print(f'Number of source poses: {str(poses.shape[0])}')print(f'Source frames-per-second: {str(fps_source)}')print(f'Target frames-per-second: {str(fps_target)}')print('--------------------------------------------------')setup_scene(model_path, fps_target)scene = bpy.data.scenes['Scene']sample_rate = int(fps_source/fps_target)scene.frame_end = (int)(poses.shape[0]/sample_rate)# Retrieve pelvis world position.# Unit is [cm] due to Armature scaling.# Need to make copy since reference will change when bone location is modified.bpy.ops.object.mode_set(mode='EDIT')pelvis_position = Vector(bpy.data.armatures[0].edit_bones[bone_name_from_index[0]].head)bpy.ops.object.mode_set(mode='OBJECT')source_index = 0frame = 1offset = np.array([0.0, 0.0, 0.0])while source_index < poses.shape[0]:print('Adding pose: ' + str(source_index))if start_origin:if source_index == 0:offset = np.array([trans[source_index][0], trans[source_index][1], 0])# Go to new framescene.frame_set(frame)process_pose(frame, poses[source_index], (trans[source_index] - offset), pelvis_position)source_index += sample_rateframe += 1return framedef export_animated_mesh(output_path):# Create output directory if neededoutput_dir = os.path.dirname(output_path)if not os.path.isdir(output_dir):os.makedirs(output_dir, exist_ok=True)# Select only skinned mesh and rigbpy.ops.object.select_all(action='DESELECT')bpy.data.objects['Armature'].select_set(True)bpy.data.objects['Armature'].children[0].select_set(True)if output_path.endswith('.glb'):print('Exporting to glTF binary (.glb)')# Currently exporting without shape/pose shapes for smaller file sizesbpy.ops.export_scene.gltf(filepath=output_path, export_format='GLB', export_selected=True, export_morph=False)elif output_path.endswith('.fbx'):print('Exporting to FBX binary (.fbx)')bpy.ops.export_scene.fbx(filepath=output_path, use_selection=True, add_leaf_bones=False)else:print('ERROR: Unsupported export format: ' + output_path)sys.exit(1)returnif __name__ == '__main__':try:if bpy.app.background:parser = argparse.ArgumentParser(description='Create keyframed animated skinned SMPL mesh from VIBE output')parser.add_argument('--input', dest='input_path', type=str, required=True,help='Input file or directory')parser.add_argument('--output', dest='output_path', type=str, required=True,help='Output file or directory')parser.add_argument('--fps_source', type=int, default=fps_source,help='Source framerate')parser.add_argument('--fps_target', type=int, default=fps_target,help='Target framerate')parser.add_argument('--gender', type=str, default=gender,help='Always use specified gender')parser.add_argument('--start_origin', type=int, default=start_origin,help='Start animation centered above origin')parser.add_argument('--person_id', type=int, default=1,help='Detected person ID to use for fbx animation')args = parser.parse_args()input_path = args.input_pathoutput_path = args.output_pathif not os.path.exists(input_path):print('ERROR: Invalid input path')sys.exit(1)fps_source = 30fps_target = 30gender = 'male'start_origin = 1input_path = '~/Desktop/VIBE_pkl2fbx/data/mocap'output_path = '~/Desktop/VIBE_pkl2fbx/data/output/output.fbx'# end if bpy.app.backgroundstartTime = time.perf_counter()# Process datacwd = os.getcwd()# Turn relative input/output paths into absolute paths# if not input_path.startswith(os.path.sep):#     input_path = os.path.join(cwd, input_path)## if not output_path.startswith(os.path.sep):#     output_path = os.path.join(cwd, output_path)print('Input path: ' + input_path)print('Output path: ' + output_path)if not (output_path.endswith('.fbx') or output_path.endswith('.glb')):print('ERROR: Invalid output format (must be .fbx or .glb)')sys.exit(1)# # Process pose file# if input_path.endswith('.pkl'):#     if not os.path.isfile(input_path):#         print('ERROR: Invalid input file')#         sys.exit(1)##     poses_processed = process_poses(#         input_path=input_path,#         gender='male',#         fps_source=30,#         fps_target=30,#         start_origin=1,#         # person_id=args.person_id,#         person_id = 0#     )#     export_animated_mesh(output_path)poses_processed = process_poses(input_path=input_path,gender='male',fps_source=30,fps_target=30,start_origin=1,# person_id=args.person_id,person_id=0)export_animated_mesh(output_path)print('--------------------------------------------------')print('Animation export finished.')print(f'Poses processed: {str(poses_processed)}')print(f'Processing time : {time.perf_counter() - startTime:.2f} s')print('--------------------------------------------------')sys.exit(0)except SystemExit as ex:if ex.code is None:exit_status = 0else:exit_status = ex.codeprint('Exiting. Exit status: ' + str(exit_status))# Only exit to OS when we are not running in Blender GUIif bpy.app.background:sys.exit(exit_status)

相关 3

同是Facebook研究院出品的fairmotion项目里主要提供smpl和bvh格式相互之间的转化和其他很多tasks，但是项目文档说明较少，几乎全是接口函数，如果想对应自己的需求的话需要自己去看里面函数的定义。

环境

这个环境十分好配，采取以下两种方法都可以，只不过由于human_body_prior库有改动，会出现一些小问题，问题出来再改就行了。

运行

代码路径fairmotion/fairmotion/data/frankmocap.py 下的原代码运行后会报ValueError: could not broadcast input array from shape (8,3,3) into shape (1,3,3)问题我没有解决，用的是另外一个代码，而且需要对frankmocap得到的很多pkl进行一个处理合并为一个pkl。处理好之后就可以运行frankmocap.py文件从smpl数据格式转到bvh格式了。

数据预处理代码：

# frankdata_preprocess.py
import os
import pickle
import os.path as opprocessed_data = []
root = r'./frankmocap_output/mocap'
for name in sorted(os.listdir(op.join(root))):file_path = op.join(root, name)preprocess_data = pickle.load(open(file_path, 'rb'))motion_key = list(preprocess_data.keys())[6]motion_data = preprocess_data[motion_key]  # 找到'pred_output_list'键对应的值，[{}]processed_data.append(motion_data)  # [[{}], [{}], [{}]]pickle.dump(processed_data, open(r'./frankmocap_output/processed_data.pkl', 'wb'))

frankmocap2bvh代码：

# Copyright (c) Facebook, Inc. and its affiliates.
# fairmotion/fairmotion/data/frankmocap.py
import numpy as np
import pickle
import torchfrom fairmotion.data import amass, bvh
from fairmotion.core import motion as motion_classes
from fairmotion.utils import constants, utils
from fairmotion.ops import conversions, motion as motion_opsdef get_smpl_base_position(bm, betas):pose_body_zeros = torch.zeros((1, 3 * (22 - 1)))body = bm(pose_body=pose_body_zeros, betas=betas)base_position = body.Jtr.detach().numpy()[0, 0:22]return base_positiondef compute_im2sim_scale(joints_img,base_position,
):left_leg_sim = np.linalg.norm(base_position[amass.joint_names.index("lknee")] - base_position[amass.joint_names.index("lankle")])# indices from from frankmocap.bodymocap.constantsleft_leg_img = np.linalg.norm(joints_img[29][:2] - joints_img[30][:2])right_leg_sim = np.linalg.norm(base_position[amass.joint_names.index("rknee")] - base_position[amass.joint_names.index("rankle")])right_leg_img = np.linalg.norm(joints_img[25][:2] - joints_img[26][:2])return (left_leg_sim + right_leg_sim) / (left_leg_img + right_leg_img)def load(file,motion=None,bm_path=None,motion_key=None,estimate_root=False,scale=1.0,load_skel=True,load_motion=True,v_up_skel=np.array([0.0, 1.0, 0.0]),v_face_skel=np.array([0.0, 0.0, 1.0]),v_up_env=np.array([0.0, 1.0, 0.0]),
):processed_data = pickle.load(open(file, "rb"))# if motion_key is None:#     motion_key = list(all_data.keys())[6]   # 找到'pred_output_list'键对应的值motion_data = processed_databm = amass.load_body_model(bm_path)betas = torch.Tensor(np.array(motion_data[0][0]["pred_betas"])[:]).to("cpu")# img_shape = motion_data[0]["pred_output_list"][0]["img_shape"]num_joints = len(amass.joint_names)skel = amass.create_skeleton_from_amass_bodymodel(bm, betas, len(amass.joint_names), amass.joint_names)joint_names = [j.name for j in skel.joints]num_frames = len(motion_data)print(num_frames)T = np.random.rand(num_frames, num_joints, 4, 4)T[:] = constants.EYE_T# Use lowest point of right/left ankle from first image frame as referenceref_root_y = np.min((motion_data[0][0]["pred_joints_img"][25][1],motion_data[0][0]["pred_joints_img"][30][1]))for i in range(num_frames):for j in range(num_joints):T[i][joint_names.index(amass.joint_names[j])] = conversions.R2T(np.array(motion_data[i][0]["pred_rotmat"][0])[j])if estimate_root:R_root = conversions.T2R(T[i][0])p_root = np.zeros(3)base_position = get_smpl_base_position(bm, betas)# compute scale as ratio of limb length in img and bmim2sim_scale = compute_im2sim_scale(motion_data[i][0]["pred_joints_img"],base_position,)p_root[0] = np.mean((motion_data[i][0]["pred_joints_img"][27][0],motion_data[i][0]["pred_joints_img"][28][0])) * im2sim_scaleroot_y = np.mean((motion_data[i][0]["pred_joints_img"][27][1],motion_data[i][0]["pred_joints_img"][28][1]))p_root[2] = (ref_root_y - root_y) * im2sim_scale# p_root[1] = np.max((#     np.linalg.norm(T[i][amass.joint_names.index("root")] - T[i][amass.joint_names.index("lankle")]),#     np.linalg.norm(T[i][amass.joint_names.index("root")] - T[i][amass.joint_names.index("rankle")]),# ))# print(p_root[1])T[i][0] = conversions.Rp2T(R_root, p_root)motion = motion_classes.Motion.from_matrix(T, skel)motion.set_fps(60)motion = motion_ops.rotate(motion,conversions.Ax2R(conversions.deg2rad(-90)),)# post process to ensure character stays above floorpositions = motion.positions(local=False)for i in range(motion.num_frames()):ltoe = positions[i][amass.joint_names.index("ltoe")][2]rtoe = positions[i][amass.joint_names.index("rtoe")][2]offset = min(ltoe, rtoe)if offset < 0.05:# print(offset)R, p = conversions.T2Rp(T[i][0])p[2] += 0.05 - offsetT[i][0] = conversions.Rp2T(R, p)motion = motion_classes.Motion.from_matrix(T, skel)motion = motion_ops.rotate(motion,conversions.Ax2R(conversions.deg2rad(-90)),)return motionif __name__ == '__main__':file = r'../../daoboke/frankmocap_output/processed_data.pkl'motion = load(file=file, bm_path=r'../../body_models/smplh/neutral/model.npz')bvh.save(motion, filename=r'../../daoboke/frankmocap_output/output.bvh')

如果想要简单的从一般的smpl格式数据转到bvh的话，可以在fairmotion项目中运行以下代码，然而在fairmotion中我未能成功完成bvh2smpl代码：

# smpl2bvh.py
import torch
from human_body_prior.body_model.body_model import BodyModel
from fairmotion.data import amass, bvhdef load_body_model(bm_path, num_betas=10, model_type="smplh"):comp_device = torch.device("cpu")bm = BodyModel(bm_path=bm_path,num_betas=num_betas,model_type=model_type).to(comp_device)return bm# Load file
file = './smplData/DanceDB/20120731_StefanosTheodorou/Stefanos_1os_antrikos_karsilamas_C3D_poses.npz'
bm_path = '../body_models/smplh/{}/model.npz'.format('male')
num_betas = 10   # body parameters
model_type = 'smplh'bm = load_body_model(bm_path, num_betas, model_type)
motion = amass.load(file, bm, bm_path)
bvh.save(motion, filename='./smplData/output.bvh')