Unreal 4 C++开发，获取序列器上物体播放的自身音轨上的语音

首先说明，这是一个非常hack的做法，非常有可能因引擎版本的升级等原因带来不稳定性，并且项目打包后可能无法得到预期结果，但在编辑器中播放是可以的。读者应把本文的重点放在解决这个问题的过程以及一些开发中用到的具体技术。我做这件事情的原因是为了获得角色播放的语音并实现一些语音可视化的功能。本文内容基于UE 4.14版本。

1、找到语音在代码中的位置

使语音播放的地方是我通过给源码加断点找到的。应用播放声音，在底层都需要引擎帮其拿到Audio Device，所以在LevelActor.cpp文件第1322行加上断点，然后调试带有序列器的应用，在断点处程序暂停，通过调用堆栈就可以看到程序是怎样一步步到达底层声音播放的地方。此调用堆栈如下所示：

> UE4Editor-Engine.dll!UWorld::GetAudioDevice() 行 1322    C++UE4Editor-MovieSceneTracks.dll!FMovieSceneAudioTrackInstance::Update(EMovieSceneUpdateData & UpdateData, const TArray<TWeakObjectPtr<UObject,FWeakObjectPtr>,FDefaultAllocator> & RuntimeObjects, IMovieScenePlayer & Player, FMovieSceneSequenceInstance & SequenceInstance) 行 145    C++UE4Editor-MovieScene.dll!FMovieSceneSequenceInstance::UpdatePassSingle(EMovieSceneUpdateData & UpdateData, IMovieScenePlayer & Player) 行 255   C++UE4Editor-MovieScene.dll!FMovieSceneSequenceInstance::Update(EMovieSceneUpdateData & UpdateData, IMovieScenePlayer & Player) 行 184 C++UE4Editor-LevelSequence.dll!ULevelSequencePlayer::UpdateMovieSceneInstance(float CurrentPosition, float PreviousPosition) 行 619    C++UE4Editor-Engine.dll!AActor::TickActor(float DeltaSeconds, ELevelTick TickType, FActorTickFunction & ThisTickFunction) 行 834   C++UE4Editor-Engine.dll!FActorTickFunction::ExecuteTick(float DeltaTime, ELevelTick TickType, ENamedThreads::Type CurrentThread, const TRefCountPtr<FGraphEvent> & MyCompletionGraphEvent) 行 113    C++UE4Editor-Engine.dll!FTickFunctionTask::DoTask(ENamedThreads::Type CurrentThread, const TRefCountPtr<FGraphEvent> & MyCompletionGraphEvent) 行 256    C++UE4Editor-Engine.dll!TGraphTask<FTickFunctionTask>::ExecuteTask(TArray<FBaseGraphTask *,FDefaultAllocator> & NewTasks, ENamedThreads::Type CurrentThread) 行 868    C++UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksNamedThread(int QueueIndex, bool bAllowStall) 行 932   C++UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksUntilQuit(int QueueIndex) 行 679   C++UE4Editor-Core.dll!FTaskGraphImplementation::WaitUntilTasksComplete(const TArray<TRefCountPtr<FGraphEvent>,TInlineAllocator<4,FDefaultAllocator> > & Tasks, ENamedThreads::Type CurrentThreadIfKnown) 行 1776 C++UE4Editor-Engine.dll!FTickTaskSequencer::ReleaseTickGroup(ETickingGroup WorldTickGroup, bool bBlockTillComplete) 行 525 C++UE4Editor-Engine.dll!FTickTaskManager::RunTickGroup(ETickingGroup Group, bool bBlockTillComplete) 行 1437   C++UE4Editor-Engine.dll!UWorld::RunTickGroup(ETickingGroup Group, bool bBlockTillComplete) 行 730  C++UE4Editor-Engine.dll!UWorld::Tick(ELevelTick TickType, float DeltaSeconds) 行 1340  C++UE4Editor-UnrealEd.dll!UEditorEngine::Tick(float DeltaSeconds, bool bIdleMode) 行 1422  C++UE4Editor-UnrealEd.dll!UUnrealEdEngine::Tick(float DeltaSeconds, bool bIdleMode) 行 371 C++UE4Editor.exe!FEngineLoop::Tick() 行 2859   C++

从调用堆栈可以看出，序列器上和语音直接相关的类就是FMovieSceneAudioTrackInstance。从源码上看其主要成员如下：

private:/** Plays the sound of the given section at the given time */void PlaySound(class UMovieSceneAudioSection* AudioSection, TWeakObjectPtr<UAudioComponent> Component, float Time);/** Stops all components for the given row index */void StopSound(int32 RowIndex);/** Stops all components from playing */void StopAllSounds();/** Gets the audio component component for the actor and row index, creating it if necessary */TWeakObjectPtr<UAudioComponent> GetAudioComponent(IMovieScenePlayer& Player, AActor* Actor, int32 RowIndex);/** Utility function for getting actors from objects array */TArray<AActor*> GetRuntimeActors(const TArray<TWeakObjectPtr<UObject>>& RuntimeObjects) const;private:/** Track that is being instanced */UMovieSceneAudioTrack* AudioTrack;/** Audio components to play our audio tracks with, one per row per actor */TArray< TMap<AActor*, TWeakObjectPtr<UAudioComponent> > > PlaybackAudioComponents;

而UMovieSceneAudioTrack的成员如下：

public:/** Adds a new sound cue to the audio */virtual void AddNewSound(USoundBase* Sound, float Time);/** @return The audio sections on this track */const TArray<UMovieSceneSection*>& GetAudioSections() const{return AudioSections;}/** @return true if this is a master audio track */bool IsAMasterTrack() const;public:// UMovieSceneTrack interfacevirtual TSharedPtr<IMovieSceneTrackInstance> CreateInstance() override;virtual void RemoveAllAnimationData() override;virtual bool HasSection(const UMovieSceneSection& Section) const override;virtual void AddSection(UMovieSceneSection& Section) override;virtual void RemoveSection(UMovieSceneSection& Section) override;virtual bool IsEmpty() const override;virtual TRange<float> GetSectionBoundaries() const override;virtual const TArray<UMovieSceneSection*>& GetAllSections() const override;virtual bool SupportsMultipleRows() const override;private:/** List of all master audio sections */UPROPERTY()TArray<UMovieSceneSection*> AudioSections;

经调试发现，正是私有成员AudioSections保存了语音信息。

2、获取语音

首先，我们可以获取世界中所有的ALevelSequenceActor，而从ALevelSequenceActor我们可以得到它正在运行的实例FMovieSceneSequenceInstance，这个过程都能够用相关类的公有函数实现，代码基本形式如下：

for (TActorIterator<ALevelSequenceActor> iter(GetWorld()); iter; ++iter){ALevelSequenceActor* seq = *iter;ULevelSequencePlayer* player = seq->SequencePlayer;if (player == nullptr){continue;}if (!player->IsPlaying()){continue;}TSharedRef<FMovieSceneSequenceInstance> movieSceneInstance = ((IMovieScenePlayer*)player)->GetRootMovieSceneSequenceInstance();
}

那么如何从movieSceneInstance拿到AudioSections呢？这里实际上有这样一个调用的过程（所有的调试读者若有兴趣，可自行尝试一下）：

(1) movieSceneInstance → TMap<FGuid, FMovieSceneObjectBindingInstance> ObjectBindingInstances

ObjectBindingInstances就是此序列器上所有Object的实例，如下图的序列器，其ObjectBindingInstances是BP_BulletStable、C02S01_BP和Guide Cine各自的轨道实例。最上面的 Shots是被嵌套的另一个序列器，在此序列器实例上是其成员变量MasterTrackInstances。

(2) FMovieSceneObjectBindingInstance ObjectBindingInstance → FMovieSceneInstanceMap TrackInstances

TSharedPtr<IMovieSceneTrackInstance> TrackInstance → UMovieSceneAudioTrack* AudioTrack

TrackInstances是相应Object下面的子轨道，比如在上面的序列器中，C02S01_BP有三条子轨道，分别是生成的、Transform和Cine Command。

(3) AudioTrack → TArray<UMovieSceneSection*> AudioSections

现在我们能够从movieSceneInstance找到AudioSections，但问题是，在上面的过程中ObjectBindingInstances和AudioTrack都是相应类的私有成员，并不能直接得到。那么这里就用到了比较hack的方法。

C++编译器对类编译时，如果类成员结构不变，成员变量相对类实例的地址偏移是固定的。因此我们可以通过实例地址计算成员变量的地址，然后将其取出。地址偏移是调试时通过添加监视得到的。下面是得到ObjectBindingInstances的代码：

    uint8_t* p = reinterpret_cast<uint8_t*>(&MovieSceneSequenceInstance);TMap<FGuid, FMovieSceneObjectBindingInstanceMock>* ObjectBindingInstances = (TMap<FGuid, FMovieSceneObjectBindingInstanceMock>*)(p + 0x78);

FMovieSceneObjectBindingInstanceMock的定义实际上和FMovieSceneObjectBindingInstance相同，重新定义是因为FMovieSceneObjectBindingInstance是一个私有结构体类型，不能在外部使用。这是第二个hack的地方，即成员相同但名称不同的类型可以相互转换。

3、判断某段语音是否在播放

要判断当前是否在播放语音，需要拿到AudioComponent，它在FMovieSceneAudioTrackInstance类中，即PlaybackAudioComponents，也是私有成员。

而在第二部分内容中，我们看到AudioSections是一个Array，即一个轨道上可能有多段语音，那么需要判断哪段语音在播放。每个AudioSection都有它的起始时间和结束时间，同时ULevelSequencePlayer里含有TimeCursorPosition私有成员，即当前序列器播放的时间。从而我们能得到正在播放的那段语音。ULevelSequencePlayer的部分成员如下：

private:/** The level sequence to play. */UPROPERTY(transient)ULevelSequence* LevelSequence;/** The level sequence player. */UPROPERTY(transient)ULevelSequencePlayer* CurrentPlayer;/** Whether we're currently playing. If false, then sequence playback is paused or was never started. */UPROPERTY()bool bIsPlaying;/** Whether we're currently playing in reverse. */UPROPERTY()bool bReversePlayback;/** True where we're waiting for the first update of the sequence after calling StartPlayingNextTick. */bool bPendingFirstUpdate;/** The current time cursor position within the sequence (in seconds) */UPROPERTY()float TimeCursorPosition;/** The time cursor position in the previous update. */float LastCursorPosition;/** Time time at which to start playing the sequence (defaults to the lower bound of the sequence's play range) */float StartTime;/** Time time at which to end playing the sequence (defaults to the upper bound of the sequence's play range) */float EndTime;/** Specific playback settings for the animation. */UPROPERTY()FLevelSequencePlaybackSettings PlaybackSettings;/** The number of times we have looped in the current playback */int32 CurrentNumLoops;/** Whether this player has cleaned up the level sequence after it has stopped playing or not */bool bHasCleanedUpSequence;

此功能的完整代码如下：

USoundWave* AGuideCine::FindSound(TSharedPtr<FMovieSceneSequenceInstance> MovieSceneSequenceInstancePtr, ULevelSequencePlayer* SequencePlayer)
{uint8_t* p = reinterpret_cast<uint8_t*>((uint64_t*)SequencePlayer);float SequenceStartTime = *(float*)(p + 0x74);uint8_t* p2 = reinterpret_cast<uint8_t*>(&MovieSceneSequenceInstancePtr);p2 = reinterpret_cast<uint8_t*>(*((uint64_t*)p2));TMap<FGuid, FMovieSceneObjectBindingInstanceMock>* ObjectBindingInstances = (TMap<FGuid, FMovieSceneObjectBindingInstanceMock>*)(p2 + 0x78);for (auto& ObjectBindingInstance : *ObjectBindingInstances){TArray<TWeakObjectPtr<UObject>> RuntimeObjects = ObjectBindingInstance.Value.RuntimeObjects;//Find the binding instance of the specific objectif (RuntimeObjects.Num() == 1 && RuntimeObjects[0]->GetName().Contains(ObjectNameInSequencer)){FMovieSceneInstanceMap TrackInstances = ObjectBindingInstance.Value.TrackInstances;for (auto& TrackInstance : TrackInstances){const FString MovieSceneAudioTrackStr = "MovieSceneAudioTrack";//Find the audio track of the specific objectif (TrackInstance.Key->GetName().Contains(MovieSceneAudioTrackStr)){uint8_t* p3 = reinterpret_cast<uint8_t*>(&TrackInstance.Value);p3 = reinterpret_cast<uint8_t*>(*((uint64_t*)p3));UMovieSceneAudioTrack* AudioTrack = *(UMovieSceneAudioTrack**)(p3 + 0x8);TArray<UMovieSceneSection*> AudioSections = AudioTrack->GetAllSections();TArray<TMap<AActor*, TWeakObjectPtr<UAudioComponent>>> AudioMapArray = *((TArray<TMap<AActor*, TWeakObjectPtr<UAudioComponent>>>*)(p3 + 0x10));for (int32 i = 0; i < AudioMapArray.Num(); i++){TMap<AActor*, TWeakObjectPtr<UAudioComponent>> AudioMap = AudioMapArray[i];for (auto Audio : AudioMap){TWeakObjectPtr<UAudioComponent> AudioComponent = Audio.Value;if (AudioComponent->IsPlaying()){float TimeCursorPosition = *(float*)(p + 0x6c);for (int32 j = 0; j < AudioSections.Num(); j++){UMovieSceneAudioSection* AudioSection = (UMovieSceneAudioSection*)AudioSections[j];//Find the audio section which is being playedif (AudioSection->GetStartTime() <= TimeCursorPosition + SequenceStartTime&& AudioSection->GetEndTime() >= TimeCursorPosition + SequenceStartTime){USoundBase* SoundBase = AudioSection->GetSound();USoundWave* Sound = Cast<USoundWave>(SoundBase);if (!Sound)     //Sound played in sequencer is of class USoundCue.{USoundCue* SoundCue = Cast<USoundCue>(SoundBase);USoundNodeWavePlayer* SoundNodeWavePlayer = Cast<USoundNodeWavePlayer>(SoundCue->FirstNode);Sound = SoundNodeWavePlayer->GetSoundWave();}return Sound;}}}}}break;}}break;}}return nullptr;
}