Prototyping Custom Clips in Unity Timeline

This is a post describing my attempts at prototyping a custom Unity Timeline Playable for use in production.
It grew from a more focused question intended to be posted on the Unity Timeline Forums. There were times I got stuck, and started to clearly describe the current state of my progress. Every time I did, however, it led me to a new insight in how to tackle my problems.

The reason I share my progress so far is twofold:

1. I'm sure I made some wrong assumptions along the way, and it would be nice to have these pointed out to me. I still intend to post this on the Unity Forums, hoping to get some feedback to improve this use case.
2. Since I had to go through quite a few forum posts and make some mental somersaults to get this far, having the whole process here might help others.
   
   

The Goal

Manipulate a Transform, then when the clip has passed, the Transform should retain the manipulation.
The next clip should additively apply it's manipulation on top of the previous one.

To illustrate this example, each clip here rotates the Transform 90°, and cumulatively adds to previous transformations.

Call them 'Additive Clips' if you will. These clips would be stacked behind each other, and the transforms nested, to create complex sequences quickly, easily editable by an artist.

Why Timeline?

Maybe this raises the question: Why not simply use animation?
We tried to do this process by hand, but that required considerably more manual work. This is especially true while tweaking parts of the sequence. The manual approach will severely limit the amount of assets we can create.
Also, the next step is to store curves in assets, and referencing them in these clips, allowing more complex fragments of animation to be re-used in different situations, further increasing the speed at which these assets can be created.

Ok then, Why not create a custom tool with the same functionality?
Since the requirements of this use case are arguably trivial, we could easily create a custom tool to do this for us. And we have, but we stumbled upon the following complications:

• Other parts of our scene greatly benefit from being driven by timeline, notably supporting animations and camerawork. Having Timeline co-exist next to a custom system is a tough proposition.
• The sole reason asset creation would be faster in comparison to using standard animations would be the ease at which these transformations could be tested, edited, scaled and timed, requiring a custom GUI to be written, which represents considerable development and maintenance time (which we do not have). That approach then seems rather ridiculous considering the Timeline + Playable API combo appears to support this exact situation.

Initial Attempts this can't be that hard, now can it?

I am using Unity 2017.3.0f3 for these attempts.
Since Timeline and the Playables API are relatively young, and my comprehension of the architecture and usage of these tools through the documentation is hazy at best, I based my trails on forum posts, and the following Unite Talk:

Thanks to the Default Playables package, implementing the actual transformations is trivial. Getting them to last once the next clip plays got kind of tricky.

        For clarity, I named the 4 needed custom classes as such:
                CustomTransformationBehaviour 
                CustomTransformationClip 
                CustomTransformationMixerBehaviour 
                CustomTransformationTrack 
        and will keep on using these names in this post.
 
You should watch the video if you haven't already, but basically the gist of it (as far as I can gather) is as follows:

• The Track class represents a custom track.
• The Clip class represents a clip on this track. Note that it defines the clip caps, and contains a 'Template' of the Behaviour type. This template is used when creating the corresponding playable in the underlying playable graph.
• This Behaviour then contains the data representing the contents of the clip, exposed in the 'Template' part of the inspector when selecting a clip.
• The track creates one instance of the MixerBehaviour playable when the playable graph is build. And this mixer is responsible for producing the final outcome.

1. Implementing everything in CustomTransformationMixerBehaviour.ProcessFrame

In concrete terms, for this use case to work, the mixer would need to know if a clip has finished when mixing the different clips together.

• playable.GetInputWeight(i)
  in the mixer did not help me, since it drops to 0 at the start of the next clip.
• var InputPlayable = (ScriptPlayable<CustomBehaviour>)playable.GetInput(i);
  InputPlayable.GetTime()
  did not work either, since it does not get updated if the clip is not under the playhead.
  When scrubbing past a clip, this value is not always consistent, so can not be indicative of the end of a clip.
• I found no other data accessible in the scope of 'CustomTransformationMixerBehaviour.ProcessFrame' to find out if a clip has passed.

2. Leveraging CustomTransformationBehaviour.OnBehaviourPause

As described in this forum post, OnBehaviourPause fires when a clip has ended.
It could be used to set an 'IsClipFinished' flag on the behaviour.
The trouble is that it also fires when Timeline starts playing or is paused, which makes it and unreliable check for this use case.
The suggested fix for this would be to check
        playable.GetTime() + info.deltaTime >= playable.GetDuration()
but that did not seem to work in my attempt.
Also, this method does not seem to cover jumping around with the playhead.

3. Pass clip start and end times from CustomTransformationTrack through to CustomTransformationBehaviour

As described in this forum post, since the CustomTransformationBehaviour is accessible from the mixer, accessing the corresponding clip start and end times from there would solve this use case.

Right now I'm getting this data to the mixer by

• Overriding CreateTrackMixer in the CustomTransformationTrack class,
  getting the start & end times for each clip, and passing them to the custom clip class. 
• Overriding CreatePlayable in the CustomTransformationClip class, passing those same start & end times of that specific clip on to the custom behaviour class.
• Accessing these values in the mixer, when applying the transformations.
  For this to work I get the timeline 'playhead time' as described in this forum post.

See this Gist for a summary of the code I used to get this working.

Using this approach, I was able to correctly implement this use case.
It feels weird though, having to funnel these values around.
 
But a tad more concerning: this approach has it's own complication:
This whole process kicks in when the Playable Graph is created. This graph gets recreated when clips are added, deleted, the timeline gets re-selected, etc..., but not when a clip changes size or is repositioned.
 

Changing a clip's position or size breaks this solution

When a clip is repositioned or rescaled, this custom clip start/end data is out of sync, resulting in the mixer using old values. These are the possible avenues I came up with when tackling this problem:

1. Forcibly rebuilding the graph at CustomTransformationMixerBehaviour.ProcessFrame
Found no way to do this.
 
2. Having the CustomTransformationClip passing a reference of itself to CustomTransformationBehaviour
Since CustomTransformationClip is just a playable implementing ITimelineClipAsset, I found no way to access the actual clip start and end points form there.
Also, I'm not even sure that even if I got some usable values here, they would update upon clip move/resize, since the graph does not get rebuilt.
 
3. Passing a reference of CustomTransformationTrack straight through to CustomTransformationMixerBehaviour
If CustomTransformationMixerBehaviour has the track reference, it can fetch all the clips again on ProcessFrame, and get the correct start and end times.
I have gone down that road, and this approach seems to work, but has issues I'd rather not see in a production pipeline.
See this Gist for the actual implementation, but please; reference only. Don't try this at home.

• First of all, I'm sure I'm making someone cry when reading this.
  Fetching all the clips again every ProcessFrame seems blatantly wrong.
   
• Second, as far as I could figure out, the CustomTransformationMixerBehaviour has no real way of finding out which 'inputPlayable' is represented by which clip.
  This solution completely hinges on the assumption that the clip collection fetched from the Track is
  a - the same size as the inputCount in the mixer.
  b - are in the same order of the inputPlayable's fetched from playable.GetInput(i).

Logically these assumptions make sense, but as a programmer I can only cringe at my own solution.

At this point it started to look more like a hack than a workaround, and it would not be wise to use this tool in a production setting in it's current state.

4. Saving the actual Timeline clip reference in CustomTransformationBehaviour
This is the solution I went for in the end. I was so close in attempt 2. While hovering my cursor over the 'post new thread' in the Unity Timeline forums, something hit me. As pointed out earlier, I can get clip data in the mixer by handing it from the track class straight down to the behaviour class.
So instead of passing it's start and end times. Why not pass the actual clip reference instead? That way, accessing it from the mixer, I can always get the correct start and end times, without the nastiness of my previous attempt.

See this Gist for the code of this attempt.

Some extra notes

At the time of writing, this is roughly where the tool stands. We are currently giving it a spin to test the workflow.
A few things popped up:

The Track's GatherProperties implementation
In both the Unite talk mentioned in the beginning, as well as in some examples in the Default Playables package, the custom Track's GatherProperties is overridden with this implementation:

However, the Default Playables package did not generate this code for me.
The speaker touched upon this briefly, and while I grasp the overall  concept, I have no Idea why it is only present in some examples, or whether or not I should include it in my own custom track.

Naming Clips
As you may have noticed in the last code example, CustomTransformationBehaviour contains a CustomTransformation reference that can be set in the template dropdown when selecting a clip. This points to a scriptable object, allowing configurations to be re-used on multiple clips. 

 
Since that's the defining characteristic of our clips, I thought it would be appropriate to name them accordingly, since the default name is just the custom clip's class name.
I found this forum post, describing how to set the clip's name in the track's CreateTrackMixer function. But since I already have a nice reference to my clip in CustomTransformationBehaviour, I can neatly set it there:

Turning it into this beauty:

Since then, someone pointed me to the clip naming box, which I completely overlooked. We now prefer naming clips by hand. I still included this solution here, maybe someone finds it useful.

Stuff I did not figure out yet.

• For some reason, newly created clips have a length of 300. I'd love to be able to set a default value.
  It would be neat to hook into the actual timeline clip creation, setting it (and possibly other parameters) there. All I can reach now is the creation of the playable, which happens every time the graph is rebuilt.
• As previously noted, a ScriptableObject; CustomTransformation is the cornerstone of a clip. It would be nice to support dragging one onto the timeline, automatically creating a CustomTransformationClip, similar to dropping an AudioClip.
  (I have noticed however, that dropping an AudioClip on an existing audio track always creates a new track. I'd expect it to add a clip to the existing track instead)
  

On Closing

I hope this helps someone, and gives me some answers.
I want to thank the people at Unity for their continuous efforts to provide us with increasingly better tools. If all goes well, we will be able to save a lot of time using Timeline, while hand crafted sequences used to be painstaking to create.

Please feel free to clue me in on what nuances I am missing, or if I'm being completely oblivious to some obvious solution. It can only be to the benefit of everyone passing through here.

I would love to have some concrete feedback or pointers from the actual Unity staff on this, so we can confidently go into production.

Cheers!