The first thing was to read the VFX book that Conann gave us, this was to gain a better understanding of the subject area making it easier to research further. Here are the notes that I took from the VFX book, I tried to condense the information as much as possible but I was pretty surprised at just how technical rig removal and retouching actually was so apologies for the amount of writing 🙂
Rig removal and or retouching is a vital part of any production. E.g wire removal, crew and tracker removal etc. Its all down to a clean up artist, basically they sort out any problems in the shot. I’m sure you have heard the phrase ‘Paint it out in post’, this refers to many things that need to be removed from the shot such as electrical wires, flying wires (think superman), sets, tracking markers, accidental errors, reflections. A clean up artists ultimate goal is for their work to go unnoticed by the viewer, making the shot look untouched.
Painting frame by frame –
- However, it is almost impossible to get painted frames to tie into a sequence of images.
- If its dust or a wire that catches light on isolated frames this method works well.
- It is seen as an impractical and time consuming process.
Image stabilizing, painting and inverting –
- If the camera is moving and the object is physically still, using image stabilizing (on the shot) can save alot of time before the painting begins. Its easier to paint out if the objects appears to be still, after the painting is finished you can invert the stabilization to the painted object.
Grain is a random optical texture in processed photographic film. It is caused by small particles that have received enough light in order to show up. The faster the film speed (ISO) the more grain will appear. ISO 100 is a slow film that needs more light to achieve good exposure. ISO 600-800 is a fast film speed that needs less light for good exposure. In good strong sunlight an ISO speed of 100 can be used and still only get subtle grain.
Digital noise is a random variation in image brightness and color information, usually caused by electrical noise. We can mimic the ISO settings of film by increasing gain setting of a digital camera’s sensor. Increased gain means far less light is needed for good exposure (like high ISO), this also increases the electrical noise. When a clean up artist is painting things out it is extremely important for them to match the grain/noise levels for a seamless outcome.
matching grain with a grain card –
A noise card provided from each camera and lens combination that’s used on set would be used by the clean up artist to match them, this way is easier to match type and feel of grain. If there isn’t a clean grain card shot, the type of grain can be estimated by using a plate with a flat consistent tone. Green screen is perfect for this.
Matching grain by eye –
There is three main things to be considered. The size, irregularity and density of the grain. The RGB channels have to be matched separately as they all have different grain characteristics.
Matching grain by cloning –
Using cloning copies color, luminescence and the grain from an area of shot. This is a preferred method over painting from scratch.
De-graining and re-graining –
Using the matched grain, remove all the grain completely. This leaves the shot completely clean. Now the painting process begins and the grain is re-added back to the whole shot. This can sometimes be seen as a waste of resources as not every pixel needed changing. It also degrades the film footage by stripping all the grain out then putting it all back again.
Tracking marker removal->
This is a very common job for the clean up artist, the method used to do so varies from shot to shot depending on which is best sorted to each shot.
Patch removal – When removing tracking markers from green screen they can just be cover with the same shade of green as the entire screen will be removed anyway.
Simply copy one section of the image and cover the marker with this section. Its not a live effect and the grain is maintained throughout the shot.
This is a live effect. Its good for matching grain, luminescence and hue changes. The area selection is vitally important as anything that moves into the selected spot will be duplicated in the painted area.
Marker removal tools –
This is a specific tool for marker removal. It allows you to rotoscope shapes to be drawn around the marker. It then takes the pixel values at the edge of those shapes to fill the hole created by the shape. However poor source location can be a problem. If the marker is out of the boundary of the shot it will cause errors. If there is already a matte created this can be used when using the marker removal tool
Dust marks tend to only be a problem over a couple of frames, cloning from a shot in the future or past (A frame or two) can remove the issue effectively, this maintains the luminescence, hue and grain. Only when the camera focus is moving very fast this method will not work as the background will change rapidly.
Just like removing markers there are specific tools made for removing wires. You can draw a rotospline over the wire, then specify the thickness of the wire and the tool will use the surrounding pixel values to fill the hole. The width can be animated if the wire moves closer to the camera. If the wire is moving on a static shot, a temporary clone from another frame can be used.
Rig removal ->
This refers to filming equipment, sets or props that need to be removed from shot. This is another very common job. There are a few methods to do this –
2.5D repainting –
You can paint out objects from a single frame, then use it over the rest of shot. Using planar tracking, the painted object will match movement, perspective and distortion from other frames. Start with a good frame where the object is large and as flat as possible and has little motion blur. It will then need painted over and then it can be applied to the planar track. This method will only work if the artist can convincingly paint with perspective and depth. A down point would be that this can be very time consuming. A full 3D environment can be created to make it quicker and easier.
3D techniques –
It is possible to use a 3D environment and camera projections to make the job of removing objects alot easier. (Software like Nuke, Fusion or Flame are needed).
Building a 3D clean up setup- Projection based –
Firstly an accurate 3D camera track of the scene is needed. Using Nuke’s internal camera tracker the image can be tracked. A piece of geometry is then added to the area that needs the clean up. Its essential that the geometry is placed correctly on the axis’. (Scaled over the object by about 25% extra). Ensure that camera track is turned on. Then paint a single frame and applying this as a texture to the created geometry. Then camera can then be copied and attached to a project 3D node. The animation from the projection camera is then removed. To fix the perspective you can add a second clean plate and use it to create a new camera projection set up based on a new frame. Another option is to add a roto when the clean plate (painting) and then soften the edges.
Building a 3D clean up setup – UV based –
Removing the objects with the unwrapped UV’s of the geometry. Again a 3D camera track is needed along with the geometry with the 25% extra coverage. Then disconnect the camera from the scanline render and create a project 3D node. In the scanline render (with only the scene node going into it) change the projection mode to UV. Then paint a single frame and retexture the geometry with it. Add the roto node to the painted geometry, just to smooth it into the surface.
The straight camera projection method is quicker and easier to set-up but requires the added work in dealing with the perspective issue. The UV method is more complex to setup but means you dont have to paint with the perspective or distortion.