research for a digital cinema workflow under linux


Introduction

Research published in October, 2010.

The research presented here aims at establishing a digital cinema editing workflow made exclusively under Linux machines. The steps described below start at the point in which we already have a sequence, or many sequences, of digital negative (DNG) files. In the diagram, they correspond to the stages immediately after the third grey box. Everything described from there on has been tested and is documented here.

Even though it is the aim of this documentation to gather information related to the whole process – which includes capturing and monitoring the recording, transferring the data to the computer, playing the JP4 files and finally converting them to DNG sequences – the previous parts are can be retraced by putting together information that can be found at Apertus’ site or that is spread throughout Elphel’s Wiki. We can consider that, therefore, as a second step towards our objective.

This research first tests which format is best to be used as proxy. It takes into consideration that editors will need to do many test renders during editing and that a fine photographic adjustment of the images will be done only at the final stages (those proxies, then, must be easily replaceable), in which the workflow is divided between the photographer, the audio technician and the final retouches by the editor.


Preparations for editing


files to be used as proxies

We could render the DNG files generated by movie2dng using ufraw-batch. At this moment of the workflow, we are interested in generating proxies files – that is, light files that have three characteristics: they have to be able to used for editing; they have to present a good preview of the final video without compromising too much of the quality; and they have to register fast rendering times in our NLE (be it Cinelerra or Blender) so that we can do preview-renders of our edited video quickly and leave high CPU demands for post-editing.

To achieve so, however, we have to test which type of file would best fit into all these requirements. Would the best format be TIF of JPG? For the test below, I used 4 sample DNGs generated by Elphel downloaded from Apertus Project’s website. I copied them and pasted them into the same folder so that I would have 360 frames – or a good preview of what to expect from 15 seconds of a CIMAX recording (2592×1120) at 24fps. Note that the frames I used were even larger than the CIMAX format.

Test 1

Process 360 DNG frames (occupying 3,4GB of disk space), or 15s of RAW video footage

Command line used:

ufraw-batch --conf=apertus_teste.ufraw *.dng

format debayer pattern space in disk
(all files)		 processing time
for 15s		 processing time
for 60s (est.)		 processing time
for 10m (est.)
JPEG (50% compression) bilinear 44MB 4m 16m 2h40m
JPEG (50% compression) PPG 45MB 3m55s 15m40s 2h36m
JPEG (50% compression) VNG 44,5MB 5m25s 21m40s 3h36m
JPEG (50% compression) AHD 45MB 4m35s 18m10s 3h10m
JPEG (100% compression) bilinear 1,9GB 7m30s 30m 5h
JPEG (100% compression) AHD 2,0GB 9m25s 37m40s 6h15m
TIF (uncompressed) bilinear 5,1GB 4m15s 17m 2h50m
TIF (uncompressed) PPG 5,1GB 4m17s 17m08s 2h51m
TIF (uncompressed) VNG 5,1GB 5m57s 23m48s 3h58m
TIF (uncompressed) AHD 5,1GB 6m20s 25m20s 4h15m

Results: For what we can see from this first test, four formats have the potential to be used as proxies: JPEG 50% PPG, JPEG 50% Bilinear, TIF Uncompressed Bilinear and TIF Uncompressed PPG. The first two have the advantage of occupying very low disk space if compared to the third and fourth ones (44MB~ x 5,1GB). They can be a very interesting solution, especially for larger projects. But if we consider the workflow as a whole, the TIF formats should make out life easier at post-production.

The problem with this test is that the command line above processes only one image at a time. With some research, I came across a very simple software called parallel, that can be easily compiled (don’t use the pre-packaged versions, they are too old) and will help us to use all the cores of a multi-threaded processor, in my case, the Intel i7 860. Dividing the work between the cores of the processor dramatically reduced the time in my tests – generally, it took him half the time to complete the task; in some cases, it took him one third of the time.

Test 2

Process 360 DNG frames (occupying 3,4GB of disk space), or 15s of RAW video footage – using multi-threaded processing

Command line used:

ls *.dng | parallel -j +0 ufraw-batch --conf=apertus02.ufraw --silent {}

format debayer pattern space in disk
(all files)		 processing time
for 15s		 processing time
for 60s (est.)		 processing time
for 10m (est.)
JPEG (50% compression) bilinear 44MB 1m41s
(previous: 4m)		 6m44s 67m20s
JPEG (50% compression) PPG 45MB 1m44s
(previous: 3m55s)		 6m56s 69m20s
JPEG (50% compression) VNG 44,5MB 3m
(previous: 5m25s)		 12m 120m
JPEG (50% compression) AHD 45MB 2m26s
(previous: 4m35s)		 9m44s 97m20s
JPEG (100% compression) bilinear 1,9GB 2m20s
(previous: 7m30s)		 9m20 93m20s
JPEG (100% compression) AHD 2,0GB 3m05s
(previous: 9m25s)		 12m20s 123m20s
TIF (uncompressed) bilinear 5,1GB 2m59s
(previous: 4m15s)		 11m56s 119m20s
TIF (uncompressed) PPG 5,1GB 2m57s
(previous: 4m17s)		 11m48s 116m
TIF (uncompressed) VNG 5,1GB 3m13s
(previous: 5m57s)		 12m52s 128m40s
TIF (uncompressed) AHD 5,1GB 2m58s
(previous: 6m20s)		 11m52s 118m40s

Results: As it turns out, it seems that the best formats to be used as proxy are the JPEG 50% Bilinear and the JPEG 50% PPG. The observation about disk space occupied by both (see previous results) is still pertinent and reducing further the quality of the JPEGs (below 50%) may even fasten the overall conversion, but that must be tested in the timeline of the NLE, during a real editing project. The JPEG formats also benefited most from the multi-threaded task.

Space in disk worries

For the TIF formats, we must consider the enormous amount of disk space occupied by them. The table below is just a rough preview. I take into consideration only the DNGs converted by movie2dng and their processed TIF counterparts, by ufraw-batch. You should have in mind that there are still the original Elphel’s JP4 files, (many) Cinelerra preview renders you should make along the way, the temporary files you should use as the project goes through the whole process, original audio, audio for post-production and the final movie render.

source video DNGs UFRaw’s TIFs total
15s 3,4 GB 5,1 GB 8,5 GB
1min 13,6 GB 20,4 GB 34 GB
10min 136 GB 204 GB 340 GB
30min 408 GB 612 GB 1020 GB

Rendering tests using the proxy files

It is now time to check how these image sequences will behave in our NLE. My initial intent is to use Cinelerra as editor and Blender for effects, such as titles or post-production. So I imported the files generated by the tests above into a timeline, using CIMAX standard as reference (2592×1120 at 24fps). Note that for this test, I use only the video stream, since it’s too soon to preview which will be the best workflow for audio.

Test 3

Render 360 frames, or 15s of 2592×1120 video footage at 24fps from Cinelerra’s Timeline

source render characteristics time taken to
render 15s of video		 space in disk
occupied by renders
JPEG (50% compression) Bilinear MOV with JPEG Photo at 100% 34s 240MB
JPEG (50% compression) Bilinear MOV with JPEG Photo at 50% 32s 35MB
JPEG (50% compression) PPG MOV with JPEG Photo at 100% 34s 241MB
JPEG (50% compression) PPG MOV with JPEG Photo at 50% 32s 35,5MB
JPEG (50% compression) AHD MOV with JPEG Photo at 100% 1m51s 700MB
JPEG (50% compression) AHD MOV with JPEG Photo at 50% 1m59s 35,4MB
TIF (Uncompressed) Bilinear MOV with JPEG Photo at 100% 1m53s 640MB
TIF (Uncompressed) Bilinear MOV with JPEG Photo at 50% 1m49s 34,6MB
TIF (Uncompressed) Bilinear TIF Sequence (Uncompressed) 3m26s 2,96GB
TIF (Uncompressed) PPG MOV with JPEG Photo at 100% 1m56s 672MB
TIF (Uncompressed) PPG MOV with JPEG Photo at 50% 1m44s 35MB
TIF (Uncompressed) PPG TIF Sequence (Uncompressed) 2m38s 2,96GB
TIF (Uncompressed) AHD TIF Sequence (Uncompressed) 3m08s 2,96GB

Results: The JPEG 50% Bilinear and JPEG 50% PPG are the fastest, both rendering at similar speeds. The difference in time when rendering these formats with the JPEG Photo codec at 50% or 100% is almost irrelevant, but the space in disk occupied by them should weight in favour of JPEG Photo at 50%’s side. It is worthy noticing the behaviour of JPEG brought to Cinelerra using AHD. It takes about 4x to render when compared to its cousins – if rendered with JPEG Photo at 100%, it also takes way more space in disk. Working with TIF here serves for us to have an idea of the necessary time to render the final version of the video, that should be brought here using TIF Uncompressed AHD and rendered either as a TIF Uncompressed Sequence (to be encoded by MEncoder for standard outputs) or as JPEG Photo at 100% with PCM audio in a MOV container.

Overall conclusion from the tests

There are two main conclusions to be drawn from these tests. The first one is that the JPEG 50% Bilinear and JPEG 50% PPG are the best ones to be used as proxies. They are the fastest to be processed, both during ufraw’s batch conversion and cinelerra’s render: they take 7x real-time to be processed at the first step and only 2x real-time at the second one. They also occupy minimum space in disk, and can be easily previewed by MPlayer at anytime during the workflow.

But there is a major drawback in using the JPEG formats. If we combine them with img2list, we’ll have a hard time replacing the JPEGs at Cinelerra’s timeline with the final TIFs due to how img2list and Cinelerra work together (Cinelerra’s XML don’t point to the images, but to img2list’s generated list, which can’t be changed without harming Cinelerra’s interpretation of it). That should leave you two options. You can invert the workflow and do the photographic treatment before editing. That can be done for some movies; for others, it will be unthinkable. Or, more reasonably, you could replace the JPEG image blocks manually in Cinelerra’s timeline for the TIF ones. That can be less work than it seems, but you’d have to have the very final cut of the movie at the moment of replacement, since further editing – even a minor tweak – will become quite hard. In other words, you’d be fronzen there.

Update: The above mentioned is not necessarily true. In further research conducted on this workflow, we have been able to establish a link between automation scripts and the updating of Cinelerra’s XML, resulting in the best of bost worlds – we can use JPEG 50% as proxies and use TIF for the final render, gaining both speed and quality. Check out the new findings here.

The second conclusion is that, even though the TIF Uncompressed PPG and the TIF Uncompressed Bilinear will occupy way more space than the JPEGs and will take longer to be processed both at ufraw’s batch and cinelerra’s render phases, they may have advantages if you consider the workflow as a whole. Firstly, they will take 12x real-time at the first step and 7x real-time at render, should you combine them with JPEG Photo 50% for render previews. That is slow. However, that might be compensated at post-production. When you do the photographic treatment in UFRaw and generate new TIFs, you’ll be able to simpy replace the TIFs you had used for the new ones in the folder. Cinelerra will read the alterings just fine, even though its XML points to the img2list’s lists. The lists, in their turn, are already pointing to TIF files (this won’t work for JPEG, though. You won’t be able to use, for example, JPEG at 100% and replace the JPEG proxies at the folder – Cinelerra will break if you do that).

This means that you will still have your image sequences behaving like movie blocks at the timeline, which is crucial. In case you must change that single frame that went unnoticed or change the duration or order of anything, that should be quite smooth and effortless. You will also lose MPlayer’s ability of previewing a sequence of images as a file, which can be vey handy – but since the TIFs can be replaced at the folder and instantly read by Cinelerra, you might be able to check them directly at the timeline.

In both cases, though, you will have to be very organized with your files and UFRaw’s configuration files. Which way is best? Consider your project; consider the gear you have and judge for yourself.


Editing

To edit the image sequences, we should use a tool called img2list, developed by Claudio ‘Malefico’. If we simply import these frames into Cinelerra, they will be treated as single images by the software. Well, that’s what they actually are, but img2list will help Cinelerra read the frames as a ‘sequence of images’ (that is, a movie), which is exactly what we want to do. Now, they will behave exactly as movie blocks in the timeline. You will be able to split them, to stretch or shrink them exactly as if they were, for example, a DV file. As an aditional comment, img2list will work only if the image sequence is named in a certain pattern, which happens to be compatible with the pattern used by movie2dng.

Editing in Cinelerra is quite well known and very well documented, so I will skip the introductory steps here. Rendering the video in Cinelerra to preview the final result should take into consideration the tests presented above, so you should probably want to render the video in a MOV container, using JPEG Photo at 50% or 100% as codec settings.


Post-production

Image treatment should be done using the original DNG files, for the simple reason that they are RAW. Both Cinelerra and Blender are able to open DNG files from cameras, such as Pentax’s DNGs. But it seems that only Cinelerra will open Elphel’s converted DNGs without having to recompile the software. To work with DNGs in Cinelerra will be extremely time consuming, though. Minor tweaks in colour or slightly altering contrast will take an enormously long time to be previewed, transforming a delicate process into nightmarish hell (that’s the main reason why we transformed the original DNG into JPEG proxies in the first place).

A reasonable option would be to make Blender read Elphel’s DNG-converted files and use its compositing nodes tool to do the colour correction. That has yet to be tested. Also, we would have to establish a communication between Cinelerra’s EDL (a XML file) and Blender, so that we could import our EDL in Blender.

UFRaw, however, has the right tools and immediate preview. Its main problem is that it lacks the time-lapse factor (that is, you can only view a single, still, frame). That can arranged in terms, using MPlayer to preview the processed sequence (see below), but it should present difficulties in scenes that have moving cameras or strong changes in contrast. The following will consider a workflow using UFRaw.

First of all, we need to know which DNG files we’ll be working with. It would make no sense to go through DNGs that belong to recorded sequences we didn’t use in the final editing cut. The information we need is inside Cinelerra’s EDL, which is a XML file. By going through this file, you can know precisely which frames have to go through post-production. An example of the section we need inside Cinelerra’s XML is (click on image to enlarge):

This excerpt show two very small blocks of video in a single track, called “Video 1”. The first one uses 7 frames, there is a 3-frames space between the blocks and then there is a 6-frames video block. Visually, it would look like this in your timeline:

Now we must translate that information into human-readable terms. It must be simple to understand. We can make a script using the long command line below. For file “cinelerra.xml” as input, it will give us a file called “List_of_DNGs_for_post_production.txt”, which is a text file you can print or read in the computer, the way you feel more comfortable with. The line:

grep “EDIT STARTSOURCE=” cinelerra_outra_pasta.xml | cut -d”>” -f1,2 | cut -d”=” -f1,2,4,5 > temp_readableXML.txt && sed -ie ‘s/<EDIT STARTSOURCE=”/- frames /g’ temp_readableXML.txt && sed -ie ‘s/” CHANNEL=”/ to /g’ temp_readableXML.txt && sed -ie ‘s/”><FILE SRC=/ File: /g’ temp_readableXML.txt && grep File temp_readableXML.txt > temp_readableXML2.txt && gawk ‘{ $8 = $5 + $3; $9 = $3+1 ;print $6,$7,$1,$2,$9,$4,$8 }’ temp_readableXML2.txt > List_of_DNGs_for_post_production.txt && rm temp_readableXML*.txt temp_readableXML.txte


Will give us this more reassuring output in the text file:

File: “/home/livre/Desktop/testes_e_exemplos_elphel/dngs/img2list/lista” – frames 1 to 7
File: “/home/livre/Desktop/testes_e_exemplos_elphel/dngs/img2list/lista” – frames 8 to 13


Now it is clear which DNGs we should use. In this case, I’d just go through my file “lista” (which is a img2list file I had created previously for editing) and see which DNGs I’ll have to reprocess in UFRaw and ufraw-batch. The easiest way to do that would be to copy those files to a temporary folder, treat them and check them out directly at Cinelerra’s timeline or with MPlayer:

Command line used:

mplayer “mf://*.jpg” -mf fps=24:type=jpg -fs -vf dsize=2592:1120


When you’re satisfied with the results, copy the resulting files (probably Uncompressed AHD TIFs) and paste them into the original DNG’s folder. If you have used Uncompressed TIFs as proxies, you will be prompted to replace the TIFs in that folder. Do it, replace them. And you’re done.

Now, when you open your Cinelerra project again (that cinelerra.xml file, in our example), the program will read your new TIFs instead of the old ones and you’re ready to mix the other final sources (audio and lettering) for a final render.

Lettering and other effects should be done in Blender. Depending on Blender’s behaviour, we can use proxies to do it and export the result using an alpha channel, so that it can be brought into Cinelerra’s timeline for the final render. In case Blender is able to read our original files, we can do the final render inside it. In both scenarios, a Render Farm can be built to help the CPU efforts.

Research continues…

Update: the continuation to this research has been published. It shows how all the steps mentioned here can be easened by the use of automation scripts. Check out here.


Back to the main linux page.


Publicidade

Written by qazav_szaszak

28 de outubro de 2010 às 0:02

Uma resposta

Subscribe to comments with RSS.

  1. […] ver una versión ampliada de esta imagen […]

    Más cerca de un verdadero cine digital de código abierto de lo que pensamos 【2022】

    22 de janeiro de 2022 at 9:40


Comentários encerrados.