PNG-Tinterr
Manual in development =)
interactive coloring of film recordings
Beware! This manual was originally written in Czech, then automatically translated to English and then I corrected the most obvious translation errors but I suppose there many others that still need to be found. Let me know when you spot something that does not make sense.
© Vít Čondak 2020-2024
0. Introduction
The PNG-Tinterr program is intended for coloring black and white films, but it can also be used for coloring individual photos.
To facilitate the colorization of movies, he uses the Frankensteiner utility (PNG-Frank2 (C# .NET) or PNG-Frank3 (C++ .exe)), which is used to add colors on adjacent frames, saving part of the work.
For reasons that are not entirely clear even to me, I have used English descriptions everywhere in the program, and I have not ventured yet to support other languages. =}
The publicly published version is fully functional, only adding the Tinterr logo to the upper right corner of the image.
1. Preparation
The program deliberately does not work directly with a compressed video file, because most video files use lossy compression, and every time you save the edited frames back, they would inevitably be distorted by the compression algorithm.
The program therefore uses images exported to .png format as working material.
It does not export the images itself, so you need to use another program for export - preferably the freely available FFMPEG utility.
The current version can be downloaded from ffmpeg.org
The utility is controlled from the command line, and the command to extract all images looks something like this:
ffmpeg -i "name-of-video-file.mp4" %06d.png
The name of the video file will of course vary depending on what file you want to edit. =)
The second parameter specifies the format of how the image files will be named. "%06d" means that the names will be six-digit numbers (000000-999999), which should usually be plenty for a normal movie. (With the usual 25 frames per second and a footage of 90 minutes, we get about 135,000 frames. In extreme cases, the name would need to be extended to 7 digits and the parameter would therefore be "%07d".)
I prefer to point out that unzipped images will of course take up a lot of disk space, so you have to take that into account. The size of each frame varies depending on the resolution and complexity, but for a black and white movie it will probably be between 1 and 2 MB, so roughly 250 GB of free space is usually needed to unpack the entire movie. (And of course another 250GB for any backups or working versions.)
If there are black edges in the video (e.g. the video has a size of 1920x1080 (16x9), but the used part of the image is only 1440x1080 (4x3), so there are black edges of the same width on both sides, it is advisable to cut the edges directly during extraction, because this will make them smaller the size of all the files on the disk, and in addition, it will make all subsequent operations a little faster.
The command line will then look like this:
ffmpeg -i "name-of-video-file.mp4" -filter "crop=1490:1080" %06d.png
("Crop" means cropping, the first number (in this case 1440) is the width of the resulting area, the second number (1080) is its height. If the edges are of different widths, two more numbers can be added (here it would be 240 and 0) indicate the offset of the upper left corner of the cropped area relative to the original.)
Most movies also contain a soundtrack, so it is advisable to export it to a separate file as well. If you are also going to edit the audio somehow (PNG-Tinterr doesn't handle that), it is advisable to use the WAV format:
ffmpeg -i "name-of-video-file.mp4" audio.wav
Otherwise, it is also possible to export the original audio track without decompression. E.g.
ffmpeg -i "name-of-video-file.mp4" -vn -acodec copy audio.aac
In practice, it is a relatively good idea to remove scratches from black-and-white images before coloring them (I use the Png-Denny program for this) and, if necessary, somehow unify their brightness, because its fluctuation between adjacent images will have a negative effect on the quality of the calculated color masks.
2. Startup and Setup
PNG-Tinterr is written in C# and uses the .NET Framework 4, so it's quite small and doesn't need to be installed, but just copied into the project directory. For proper function, the png-frank3.exe program must be placed in the same directory.
The calculation of the colors of adjacent frames is determined by two parameters:
| Mask reduction level | Since the calculation time depends on the number of pixels in the image and video compression usually merges the color shade of neighboring pixels in the same way, by default the program uses color masks with half the resolution (i.e. with a quarter size) compared to the original black and white images, which corresponds to a value of 1 (reduction of the first degree). By setting a different value, it is possible to increase the details (4x) at the cost of slowing down the calculation (5x) or vice versa. For a FullHD movie, a value of 1 is a reasonable compromise. |
| Limiting the search distance | Lower values mean that the application tries to find objects in the image that are moved further away (with a value of 0 up to about 20% of the total image size), but this may result in finding more areas that are similar in shape and color, but it is a different object. Setting the ideal value also depends on the type of video (how much movement there is in it) and on the number of key frames (the more there are, the more practical it is to reduce the distance). The default value is 2 (ie about 5% of the total size of the image). |
Other subdirectories have fixed names and the program creates them as needed:
| 4-tpng | working directory (cache) where calculated transformation files are stored |
| 5-master | directory where hand-drawn color masks are stored (in half resolution) |
| 6-mask | working directory with the generated transformed color masks |
| 7-blended | output directory where the merged color masks are saved (by default this function is disabled) |
| 8-colored | output directory where colored images are saved (each time a changed box is visited) |
The interpolation calculations can be improved a bit by adding another input directory 2-depth and saving the frames with grayscale depth maps in it. However, these must first be generated using other applications. There are several of these and they behave differently in different videos:
| MiDaS | https://github.com/isl-org/MiDaS |
| Vita | https://github.com/KexianHust/ViTA |
| NVDS | https://github.com/RaymondWang987/NVDS |
3. Principle of work
To use the program, it is good to get at least a rough idea of how it works inside, what its capabilities are, and what its limitations are.
At the beginning of the work, it is necessary to select a sufficiently representative image (a schematic "map" of neighboring frames is on the top bar), change it to a key (for example, with the K key) and color it all manually. The color is stored in a separate layer and the program continuously applies it to the black and white image. For better orientation, it is possible to switch the display and display only the original black-and-white image or, conversely, only the color mask (Z and X keys).
Png-Frank will immediately start generating transform files for the neighboring boxes in the background, so if you move to them (for example with the left and right arrow keys), you should soon see a colored version of them calculated based on the key (pattern) box.
Unfortunately, the recalculation is not intelligent enough to perfectly color any box according to the pattern - this is not even possible in principle, because new objects or characters that were not in the original image may appear in the shot. The more the other boxes differ from the original key, the more deviations and errors will appear in the calculated color mask. For a frame where the deviation exceeds some tolerable limit, it is usually necessary to insert a new key frame and manually correct all deviations and errors.
The colors from the new key frame are automatically applied for the calculation of the following frames, but also for the calculation of the color of the previous generated frames up to the previous key frame. For the generated frames, it is usually the case that their colors are added from the two nearest key frames and (if they exist) also the two nearest partial frames (a higher number can be set on the right bar) and the result is mixed according to the set method. The default setting is Best match, which selects the color of each pixel from the calculated version that has higher reliability at that location. Other methods of color computed in different versions mix according to confidence ratios or according to keyframe distance.
For work, it is usually more convenient to use Best match, as it allows more precise control over the calculated frames, but for exporting the result, the Error ratios method may be more convenient. This will display suspicious areas in less saturated colors and hide them in a merciful gray haze, but it usually also changes uncertain areas that would otherwise be the correct color, so the differences need to be checked.
The color mask can be smoothed before application by setting the Blur color mask option, but the effect on the appearance of the resulting image is almost imperceptible. (However, the saved file is usually a bit smaller.)
If files with image depth are part of the project, it is also possible to set the ratio of importance between the black and white image and the depth map for the calculations.
4. Editing
You can use a mouse or a tablet pen to edit the color mask. All actions are classically performed by clicking or holding down the left mouse button, while holding down the right mouse button allows you to move the image if the entire image does not fit in the window in the current view. When working with the pen, scrolling mode can be invoked by holding down the Space key.
Similarly, the Alt key can be used to invoke the working color setting mode by clicking directly into the image (i.e. the Eyedropper or Colour-picker function).
Working modes:
| Brush (B) | The brush allows you to draw using a set color. Double-click can be used to draw straight lines between the point of the double-click and the point where the previous brush stroke ended. |
| Eraser (R) | The eraser at the point of use removes all information about the color from the color mask and thus creates a hole in the mask. If it was a key frame, the intervention will change it to a partial frame and the deleted area will be added to the neighbors. |
| Fill (F) | The fill applies the set color to an area with similar parameters (color, lightness). It is possible to set the tolerance and which layer is used to find the boundary. It can be a layer with a black and white image, with a color mask, or both at the same time. |
| Separate (S) | It replaces the defined shades (again it is possible to set which layer is used as a criterion for the application - black and white, color or both) with the set color. This is a tool for situations where you need to adjust some color shades in the entire image at once. |
| Brush Shape (Shift+B) | Toggles between a round brush with a soft edge and a square brush with a sharp edge. |
Color palette:
The palette is used to store colors that are used repeatedly, so that the user can always paint the same object with the same color on different images.
| Left thickness mice | sets the color from the palette as active |
| Right thickness mice | saves the active color to the palette |
| Alt + Left Shift | allows you to rename the color in the palette |
Gradient:
This mode is used for "variable color" drawing and is especially useful for coloring the jagged border of two areas with different brightness, such as dark tree branches with a light sky behind them. A gradient is defined by two color shades and their corresponding brightness levels. When drawing, a gradient color corresponding to the brightness of the original black and white pixel is then applied to each pixel. So in the case of tree branches, dark pixels should be colored brown, light sky pixels blue, and gray pixels a mix of brown and blue.
| A | toggles gradient mode on and off |
| Q | shuffles primary and secondary gradient points (colors and their limits) |
| W | it only shuffles black and white gradient boundaries |
| E | turns on the automatic setting of the gradient bounds based on the location of the first click |
Clipboard:
| Ctrl+C | copies the entire local color mask to the clipboard |
| Ctrl+Shift+C | copies the entire mixed color mask to the clipboard |
| Ctrl+V | pastes the contents of the clipboard into the local color mask |
| Ctrl+Shift+V | if there is a keyframe in the clipboard, it will set it as the first morph |
| Ctrl+Alt+Shift+V | if there is a keyframe in the clipboard, it will paste the corresponding areas into the local color mask |
Keyboard shortcuts when editing:
| arrows ←→ | move to an adjacent frame |
| Ctrl+←→ | move by 10 frames |
| Ctrl+Shift+←→ | move by 100 frames |
| Ctrl+Alt+Shift+←→ | move by 1000 frames |
| Shift+←→ | move to the nearest key frame in the given direction |
| Alt+arrows | moves the entire local color mask in the direction of the arrow used |
| TO | converts the current frame to a key frame |
| Shift+Delete | deletes the contents of the local mask and changes the frame to generated |
| Ctrl+D | invokes a dialog for creating a dissolve (without calculating transformations) |
| Ctrl+U | invokes a dialog for local mask color adjustments (saturation and RGB ratio) |
Search:
| Ctrl+F | finds a frame with the highest calculated error |
| F3 | finds a frame with the following error value |
Debugging information:
| Ctrl+L | calculation log |
| Ctrl+M | log of mouse actions |
5. Creating an edited video
If you feel that everything is colored correctly, then move to the first frame and use the Render button, which will start regenerating all the frames (i.e., it will gradually move forward through the frames and save the colored versions (overwriting the existing files) in the 8-colored directory )
The colored images must first be combined into a new video file without sound:
ffmpeg -framerate 25 -i %06d.png -c:v libx264 -pix_fmt yuv420p -preset slower -crf 20 new-video-without-sound.mp4
Then you have to merge it with the audio track:
ffmpeg -i new-video-without-sound.mp4 -i audio.aac -c copy new-video-with-sound.mp4
6. Updates
2024-11-19 Ctrl+Alt+Shift+V
2024-01-06square brush, depth, E/R, Frank3
2021-12-23Ctrl+U
2021-12-11basic morphing settings
2021-12-08added more keyboard shortcuts and gradient work
2021-11-19added additional keyboard shortcuts (arrows, dissolve)
2021-11-17search with ctrl+F and F3
2021-04-02ctrl+shift+v
2021-03-07working with the palette
2021-01-06new blur option
2020-12-03shortcuts for debugging windows and modifying the appearance of the document
2020-11-12modified clipboard section and converted to tables
2020-11-10clipboard
2020-11-02opening image (base64) and editing basics
2020-10-31modifications related to working directory settings
2020-10-03first rough version