Rawtoaces - Calling All “developer-types”

cameras
idt
Tags: #<Tag:0x00007f3c68205278> #<Tag:0x00007f3c682050e8>

(KirkThibault) #21

At this point, rawtoaces does not build.

On my other machine, I changed the link.txt file made by make to explicitly point to the libAcesContainer.dylib in /usr/local/lib. The build then completed (make, then make install), and I could run rawtoaces. When I did, and used -v -v -v verbosity, the terminal output indicated that the file had made its way through libraw and then there was no more output. The process completed, but there was no file written.

Something is amiss with aces_container. Is there any way to test the function of aces_container, independent of rawtoaces?

kirk


(Alex Forsythe) #22

@kirkthibault you can test aces_container in conjunction with CTL. Try doing a brew install ctl then grab the ACES transforms from http://www.github.com/ampas/aces-dev. If you have an exr file on your machine and you want to convert it to an ACES 2065-4 file, run in through the unity CTL transform using the -format aces command on the output.

So the command will look something like this :
ctlrender -ctl aces-dev/transforms/ctl/utilities/ACESutil.Unity.ctl inFile.exr outFile.exr -format aces

If that executes fine, do :
exrheader outFile.exr

You should see a header that includes :
acesImageContainerFlag (type int): 1

and

chromaticities (type chromaticities): red (0.7347 0.2653) green (0 1) blue (0.0001 -0.077) white (0.32168 0.33767)

That would verify that aces_container is installed and working properly.


(KirkThibault) #23

@Alexander_Forsythe

Thanks for the reply. Your suggestion worked - here is the terminal transcript:

Kirks-MacBook-Pro-2:EXR kirkt$ ctlrender -ctl /Users/kirkt/Desktop/ACES-DEV/aces-dev/transforms/ctl/utilities/ACESutil.Unity.ctl LumaRiverOUTEXR.exr outfile.exr -format aces
Kirks-MacBook-Pro-2:EXR kirkt$ exrheader outfile.exr

file outfile.exr:

file format version: 2, flags 0x0
acesImageContainerFlag (type int): 1
channels (type chlist):
    A, 16-bit floating-point, sampling 1 1
    B, 16-bit floating-point, sampling 1 1
    G, 16-bit floating-point, sampling 1 1
    R, 16-bit floating-point, sampling 1 1
chromaticities (type chromaticities):
    red   (0.7347 0.2653)
    green (0 1)
    blue  (0.0001 -0.077)
    white (0.32168 0.33767)
compression (type compression): none
dataWindow (type box2i): (0 0) - (6713 4483)
displayWindow (type box2i): (0 0) - (6713 4483)
lineOrder (type lineOrder): increasing y
pixelAspectRatio (type float): 1
screenWindowCenter (type v2f): (0 0)
screenWindowWidth (type float): 1
type (type string): "scanlineimage"

Again, I am not too familiar with the need for files to access each other during compilation, but I wonder if this is somehow related to OS X Sierra sandboxing or some similar kind of silent problem.

I will keep trying!

kirk


(KirkThibault) #24

OK! Progress. I tried manually editing the link.txt file again, replacing the following section at the end of the file:

-lAcesContainer

with the hardcoded path to the dylib:

/usr/local/lib/libAcesContainer.dylib

This is what I had done previously, and it did not work. However, I think that after linking to the homebrew version (as pictured in the screenshot above and suggested by Mio) and then performing this edit, compilation worked and an EXR was output.

For whatever reason, the shortcut reference to the dylib is not working, but replacing it with the literal path to the link works.

Yay for brute force!

Thanks for your help on this, I look forward to testing it.

kirk

FYI - the link.txt file to which I am referring is located in:

rawtoaces/build/CMakeFiles/rawtoaces.dir/link.txt

(KirkThibault) #25

One initial observation I have made in the ten minutes I have tested - if you feed rawtoaces a raw file (for example a .CR2 canon raw file) it works and the EXR with “aces” added in the filename is created.

However, if you convert the raw file to a DNG and execute rawtoaces on the DNG, no EXR file is output.

Here is the terminal transcript for the conversion of a CR2 file:

Kirks-MacBook-Pro-2:EXR kirkt$ rawtoaces --mat-method 2 --wb-method 0 -v -v -v -v _MG_0112.CR2

Starting rawtoaces ...
Processing _MG_0112.CR2 ...
CB: Reading metadata  pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data  pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 5D Mark IV ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
CB: Scaling colors  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors  pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating  pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating  pass 0 of 4495 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 4495 iterations)
CB: Interpolating  pass 506 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 1012 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 1518 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 2024 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 2530 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 3036 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 3542 of 4495 (data passed=Sample data passed)
CB: Interpolating  pass 4048 of 4495 (data passed=Sample data passed)
CB: Converting to RGB  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB  pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
The IDT matrix is ...
   0.622841, -0.048300, -0.104717
   -0.445778, 1.231981, 0.309404
   -0.068663, 0.262731, 0.773828
The final white balance coefficients are ...
   1.950195   1.000000   1.444336
Writing ACES file to _MG_0112_aces.exr ...
Finished

and here is the transcript from the DNG conversion:

Kirks-MacBook-Pro-2:EXR kirkt$ rawtoaces --mat-method 2 --wb-method 0 -v -v -v -v _MG_0112.DNG

Starting rawtoaces ...
Processing _MG_0112.DNG ...
CB: Reading metadata  pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data  pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 5D Mark IV ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
CB: Scaling colors  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors  pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating  pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating  pass 0 of 4483 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 4483 iterations)
CB: Interpolating  pass 506 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 1012 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 1518 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 2024 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 2530 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 3036 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 3542 of 4483 (data passed=Sample data passed)
CB: Interpolating  pass 4048 of 4483 (data passed=Sample data passed)
CB: Converting to RGB  pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB  pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
Finished

kirk


(Miaoqi Zhu) #26

Hi Kirk,

Thanks for the effort! Glad to hear that your method worked!

For “DNG” file, we have not implemented functions to covert DNG to ACES yet, so right now “rawtoaces” just passes DGN files. We will let you know when “rawtoaces” supports DGN.

Please let us know if you have any other questions.

Thanks,

Mio


(Miaoqi Zhu) #27

Hi Kirk,

I think I found a way to fix the issue.

Can you replace the “CMakeLists.txt” in the root directory with the one from this link?

Please run it and let us know if it goes through.

Thanks and have a great day!

Mio


(KirkThibault) #28

Hi Mio,

It worked! Just to give you some details - I created a new clone of rawtoaces, replaced the CMakeLists.txt in root with your file from Dropbox. I created the build directory, entered it, did “cmake …” and then “make” and the make command was successful. BUT! Remember that previously I had linked aces_container to the Homebrew version per your recommendation in Post 17 above (in case that makes a difference to your solution).

Anyway, your solution worked for me - thank you!

As a related aside, OCIO has been implemented as a Photoshop plug-in - this is a great way of using rawtoaces to bring an EXR into photoshop as ACES and convert it to ACEScg (or whatever other form you want) for linear 32bit workflows in PS.

Here is the link to the OCIO plug-in:

Kirk


(KirkThibault) #29

As a follow up - I cloned the most recent (as of 18 July 2017) rawtoaces git repository, with the new CMakeList.txt file and compilation was successful on my work machine, the original machine that was having problems.

Thanks again,

kirk


(Jed Smith) #30

Thank you for this fantastic tool! I just discovered it and have been playing around.

I successfully compiled it on Linux (Fedora 26) after tracking down all the dependencies. It works great for CR2 raw files.

I have a lot of DNG files as well from Magic Lantern dualiso processing. I was sad to realize that DNG files aren’t yet supported, and then excited to come across the above discussion.

I tried recompiling with the above CMakeLists.txt from a new clone of the git repo. It compiled successfully, but unfortunately here the behavior is still the same. The command appears to process but no file is output. Here is the output of the command being run.

$ rawtoaces -v -v -v -v --wb-method 0 --mat-method 2 _MG_0436-2.DNG
Starting rawtoaces …
Processing _MG_0436-2.DNG …
CB: Reading metadata pass 1 of 2 (data passed=Sample data passed)
Reading metadata finished
CB: Reading RAW data pass 0 of 2 (data passed=Sample data passed)
start_timevaling Reading RAW data (expecting 2 iterations)
CB: Reading RAW data pass 1 of 2 (data passed=Sample data passed)
Reading RAW data finished
The camera has been identified as a Canon EOS 60D …
White Balance method is 0 - Using white balance factors from file metadata …
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw …
CB: Scaling colors pass 0 of 2 (data passed=Sample data passed)
start_timevaling Scaling colors (expecting 2 iterations)
CB: Scaling colors pass 1 of 2 (data passed=Sample data passed)
Scaling colors finished
CB: Pre-interpolating pass 0 of 2 (data passed=Sample data passed)
start_timevaling Pre-interpolating (expecting 2 iterations)
CB: Pre-interpolating pass 1 of 2 (data passed=Sample data passed)
Pre-interpolating finished
CB: Interpolating pass 0 of 3458 (data passed=Sample data passed)
start_timevaling Interpolating (expecting 3458 iterations)
CB: Converting to RGB pass 0 of 2 (data passed=Sample data passed)
start_timevaling Converting to RGB (expecting 2 iterations)
CB: Converting to RGB pass 1 of 2 (data passed=Sample data passed)
Converting to RGB finished
Finished

I’m not sure if DNG support is still in the works, or if the above CMakeLists.txt no longer works due to recent changes in the code, but it would be great to have!

Thank you for your hard work on this project and let me know if I can help troubleshoot or test in any way!


(Miaoqi Zhu) #31

Hi Jed,

Thanks for testing the software for us.

You are right that we are not currently supporting DNG files. I was too busy to simply make the software stop further processing when the images are identified as “DNG” files

Please try the latest version of “rawtoaces” on the “feature/optimization” branch. We will merge it into the “master” soon. The processing will be halted right after the camera is identified. The speed is faster too.

Have a great day!

Mio


(Miaoqi Zhu) #32

Hi Jed,

Can you send me the original DNG file? I am making some changes to support DNG.

Thanks,

Miaoqi


(Jed Smith) #33

Hey Miaoqi,
Yes I would be happy to help out! There are several variations of the DNG format:

  • CinemaDNG
  • Lossy Compressed DNG
  • Lossless Compressed DNG
  • Uncompressed DNG

You’ve probably already found it but there’s a pretty complete description of the DNG specification here:
https://helpx.adobe.com/photoshop/digital-negative.html

I’ve uploaded some example DNG files to a gist here - if you run this command it should download them to your computer.

git clone https://gist.github.com/jedypod/b8d8ac3c302a85948e4de3e5edf0f673

I’ve included a normal dng from my Canon 60D digital slr camera, converted from Canon CR2 to DNG using standard settings on the Adobe DNG Converter software: canon_60d_lossless_compression.dng

And there’s a dng with lossy compression converted from a Sony A7s raw ARW file: sony_a7s_lossy_compression.dng

There’s also a couple of cinemadng files, one from the Black Magic Cinema Camera and one from the Axiom Beta (links in the sources.md file): blackmagic_cinema_camera_cinemadng.dng
axiom_beta_video_camera.dng

Hopefully this is enough to start testing with. Thanks a lot for working on this and let me know if you need anything!


(Julian Martinz) #34

Hi there,

I am having some issues. The compile went well but when I am trying to convert a CR2 rawtoaces raises an assertionerror.
I already tried to unbind/uninstall/reinstall aces_container without any luck. I think aces container was already installed due to a ctl installation during the compile. Maybe that could cause the issue?
Here is the terminal transcript:

rawtoaces -H 1 --mat-method 2  -v -v -v -v /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646.CR2 

Starting rawtoaces ...
Processing /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646.CR2 ...
The camera has been identified as a Canon EOS 5D Mark III ...
White Balance method is 0 - Using white balance factors from file metadata ...
IDT matrix calculation method is 2 - Calculating IDT matrix using adobe coeffs included in libraw ...
The IDT matrix is ...
   0.649774, -0.075815, -0.116831
   -0.434096, 1.256459, 0.250427
   -0.066794, 0.228016, 0.690909
The final white balance coefficients are ...
   1.000000   0.489250   0.745342
Writing ACES file to /Users/JulianM/Desktop/acescg_sphere_test/_P0B9646_aces.exr ...
Assertion failed: (outputBufferSize < 100e6), function configure, file /tmp/aces_container20151026-55213-ashfox/aces_container-1.0.1/aces_Writer.cpp, line 206.
Abort trap: 6

Best,
Julian


(Andrew) #35

This looks like a known buffer issue. Use the “-h” flag for now.

https://github.com/ampas/rawtoaces/issues/79


(Francois Lord) #36

Hi Alex,

I downloaded rawtoaces and compiled it. It works well with the images I tried. I would like to request two things, though. I would like the ability to control the compression in the resulting exr file. Currently it exports a completely uncompressed file which can be huge compared to the raw original. Uncompressed exr is very rarely used in the wild since there are several lossless compression methods available in the exr specs. And sometimes using the dwaa lossy compression scheme can also be useful when images only serve as references.

Also, It would be nice to be able to choose AGEScg as the output colorspace for the images, since most VFX houses use that colorspace internally to store textures and plates.

I understand that these two changes break the compatibility with the SMPTE 2065-4 ACES container format. But I would certainly don’t mind if the acesImageContainerFlag was not set in those cases.


(Miaoqi Zhu) #37

Hi Jed,

I just finished a draft version of “rawtoaces” that support DNG conversion.

Can you look at the converted ACES based on your DNG files and let me know if they look good to you?

Please notice that I did not convert “axiom_beta_video_camera.dng”.

Thanks and have a great weekend!

Mio


(Alex Forsythe) #38

Hi there Francois,

This is a tough one … I understand the desire to not have to deal with transcoding, but we’re really trying to make sure what gets interchanged is ACES 2065-1 images. Granted with EXR there’s metadata to help sort out issue, but we don’t want to end up back in the same boat where you’ve got to ask the sender “How is that file encoded?”

I think there’s food for though here on both sides. I’ve personally wondered if it might be advantageous if the EXR libraries allowed for transcoding on read/write. Idk … just an idea.


(Jed Smith) #39

Hey Miaoqi,
I took a look at the converted exrs and they look great.

Looking forward to this feature getting released. Thanks a lot!


(Miaoqi Zhu) #40

Hi Jed,

You can rebuild “rawtoaces” that is able to convert DNGs:

https://github.com/miaoqi/rawtoaces/tree/feature/DNG

Please keep in mind that if the camera spectral sensitivity data is found to be available, you should still use the default setting to convert DNGs; otherwise, it is recommended to “–mat-method 1 --wb-method 0” for other DNGs.

Please test with more images and let us know how it works for you.

Thanks,

Mio