CLF Spec Review Period

This is our only chance to make CLF right. Reusing an XML element from another namespace (a.g. AMF’s) does not imply that CLF only deals with ACES.

A transformID has an additional meaning in ACES, yet it is tstill the “unique ID of a color transform”, and that’s it. XML is full of inheritance from other namespaces – that is why XML stands for eXtensible Markup Language.
There is really no XML problem in having an xmlns:amf attribute in CLF elements referencing stuff defined in AMF. But the reverse is also true. We may define transformID and other elements in clf, then have them used in AMF as well by means of a xmlns:clf attribute in the AMFs’ elements.
I am just urging to think of extensibility and case uniformity across the two: it’s now or never.

So I see no one opposing to introducing UUIDs in the end. I don’t mind about which casing is used, as long as it’s consistent across both the whole CLF and AMF specs.

Zach and all: digital signatures shall be an optional feature:

  • Signing a CLF just follows the standard W3C rules on XML-DSIG (signed XML documents).
  • CLF writers may add a digital signature (which is a self-contained <ds:Signature> root element).
  • CLF parsers may understand signed CLFs and, in case they do and a signed CLF is parsed, the signature shall be validated; otherwise
  • CLF parsers that don’t honor signed CLFs shall discard the digital signature and parse signed CLFs as well as unsigned CLFs are parsed.

Again, signed CLFs are useful to enforce one or many of the above workflows:

  1. Ensuring a CLF is not altered from the original one (e.g. trying to tweak with some camera parameters during on set pre-grading).
  2. Ensuring that the CLF comes from an authorized color-science lab (e.g. a specific DI/finishing department or a content-owning studio) and is not replaced by anyone else – this is enforced by validating the signature against the lab’s public key
  3. In some cases CLFs are kept “secret sauces” and not handed off to departments (e.g. a studio may not give the show LUT out to any B-cam departments) – yet we need to have an AMF with the correct pipeline set. So the hash or the digital signature of the CLF may still need to be put in the AMF to be later compared with the CLF’s own hash/signature.

Namespaces that are not used in an XML elements should never stay in there; it’s a waste of CLF payload.
However, quoting or crediting original creators does make sense, and this is usually accomplished in the principal namespace definition itself. For example, by means of a xmlns:clf attribute valued with a URL (rather than a URI) pointing to a web page where a document is shown clearly defining the CLF standard and its authorship / credits – which may or may not include ASC, & Co.

Note that there is an “id” in the ProcessList (and in each individual ProcessNode). It is defined as: “a string to serve as a unique identifier of the ProcessList

I made another pass through the whole document and here are a few more small issues I found and suggestions for how to fix them:

Section 3.1
The latter part of the sentence “For simplicity, the CLF does not track color spaces, nor does it require an application to convert to a particular color space before use.” does not make sense. A CLF does assume that an image is in the intended input colour space. I find that whole paragraph confusing and don’t feel it adds anything not already covered better elsewhere. I propose removing the paragraph.

Section 3.2.1
The phrase “A series of operations may be rolled into a single 3D LUT process node” is misleading. In reality, doing that conversion accurately may not be possible and I’m not aware of any published algorithms for doing so that actually work in general. I would remove that whole paragraph since it is confusing and the topic is better covered elsewhere.

Section 4.2
I would remove Note 3 from the “id” attribute. If an ACEStransformID is present, then the separate element must be present. The note implies that someone could put it in the “id” instead and that would be more difficult for applications to handle. (There is nothing preventing someone from setting the “id” to the same as the element, but it should not be a substitute for the element.)

The ACEStransformID section should add: “This element is mandatory for ACES transforms and may be referenced from ACES Metadata Files.”

You may want to put the Info element at the end of the section since otherwise one has to look carefully at the indentation to see that the following elements are siblings rather than children of Info.

4.4.4 Matrix

Please add the following to the dim attribute section:

Note: Previous versions of the specification used three integers rather than two for the Matrix Arrray dim attribute. In order to facilitate backwards compatibility, implementations should allow a third value for the dim attribute and may simply ignore it.

4.4.5 Range
Change the sentence: “The Range element can also be used in to clamp values.” to “The Range element can also be used to clamp values on only the top or bottom end.”

4.4.6 Log
OCIO has historically used a default of 2 for the base, so if no one objects I would prefer to use that here too. Also, it simplifies implementations a bit to require that the same base be used for all channels. So please change the default for the ‘base’ attribute on pg 19 and on pg 20 as the second to last sentence of the ‘channel’ paragraph add: “However, the same base value must be used for all channels.”

In the ‘logSideSlope’ and the next 3 attributes, the word “segment” is confusing since all 4 of those apply to the log segment. Perhaps “log side of the log segment” and “linear side of the log segment” would be more accurate.

In the ‘linSideBreak’ attribute paragraph, add that it is also required for cameraLogToLin.

In Note 3 change “continuous in slope” to “continuous in value” and “derivative of the logarithmic portion” to “value of the logarithmic portion”.

In the section about Eq. 4.19, change “Then, the value” to “Then, if linearSlope was not provided, the value”. Also remove the second “the” from the start of the next sentence. And in Eq. 4.19 add a multiply sign after logSideSlope.

In Example 6, remove the ‘linearOffset’ attribute.

4.4.7 Exponent

Add ‘monCurveMirrorFwd’ and ‘moncurveMirrorRev’ to the list of styles.

From the ‘exponent’ attribute you could remove the sentence “The nominal value is 1.0.” since there is no default and it doesn’t make sense to use that value in practice.

In Example 8, change the style from monCurveRev to monCurveFwd. In Example 9, change the style from monCurveFwd to monCurveRev.

4.4.8 ASC CDL

For the “Fwd” style, please add that it is the default (since the attribute is listed as optional).

As previously noted, I would suggest changing the valid range values to match what is in the ASC’s spec. Specifically, this means: slope >= 0, power > 0, sat >= 0, (offset unbounded). For example, change slope from “The valid range is [0.0, 100.0].” to “Valid slope values must be greater than or equal to zero.”

5.3 Efficient Processing

I find the existing section misleading and not helpful. I propose replacing it with:

The transform engine may merge some ProcessNodes in order to obtain better performance. For example, adjacent Matrix operators may be combined into a single Matrix. However, in general, combining operators in a way that preserves accuracy is difficult and should be avoided.

Hardware implementations may need to convert all ProcessNodes into some other form that is consistent with what the hardware supports. For example, combining all ProcessNodes into a single LUT3D. Using a grid size of 64 or larger is recommended to preserve as much accuracy as possible. Implementors should be aware that the success of such approximations varies greatly with the nature of the input and output color spaces. For example, if the input color space is scene-linear in nature, it may be necessary to use a “shaper LUT” or similar nonlinearity before the LUT3D in order to convert values into a more perceptually uniform representation.

5.4 Interpolation Types

This contradicts information provided elsewhere in the spec and should be removed.

5.5 Extensions

Please change “Unrecognized values” to “Unrecognized elements that are not children of the Info element”. And at the end of that paragraph add: “Applications that need to add custom metadata should place it under the Info element rather than at the top level of the ProcessList.”

6 Examples
I find example 12 to be more confusing than helpful. I’ll start another thread for proposing examples.

Appendix B

Revise IndexMap references.

Mention change to Matrix Array dim attribute.

I found another issue:

4.4.2 LUT1D

The LUT1D attributes for halfDomain and rawHalfs have disappeared. Please restore those using the text from S-2014-006. Since there is only one interpolation method allowed, I suppose we don’t need to restore the interpolation attribute.

Good, but then the format of the id element/attribute should be specified. If we simply assume it is going to be a UUID, what happens when (for sure) some implementations will put another string inside it?

Just to play devil’s advocate here…

  1. How is this enforceable?
  2. How would one assess the validity of ACEStransformID (beyond syntax)? All such mechanisms seem firmly outside the scope of CLF, and firmly within the scope of AMF; especially considering the validity of ACEStransformID is completely independent of that of the process nodes themselves;
  3. There are multiple ways to articulate the same cumulative processes within CLF. If one CLF represents a particular ‘csc’ transform in terms of Log and Matrix nodes, and another CLF samples array data elicited from rendering a CTL… are they both valid?
  4. What happens if multiple CLFs share the same ACEStransformID?

I’m still not convinced that ACEStransformID provides anything that a UUID doesn’t do better. I feel it’s AMF’s job to imbue transforms with unambiguous identities and relate them, not CLF’s.

Scott, nice job with the the December 20th version, it seems to be getting very close!

I found just a few more issues:

General comment: The equation labels seem to have disappeared (e.g., 4.18 - 4.20 in the Log section).

4.4.5 Range
Earlier in the thread I had asked to change the sentence “The Range element can also be used in to clamp values.” to “The Range element can also be used to clamp values on only the top or bottom end.” But I didn’t realize there were two such sentences. I wanted this change to be applied to the second one, right at the bottom of pg 17 rather than the one in the first paragraph (which should be changed back). The motivation here is that the second one is specifically introducing the case where there is only a min or only a max.

4.4.6 Log
We need to specify how to handle values <=0 for the logarithms. In the equations for log10, log2, linToLog, and cameraLinToLog, where you take the log of x, please insert MAX(x, FLT_MIN), where FLT_MIN = 1.175494e-38.

Note 3 is not quite right. The linearOffset still needs to be computed even if linearSlope is provided and the derivative is not continuous at the break. I suggest replacing the middle sentence of that paragraph with: “This value is calculated using the position of the break point and the linearSlope in order to ensure continuity of the two segments.”

4.4.7 Exponent
For the exponent attribute section, please restore the sentence: “If “style” is any of the ”monCurve” types, the valid range is [1.0, 10.0]. The nominal value is 1.0.” For monCurve, the exponent really cannot go below 1 or else the math falls apart. The 10 is an arbitrary limit, which I know we are trying to remove, but is probably fine in this particular instance. Likewise restore the offset sentence: “The valid range is [0.0, 0.9].”

The quote symbols in examples 8-11 got messed up in the latest draft.

5.1.3 Input to and Output From a ProcessNode
At the end of the third paragraph please change “scale factor of 4095” to “scale factor of 1/4095”.

Otherwise, looks good!

1 Like

Thanks, @doug_walker for the review.
I’ve made the suggested changes and replaced the document on Dropbox with the current version.

I implore others to also take a chance to review the current version of the document: http://j.mp/S-2014-006

Hello Zach.
I agree with you that ACEStransformID is not really enforceable the way it is written here
Again, I’d favor its attribute name to be namespaced like something:transformID. When TransformID uses the ACES naming convention in S-2014-002, then it makes completely sense having something = amf or something = aces, since that is its ontology.

Going back to the generic id attribute comment. That may be formatted as either:

  1. a UUID-style identifier… urn:uuid:.....
  2. an ACES Transform ID according to syntax in S-2014-002
  3. an hash of the transform or the CLF itself
  4. any other identifier string

So, without any further and “stronger” XML typing, how is an implementation supposed to parse this attribute value? Try to match a reg.exp. against #1, otherwise against #2, then vs #3, and fallback to #4 if anything else fails?
I think there are more streamlined ways to implement this in an XML schema.
Proposal: Instead than having just id attribute, use of either:

  1. a uuid attribute (in case of #1), and/or
  2. an ACEStransformID attribute (in case of #2), and/or
  3. a hashalg and digest attributes couple (in case of #3)
  4. a generic id attribute (in case of #4).

As a last comment, I find ACES Transform IDs utility in CLF for three usecases:

  • when a CLF implements an ACES standard transform, but even more so
  • when a CLF is some creative or technical LUT made up of several components, some of which are exactly ACES standard transforms.
  • to improve the capability of LUT managers to address and aggregate and catalog large collections of LUTs by segmenting into and acknowledging by common components/nodes.

While the first usescase is trivial, the second and third are outstanding (and the second the most subtle of all):

Just think of a show LUT from Log.C to P3 which internally uses a creative LUT baked in ACEScg.
Such a LUT is probably a concatenation of at least:

  • an IDT 3d LUT from Log.C to ACES2065-1,
  • a 3d LUT from ACES2065-1 to ACEScg,
  • the creative 3D LUT itself (in an ACES workflow, this node would be an LMT),
  • a 3d LUT from ACEScg to ACES2065-1
  • the RRT 3d LUT
  • the ODT 3d LUT from ACES2065-1 into DCI P3

Please not this may but, in fact, is not AMF. Such a CLF concatenates 5 out of 6 nodes which implement standard transforms that can be addressed by Transform ID. The CLF is a non-ACES monolithic LUT, so it may not make sense to have it represented as an AMF.

If the CLF standard is capable of flagging one of the passages of the LUT with the proper ACES transform ID, an application parsing the CLF may acknowledge what the passage is and, possibly, replacing the LUT mechanics with its own ACES implementation, which may be either more performing and more accurate than an interpolation baked in one LUT.

If one needs the exact mathematical implementation of the LUT, instead, this may even be counterproductive. Yet, id-like attributes are still useful for the last of the three usecases… so utilities can factor in “common parts” in a series of LUTs that do similar things.

Of course, nothing prevents applications to honour other IDs and UUIDs to trigger internal operations for non-ACES color transforms as well – if they have means to do so.

Hope my post is clear and explanatory.

1 Like

As we finish up our implementation, we found a few minor issues to report with the latest draft:

  1. Examples 3, 4, and 14 still use the old Matrix array dims.

  2. For Range, I propose adding the following clarifications. At the end of the last paragraph on pg 17 add: “(The style shall not be ‘noClamp’ for this use-case.)” In the maxInValue element definition add: “The maxInValue shall be greater than the minInValue.” In the maxOutValue element definition add: “The maxOutValue shall be greater than or equal to the minOutValue.”

XML Schema issues

  1. The exponent attribute in ExponentParams is not optional.

  2. The ExponentType is missing the required style attribute.

  3. ExponentParams and LogParams should be minOccurs=1, maxOccurs=3.

  4. In the LogParamsType, base should have a default of 2. There should not be a default for linSideBreak. The linearOffset is not an attribute and should be removed. (All as per the main spec.)

  5. In ArrayType, the dim attribute is required and its minLength should be 2.

We don’t find XML schema validation useful since it misses so many things, so we don’t use it. Given how many errors I was able to find just from a quick read-through of the current schema, I don’t have very high confidence that it is entirely correct. Therefore, my preference would be to remove the “normative” indicator above the section title so that it is clear the main text takes precedence in case of any errors.

1 Like

Thanks again, Doug. I’ve rolled in the fixes and and replaced the document on Dropbox with the current version: S-2014-006 - A Common File Format for Look-Up Tables

Hey @sdyer - shouldn’t the date specified in the document be January 17, 2020?

Yes, it should. I will update that along with a few other typo corrections that I have pending which will be fixed when 1.2 officially goes out (I’m just waiting on AMF specification to be revised based on feedback on the Release Candidate).

Thanks!

1 Like

Would anyone object if we retitled S-2014-006
from
"A Common File Format for Look-Up Tables"
to
"Common LUT Format (CLF)"
or, if verbosity it preferred,
"Common LUT Format (CLF) – A Common File Format for Look-Up Tables"
?

It just seems weird to me that we don’t say either Common LUT Format or CLF in the title of the document, and yet that is what everyone calls it. This is something that has always bothered me but just occurred to me that we have an opportunity to fix since we’re updating the document anyway…

Any opposed? Or any better suggestions?

1 Like

I quite like the verbose one:

Common LUT Format (CLF) – A Common File Format for Look-Up Tables

Cheers,

Thomas

Sounds fine, whichever you gents prefer!

I also favored this one and is what I changed it to. I think it’s still formal but also more clearly states what it is. And it will much more likely to return results when people inevitably search for “CLF” or “Common LUT Format” on Google.

I would propose
Common LUT Format (CLF) – A common file format for color look-up tables
Because, strictly speaking, a LUT is what it is: a look-up table, but what we really refer by “LUT” to within the motion picture industry is, actually, a “color look-up table”.

I don’t think we can get too hung up on this. Two points:

  1. CLF has a far wider range of operators than just look-up tables. Indeed many CLF files may contain no look-up table at all, particularly with the introduction of the log process nodes. So the name is not strictly accurate in the first place.

  2. While in the context of the film industry a CLF is intended to process image data, a LUT is “blind” to the meaning of the data it processes. There is nothing to say that if CLF is widely adopted that other industries couldn’t used it on other types of data.

Another suggestion.
I can’t find the file extension recommendations written anywhere in the specs.

Despite I’m a fan of embedding XMLs in other XMLs by extending the XML language, I do belive 90% of the usage will still be putting a CLF in a file of its own. Therefore, clear indication as to the file extension and MIME types should be written in the final spec?

I propose:

  • File Extension: .CLF.xml
  • MIME type: application/xml+clf

The reason why I propose to use a double extension ending with .xml is because, whatever software is installed with your Operating System, you have a chance to properly open up for review/edit the file because:

  1. for those systems with a CLF-aware app installed (e.g. a LUT manager or color-grading tool), the app acknowledges the .CLF.xml “full” extension and you can still open up the CLF with such app
  2. for whoever does not have a CLF-aware app, any application opening XML files would still be associated to CLF, allowing to view and edit it in the corresponding semantic editor, because the final .xml extension would still be acknowledged.

Let me know what you think and please note that I’m posting an analogue suggestion in the ACES Metadata File Spec draft thread (.AMF.xml file extension in this case).