When it comes to conformance testing, it takes more than just binary pass or fail baseline tests to provide reports that encourage the strongest possible implementation of a standard. The Internet Engineering Task Force (IETF) uses the keywords “MUST” and “MUST NOT”, as defined in RFC 2119, to mandate specific behaviors across their standards. Neglecting to adhere to these requirements will result in failed compliance tests. However, defining "correct" behavior is not always black and white. To address this issue, the IETF uses the keywords “MAY, MAY NOT, SHOULD” and “SHOULD NOT” to define optional and recommended behaviors. Violating clauses operating under these keywords does not indicate a violation of protocol, but may suggest a less than robust implementation of such protocol. Protocol focused informative tests detail how devices respond to test cases in which the standard does not explicitly state the expected behavior of a device.
A good example of how informative tests are used to strengthen the implementation of a standard can be seen in the iSCSI Consortium at the University of New Hampshire InterOperability Laboratory (UNH-IOL). Here, informative tests are useful in identifying potential security weaknesses, specifically in vendor implementations of the Challenge Handshake Authentication Protocol (CHAP). Although CHAP is defined by its own standard (RFC 1994), the iSCSI standard defines its own requirements for CHAP to which all conforming iSCSI implementations must adhere. These iSCSI specific CHAP requirements provide the foundation for the iSCSI Initiator and iSCSI Target CHAP test suites. Informative tests within these test suites detail how iSCSI implementations behave in accordance with certain stronger, more secure, CHAP requirements defined in RFC 1994. For example, RFC 7143 (the document defining the iSCSI protocol) does not mandate that received CHAP_C values be checked for reuse. However, a strong implementation of CHAP may perform exhaustive protocol conformance checking on received protocol data units (PDUs) and detect the reused CHAP_C value. The benefit of the detecting a reused CHAP_C value is that it could indicate an attempt at a replay attack, but checking all incoming PDUs for non-mandated security considerations could have an adverse impact on performance. In this instance, informative tests determine how vendors weigh performance over security.
Some informative tests, such as the example cited above, deal directly with the strength of a protocol implementation, while others are used to provide observed, quantitative measurements to vendors. An example of such quantitative measurement focused, informative tests can be seen within the SATA Consortium at the UNH-IOL. Within the SATA-IO test suite precisely measured impedance values observed during testing are included alongside conformance test results. The measurements obtained through this testing may be more precise than the vendors own in house test equipment can produce. Providing said measurements to the engineers responsible for designing and updating these products can be beneficial for the designers, as they would neither have to acquire the measurement tools necessary to provide this precision, nor perform the testing; both of which may be cost prohibitive. Maintaining a collection of top of the line test and measurement equipment capable of precise measurements is an example of the UNH-IOL staying true to our vision to be the world's premier data networking resource. We do this by using our knowledge, tools, services and relationships to enable the next generation of technology and engineers.
It is our hope that the inclusion of informative testing, in addition to our conformance and interoperability based testing will help to foster multi-vendor interoperability, conformance to standards, and the improvement of data networking. Passing protocol conformance based tests and exhibiting the recommended behavior within informative tests is an indication of an implementation that not only conforms to a standard, but also performs more than the minimum specified requirements. In addition, providing vendors with precise quantitative measurements can aid in analyzing and improving device behavior, both of which lead to increased robustness. Stronger implementations result in better interoperability and improve data networking as a whole.
Learn more about the iSCSI Test Suites, here.