By Chema Gonzalez, 2021-07-21
This document describes h264nal, a simpler H264 (aka AVC aka h.264 aka ISO/IEC 14496-10 - MPEG-4 Part 10, Advanced Video Coding) NAL (network abstraction layer) unit parser.
Final goal it to create a binary that accepts a file in h264 Annex B format (.264) and dumps the contents of the parsed NALs.
h265nal is a similar project to parse H265 NAL units.
Get the git repo, and then build using cmake.
$ git clone https://github.com/chemag/h264nal
$ cd h264nal
$ mkdir build
$ cd build
$ cmake ..
$ make
Some cmake options:
- "
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo": - "
cmake -DBUILD_H264_TESTS=OFF": do not build the tests - "
cmake -DBUILD_CLANG_FUZZER=OFF": do not build the fuzzing tests
Feel free to test all the unittests:
$ make test
Running tests...
Test project ...h264nal/build
Start 1: h264_common_unittest
1/23 Test #1: h264_common_unittest ......................................... Passed 0.02 sec
...
Start 23: h264_nal_unit_parser_unittest
23/23 Test #23: h264_nal_unit_parser_unittest ................................ Passed 0.02 sec
100% tests passed, 0 tests failed out of 23
Total Test time (real) = 0.44 sec
Or to test any of the unittests:
$ ./test/h264_bitstream_parser_unittest
Running main() from /builddir/build/BUILD/googletest-release-1.10.0/googletest/src/gtest_main.cc
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from H264BitstreamParserTest
[ RUN ] H264BitstreamParserTest.TestSampleBitstream601
[ OK ] H264BitstreamParserTest.TestSampleBitstream601 (0 ms)
[ RUN ] H264BitstreamParserTest.TestSampleBitstream601Alt
[ OK ] H264BitstreamParserTest.TestSampleBitstream601Alt (0 ms)
[----------] 2 tests from H264BitstreamParserTest (0 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (0 ms total)
[ PASSED ] 2 tests.
Check the included vector file:
$ ./tools/h264nal --add-offset --add-length --add-parsed-length ../media/601.264
h264nal: original version
nal_unit { offset: 0x00000004 length: 24 parsed_length: 0x00000016 nal_unit_header { forbidden_zero_bit: 0 nal_ref_idc: 3 nal_unit_type: 7 } nal_unit_payload { sps { profile_idc: 66 constraint_set0_flag: 1 constraint_set1_flag: 1 constraint_set2_flag: 0 constraint_set3_flag: 0 constraint_set4_flag: 0 constraint_set5_flag: 0 reserved_zero_2bits: 0 level_idc: 22 seq_parameter_set_id: 0 log2_max_frame_num_minus4: 1 pic_order_cnt_type: 2 max_num_ref_frames: 16 gaps_in_frame_num_value_allowed_flag: 0 pic_width_in_mbs_minus1: 19 pic_height_in_map_units_minus1: 14 frame_mbs_only_flag: 1 direct_8x8_inference_flag: 1 frame_cropping_flag: 0 vui_parameters_present_flag: 1 vui_parameters { aspect_ratio_info_present_flag: 0 overscan_info_present_flag: 0 video_signal_type_present_flag: 1 video_format: 5 video_full_range_flag: 1 colour_description_present_flag: 0 chroma_loc_info_present_flag: 0 timing_info_present_flag: 1 num_units_in_tick: 1 time_scale: 50 fixed_frame_rate_flag: 0 nal_hrd_parameters_present_flag: 0 vcl_hrd_parameters_present_flag: 0 pic_struct_present_flag: 0 bitstream_restriction_flag: 1 motion_vectors_over_pic_boundaries_flag: 1 max_bytes_per_pic_denom: 0 max_bits_per_mb_denom: 0 log2_max_mv_length_horizontal: 10 log2_max_mv_length_vertical: 10 max_num_reorder_frames: 0 max_dec_frame_buffering: 16 } } } }
nal_unit { offset: 0x00000020 length: 6 parsed_length: 0x00000006 nal_unit_header { forbidden_zero_bit: 0 nal_ref_idc: 3 nal_unit_type: 8 } nal_unit_payload { pps { pic_parameter_set_id: 0 seq_parameter_set_id: 0 entropy_coding_mode_flag: 0 bottom_field_pic_order_in_frame_present_flag: 0 num_slice_groups_minus1: 0 num_ref_idx_l0_active_minus1: 15 num_ref_idx_l1_active_minus1: 0 weighted_pred_flag: 0 weighted_bipred_idc: 0 pic_init_qp_minus26: -8 pic_init_qs_minus26: 0 chroma_qp_index_offset: -2 deblocking_filter_control_present_flag: 1 constrained_intra_pred_flag: 0 redundant_pic_cnt_present_flag: 0 transform_8x8_mode_flag: 0 pic_scaling_matrix_present_flag: 0 pic_scaling_list_present_flag { } second_chroma_qp_index_offset: 0 } } }
nal_unit { offset: 0x00000029 length: 628 parsed_length: 0x00000001 nal_unit_header { forbidden_zero_bit: 0 nal_ref_idc: 0 nal_unit_type: 6 } nal_unit_payload { } }
nal_unit { offset: 0x000002a0 length: 244 parsed_length: 0x00000005 nal_unit_header { forbidden_zero_bit: 0 nal_ref_idc: 3 nal_unit_type: 5 } nal_unit_payload { slice_layer_without_partitioning_rbsp { slice_header { first_mb_in_slice: 0 slice_type: 7 pic_parameter_set_id: 0 frame_num: 0 idr_pic_id: 0 ref_pic_list_modification { ref_pic_list_modification_flag_l0: 0 ref_pic_list_modification_flag_l1: 0 } dec_ref_pic_marking { no_output_of_prior_pics_flag: 0 long_term_reference_flag: 0 } slice_qp_delta: -12 disable_deblocking_filter_idc: 0 slice_alpha_c0_offset_div2: 0 slice_beta_offset_div2: 0 } } } }
...
Parse all the NAL units of an Annex B (.264 extension) file.
$ ./tools/h264nal file.264 --noas-one-line --add-length --add-offset --add-parsed-length
h264nal: original version
nal_unit {
offset: 0x00000004
length: 24
parsed_length: 0x00000016
nal_unit_header {
forbidden_zero_bit: 0
nal_ref_idc: 3
nal_unit_type: 7
}
nal_unit_payload {
sps {
profile_idc: 66
constraint_set0_flag: 1
constraint_set1_flag: 1
constraint_set2_flag: 0
constraint_set3_flag: 0
constraint_set4_flag: 0
constraint_set5_flag: 0
reserved_zero_2bits: 0
level_idc: 22
seq_parameter_set_id: 0
log2_max_frame_num_minus4: 1
pic_order_cnt_type: 2
...
There are 3 ways to integrate the parser in your C++ parser:
If you just have a binary blob with the full contents of a file in Annex B
format, use the H264BitstreamParser::ParseBitstream() method. This case
is useful, for example, when you have an Annex B format file (a file with
.264 or .h264 extension). You read the whole file in memory, and then
convert the read blob into a set of parsed NAL units.
The following code has been copied from tools/h264nal.cc:
// read your .264 file into the vector `buffer`
std::vector<uint8_t> buffer(size);
// create bitstream parser from the file
h264nal::ParsingOptions parsing_options;
std::unique_ptr<h264nal::H264BitstreamParser::BitstreamState> bitstream =
h264nal::H264BitstreamParser::ParseBitstream(
buffer.data(), buffer.size(), parsing_options);
The H264BitstreamParser::ParseBitstream() function receives a generic
binary string (data and length) that you read from the file, plus
some options (whether to add options, length, and parsed length to each
NAL units).
It then:
- (1) splits the input string into a vector of NAL units, and
- (2) parses the NAL units, and add them to the vector
If you have a series of binary blobs with NAL units, use the
H264NalUnitParser::ParseNalUnit() method. This case is useful for example
if you have a producer of NAL units (e.g. an encoder), and you want to
parse them as soon as they are produced.
The following code has been copied from tools/h264nal.cc:
// 2. get the indices for the NALUs in the stream. This is needed
// because we will read Annex-B files, i.e., a bunch of appended NALUs
// with escape sequences used to separate them.
auto nalu_indices =
h264nal::H264BitstreamParser::FindNaluIndices(data, length);
// 3. create state for parsing NALUs
// bitstream parser state (to keep the SPS/PPS/SubsetSPS NALUs)
h264nal::H264BitstreamParserState bitstream_parser_state;
h264nal::ParsingOptions parsing_options;
// 4. parse the NALUs one-by-one
auto bitstream =
std::make_unique<h264nal::H264BitstreamParser::BitstreamState>();
for (const auto &nalu_index : nalu_indices) {
// 4.1. parse 1 NAL unit
// note: If the NALU comes from an unescaped bitstreams, i.e.,
// one with an explicit NALU length mechanism (like mp4 mdat
// boxes), the right function is `ParseNalUnitUnescaped()`.
auto nal_unit = h264nal::H264NalUnitParser::ParseNalUnit(
&data[nalu_index.payload_start_offset], nalu_index.payload_size,
&bitstream_parser_state, parsing_options);
...
}
The H264NalUnitParser::ParseNalUnit() function receives a generic
binary string (data and length) that contains a NAL unit, plus
a H264BitstreamParserState object that keeps all the SPS/PPS it
ever sees. It then parses the NAL unit, and returns it (including the
parsing offsets).
It will also update the input H264BitstreamParserState object if it
sees any PPS/SPS. This is important if the parsed NAL unit has state
that needs to be used to parse other NAL units (SPS, PPS): In that
case it will be stored into the BitstreamParserState object that is
passed around.
Note that H264NalUnitParser::ParseNalUnit() will only parse 1 NAL unit.
There are some producers that will instead produce multiple NAL units
in the output buffer. For example, an h264 encoder producing a key frame
may return 3 NAL units (PPS, SPS, and slice header).
If you want to just pass consecutive RTP packets (rfc6184 format), and get
information on their contents, use the H264RtpParser::ParseRtp method.
The following code is inspired from test/h264_rtp_parser_unittest.cc.
// keep a bitstream parser state (to keep the PPS/SPS NALUs)
H264BitstreamParserState bitstream_parser_state;
// parse packet(s)
std::unique_ptr<H264RtpParser::RtpState> rtp = H264RtpParser::ParseRtp(
buffer, arraysize(buffer),
&bitstream_parser_state);
// packets will return the actual contents into `rtp`, and update the
// bitstream parser state if the RTP packet contains a SPS/PPS.
// check the main packet contents
if (rtp->nal_unit_header->nal_unit_type <= 23) {
// a packet containing a single NAL Unit
// header := rtp->rtp_single.nal_unit_header
// payload := rtp->rtp_single.nal_unit_payload
} else if (rtp->nal_unit_header->nal_unit_type == RTP_STAPA_NUT) {
// a STAP-A (Aggregation Packet) packet contains 2+ NAL Units
// number_of_packets := rtp->rtp_ap.nal_unit_payloads.size()
// packet_i := rtp->rtp_ap.nal_unit_payloads[i]
} else if (rtp->nal_unit_header->nal_unit_type == RTP_FUA_NUT) {
// an FU-A (Fragmentation Unit) packet contains a piece of a NAL unit
// has_start_of_packet := rtp->rtp_fu.s_bit
// internal_type := rtp->rtp_fu.fu_type
// packet := rtp->rtp_fu.nal_unit_payload
}
// access to the SPS/PPS map
// e.g. bitstream_parser_state.sps[sps_id].pic_width_in_luma_samples
Requires gtest-devel, gmock-devel Requires llvm-tooset (or llvm-toolset-compiler-rt) for libfuzzer support
The media directory contains information on testing the parser using media files.
The rtc_common.h|cc code contains an RBSP parser copied from an old
version of webrtc.
The fuzz directory contains information on fuzzing the parser.
h264nal has been run against the JVT H.264 conformance suite (the 201602 release), which is 572 bitstreams across 9 folders. Results can be regenerated whenever the parser changes.
| suite | files | clean | unimplemented | invalid | silent |
|---|---|---|---|---|---|
| avcv1 | 135 | 135 | 0 | 0 | 0 |
| frext | 71 | 71 | 0 | 0 | 0 |
| professional_profiles | 38 | 38 | 0 | 0 | 0 |
| svc | 258 | 241 | 17 | 0 | 0 |
| 3davc | 12 | 0 | 12 | 0 | 0 |
| mvc | 27 | 0 | 27 | 0 | 0 |
| mvcd | 17 | 0 | 17 | 0 | 0 |
| mfc | 7 | 0 | 7 | 0 | 0 |
| mfcd | 7 | 0 | 7 | 0 | 0 |
| TOTAL | 572 | 485 | 87 | 0 | 0 |
The columns are the four outcomes h264nal can produce, which are worth keeping apart:
- clean: parsed in full, and every slice NAL unit yielded a slice header. Exit code 0.
- unimplemented: the bitstream needs syntax h264nal does not implement. It says which, on stderr, and exits 2. This is a deliberate refusal rather than a failure.
- invalid: h264nal rejected the bitstream. Exit code 1. On this suite that would mean a bug, since every file in it is conforming.
- silent: exited 0 but produced fewer slice headers than there were slice NAL units, so something was dropped without being reported. The CSV tracks this in its own column because such a file looks clean otherwise.
There are no crashes, no invalid verdicts and no silent shortfalls. All 87 unimplemented files are asking for annexes that are not written:
| missing syntax | files |
|---|---|
| nal_unit_header_mvc_extension() | 56 |
| nal_unit_header_3davc_extension(), plus the above | 14 |
| dec_ref_base_pic_marking() | 17 |
which is Annex G (MVC), Annex I (3D-AVC) and part of Annex F (SVC). That accounts for the 5 folders at zero clean: they are entirely MVC, 3D-AVC or MFC content. See section 9 for what is not supported.
$ mkdir conformance
$ cd conformance
$ make # every suite
$ make conformance.avcv1.csv # just one
$ make DATASET=/path/to/201602 # a copy elsewhere
$ make -B # rebuild everything
$ make clean
The dataset is not in this repository. DATASET defaults to
$(HOME)/work/video/dataset/h264_conformance/201602 and has to point at a
directory holding the 9 suite folders.
Build the binary without the fuzzing sanitizers, which is the default:
$ mkdir build && cd build && cmake .. && make
A build configured with -DBUILD_CLANG_FUZZER=ON is not usable here. The
UBSan checks it turns on report libstdc++'s own reference counting on
every shared_ptr release, and the script reads stderr as parser output,
so every file comes back changed. Pass H264NAL= to use a binary from
somewhere other than ../build/tools/h264nal.
Underneath, the Makefile runs tools/h264nal-conformance.py, which can also be used on its own for any set of files:
$ ./tools/h264nal-conformance.py -o out.csv /path/to/*.264
It writes one row per file with the resolution, profile, level, NAL unit counts, and the parser status, and prints a summary to stderr. A full run over the 572 files takes about 10 seconds.
List of tasks:
- add lacking parsers (e.g. SEI)
- remove TODO entries from the code
- move headers to separate
include/file to allow easier programmatic integration - add a set of Annex B files for testing
- no support for STAP-B, MTAP16, MTAP24, or FU-B RTP packetization.
h264nal is BSD licensed, as found in the LICENSE file.
If you want to build with clang, then you need to specify the c/cpp compilers:
$ CC=clang CXX=clang++ cmake ..
If you want to build with gcc, then you need to specify the c/cpp compilers and disable clang's fuzzer sanitizer:
$ CC=gcc CXX=g++ cmake -DBUILD_CLANG_FUZZER=OFF ..
If you want to build in debug mode, you need to add some variables:
$ cmake -DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_C_FLAGS_DEBUG="-g -O0" -DCMAKE_CXX_FLAGS_DEBUG="-g -O0" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
- install gtests (see here)
$ brew install googletest
- install llvm (see here)
$ brew install llvm
$ brew install clang-format
$ ln -s "$(brew --prefix llvm)/bin/clang-format" "/usr/local/bin/clang-format"
$ ln -s "$(brew --prefix llvm)/bin/clang-tidy" "/usr/local/bin/clang-tidy"
$ ln -s "$(brew --prefix llvm)/bin/clang-apply-replacements" "/usr/local/bin/clang-apply-replacements"