Merge pull request #176 from openitu/itu_submission_april_2024

ITU-T submission - April 2024 received consent.

Author: ludomal

doc/draft_g191.docx

@@ -12,17 +12,31 @@ Please refer to the official Recommendation ITU-T G.191 for the complete text: [
 
 Recommendation ITU-T G.191 provides source code for speech and audio processing modules for narrowband, wideband and super-wideband telephony applications. The set includes codecs, filters, noise generators.
 
-This edition introduces changes to Annex A, which describes the ITU-T Software Tools (STL) containing a high-quality, portable C code library for speech processing applications. This release of the STL, also known as STL2023, incorporates:
+This edition introduces changes to Annex A, which describes the ITU-T Software Tools (STL) containing a high-quality, portable C code library for speech processing applications. This release of the STL, also known as STL2024, incorporates:
 
-* An implementation of P.50 fullband MNRU as described in ITU-T P.810.
+* BS1770demo improvements:
 
-* A tool for automatic instrumentation of speech and audio codecs to measure their computational complexity and memory.
+    - a new RMS option disabling the gating function for background noise measurement, 
 
-Recommendation ITU-T G.191 includes an electronic attachment containing STL2023 and manual.
+    - the handling of the edge case where all gating blacks are below -70 LKFS,
+
+    - improved reporting with scaling factor being reported in the linear and log domains.
+
+* WMC Tool updates:
+
+    - New command line parameter to allow control on the number of frames per second (default still 50)
+
+    - Export information on all memory allocations occurring during runtime
+    
+    - Example script for graphical analysis and profiling of dymanic memory allocation
+
+    - Bugs fixes and code improvements
+
+Recommendation ITU-T G.191 includes an electronic attachment containing STL2024 and manual.
 
 ## Keywords
 
-DSP operators, filters, MNRU, P.50 FB MNRU, open source, reverb, STL2022, G.711, G.722, G.726, G.728, sv56, BS.1770, ESDRU, WMC Tool
+DSP operators, filters, MNRU, P.50 FB MNRU, open source, reverb, G.711, G.722, G.726, G.728, sv56, BS.1770, ESDRU, WMC Tool
 
 ## 1. Scope
 

doc/g191.md

@@ -18,7 +18,7 @@
 \addtolength{\itemsep}{-20pt}
 
 % Define headers
-\def\ugst_title{ ITU-T Software Tool Library, release 2023}
+\def\ugst_title{ ITU-T Software Tool Library, release 2024}
 \def\us{$\mu$s}
 \markboth{  \hspace{1cm} \hfill \ugst_title }%
          { Version: \today \hfill \hspace{1cm} }
@@ -61,7 +61,7 @@
 \pagenumbering{roman}
 
 %==============================================================================
-\title{ITU-T Software Tool Library 2023 User's Manual}
+\title{ITU-T Software Tool Library 2024 User's Manual}
 \author{ITU-T Users' Group on Software Tools}
 
 %------------------------------------------------------------------------------
@@ -71,7 +71,7 @@
 
 \ruley{100mm}
 
- Copyright \copyright~ 2005, 2006, 2009, 2019, 2022 and 2023 by the International
+ Copyright \copyright~ 2005, 2006, 2009, 2019, 2022, 2023 and 2024 by the International
  Telecommunication Union (ITU)
 
 \ruley{15mm}

doc/manual/STLmanual.tex

@@ -103,12 +103,18 @@ \section{Description of the Algorithm}
 \texttt{fabs( 1.0 - fac / last\_fac ) \textless RELATIVE\_DIFF}, where \texttt{RELATIVE\_DIFF = 0.0001} or when the maximum number of iterations \texttt{MAX\_ITERATIONS = 10}
 has been reached.
 
+When measuring the loudness of signals with a large dynamic range, it may be desirable to disable the gating function
+to obtain the level considering the full signal. This is useful e.g. for noise level adjustment in listening tests
+including added background noise. For this reason, the gating function may be disabled using an RMS (Root-Mean-Square)
+energy measurement option \texttt{-rms}. Note that this feature is not part of \cite{BS1770}.
+
 {\tt\small
 \begin{verbatim}
 double find_scaling_factor(            /* o: scaling factor                 */
     const double *gating_block_energy, /* i: gating_block_energy            */
     const long n_gating_blocks,        /* i: Number of gating blocks        */
     const double lev,                  /* i: Target level                   */
+    const short rms_flag,              /* i: Flag for RMS (no gating)       */    
           double *lev_input,           /* o: Input level                    */
           double *lev_obtained         /* o: Obtained level                 */
 )
@@ -158,6 +164,7 @@ \section{Usage of bs1770demo}
 Options:
 -nchan N          Number of channels [1..24] (Default: 1)
 -lev L            Target level LKFS (Default: -26)
+-rms              Disable gating (for background noise level measurement)
 -conf xxxx        Configuration string:
                       '1' ldspk pos within |elev| < 30 deg, 60 deg <= |azim| <= 120 deg
                       'L' LFE channel (weight zero)

doc/manual/bs1770demo.tex

@@ -29,6 +29,8 @@ \chapter{Introduction}
 
 In 2023, STL release incorporates an implementation of P.50 fullband MNRU as described in ITU-T Rec. P.810, as well as WMC tool, a tool for automatic instrumentation of speech and audio codecs to measure their computational complexity and memory.
 
+In 2024, the revision of the STL offers additional features and bug fixes to the BS.1770 demo and WMC tool.
+
 Since STL2019, the build toolchain uses CMake to generate platform-dependent and tool-dependent build scripts as well as to execute regression tests for each module in the STL.
 Modules have been tested on Windows, MacOS and several Linux flavors.
 
@@ -191,6 +193,8 @@ \section{Acknowledgements}
 
 For the STL2023, the implementation of the P.50 Fullband MNRU was kindly provided by Mrs Anna Llagostera and Mr Jens Berger from SwissQual AG/Rohde \& Schwarz, with a contribution from NTT on filter routines, the integration into the existing MNRU code and documentation from Ludovic Malfait, and the DC filter and amplitude clipping improvement suggestions from Anssi Ramo (Nokia). The implementation of the WMC tool was kindly provided by Mr Guy Richard and Mr Vladimir Malenovsky from VoiceAge Corporation.
 
+For the STL2024, the improvements to BS.1770 demo and WMC Tool were kindly provided by Mr Erik Norvell from Ericsson and Mr Vladimir Malenovsky from VoiceAge Corporation respectively.
+
 Above all, special thank goes to ITU-T SG16 Counselor Mr~Sim\~ao Ferraz de Campos Neto, the ``father'' of the STL.
 
 %=============================================================================
@@ -230,6 +234,8 @@ \chapter{Tutorial}
 \item[STL2009] ITU-T Software Tools Library, release 2009.
 \item[STL2019] ITU-T Software Tools Library, release 2019.
 \item[STL2022] ITU-T Software Tools Library, release 2022.
+\item[STL2023] ITU-T Software Tools Library, release 2023.
+\item[STL2024] ITU-T Software Tools Library, release 2024.
 \item[UGST]  Users' Group on Software Tools, of ITU-T Study Group 16.
 \end{Descr}
 

doc/manual/intro.tex

@@ -56,8 +56,11 @@ \subsection{Usage}
         note: filename shall point to a .c file containing the print_mem() function
     -b [--no-backup]: no backup of original files
     -c dirname [--generate-wmc-files dirname]: copy wmc_auto.h and wmc_auto.c to a user-specified directory
+    -f value [--frames-per-second value]: set the number of frames per second (default 50.0)
 \end{Verbatim}
 
+The optional command-line parameters may be used to control either the instrumentation or the couting process. With the \verb|-i| optional argument the WMC tool prints the list of functions that are instrumented. It doesn't run the WMC tool. The parameter \verb|-d| runs the des-instrumentation phase only. The des-instrumentation phase removes all meta-information that has been previously added to the source code with the WMC tool. When \verb|-m| argument is provided the WMC tool adds code calculating ROM and RAM memory consumption and fills the \verb|print_mem()| function to print the statistics about memory usage. When \verb|-m| is provided the user needs to specify the name of the \verb|.c| file containing the \verb|print_mem()| function. The parameter \verb|-b| instructs the WMC tool not to create backup copies of the instrumented \verb|.c| files. The parameter \verb|-c| copies the control files \verb|wmc_auto.h| and \verb|wmc_auto.c| to a user-specified directory. The control files must be added to the project for calculating the complexity and memory consumption of all instrumented source files. The \verb|-f| parameter allows the users to override the default values for the number of frames per second. This value is important for the correct calculation of complexity in "WMOPS".
+
 \subsection{Instrumentation process}
 \label{ch:instrumentation_process}
 
@@ -107,7 +110,9 @@ \subsubsection{Warnings}
 \section{Instrumentation of complexity}
 \label{ch:instrumentation_of_complexity}
 
-The WMC tool only instruments functions but not macros. System functions and functions with names that are considered as basic operations are skipped. The instrumentation mechanism has been designed to be as least intrusive as possible. The instrumentation code is inserted at the beginning of each instrumented line in the source code with a single macro that has the following form \verb|$("ops")| where \verb|ops| is a string of letters and symbols indicating individual operations. The WMC tool parses each source file and calculates the length of each instrumentation string. The maximum length is then used to indent the entire source file to make space for the instrumentation strings. This is shown in the example below.
+The instrumentation mechanism has been designed to minimize intrusiveness. It skips system functions and functions labeled as basic operations. When instrumenting \verb|.c| source files, the WMC tool omits macros by default. However, it detects macros defined within the same \verb|.c| source file using the \verb|#define| pragma. Macros defined in external header files and included using \verb|#include| pragma cannot be directly detected. To address this, the instrumentation mechanism employs a heuristic: if a string contains more than 70\% capital letters, it's treated as a macro and skipped during instrumentation.
+
+The instrumentation code is integrated at the start of every instrumented line in the source code using a singular macro format: \verb|$("ops")|, where \verb|ops| represents a string encompassing individual operations denoted by letters and symbols. Upon parsing each source file, the WMC tool computes the length of each instrumentation string. Subsequently, the maximum length determines the indentation of the entire source file to accommodate the instrumentation strings, as illustrated in the example below.
 
 \begin{Verbatim}[fontsize=\small]
     float calc_shift_value( const float totalStep )
@@ -125,13 +130,13 @@ \section{Instrumentation of complexity}
     }
 \end{Verbatim}
 
-It is always possible to remove the instrumentation from the source code with the \verb|-d| command line option. In many cases, the source file will then return to its original non-instrumented state except for whitespace characters and text alignment. The un-instrumentation process is also invoked automatically by the WMC tool as the first step in the instrumentation process. This ensures that the results are identical even when the process is repeated multiple times. This allows the users to make modifications to an already instrumented source code by simply re-instrumenting it again. Note, that the un-instrumentation process does not remove the functions \verb|push_wmops()| and \verb|pop_wmops()| and the system functions. Lines that are preceded by
+The removal of instrumentation from the source code can be achieved using the \verb|-d| command line option. In many instances, executing this option restores the source file to its original non-instrumented state, with the exception of whitespace characters and text alignment. The un-instrumentation process is automatically triggered by the WMC tool as the initial step in the instrumentation procedure. This ensures consistent results even with repeated executions. Consequently, users can modify previously instrumented source code by simply re-instrumenting it. It's important to note that the un-instrumentation process does not eliminate the \verb|push_wmops()| and \verb|pop_wmops()| functions, nor does it remove system functions. Lines preceded by
 
 {
 {\em /*AddedByWMC\_Tool*/}
 }
 
-are removed completely. At the end of the un-instrumentation process, the original indentation is restored.
+are removed completely. At the end of the un-instrumentation process the original indentation is restored.
 
 If, for any reason, it is necessary to avoid the automatic instrumentation and un-instrumentation of certain parts in the source code, the user may encapsulate such parts in \verb|#define WMC_TOOL_SKIP| \ldots \verb|#undef WMC_TOOL_SKIP| macro pairs. For example
 
@@ -224,18 +229,21 @@ \subsection{Mathematic functions}
 \verb|sign()| & 1 \\
 \verb|round()| & 0 (not supported by the WMC tool) \\
 \verb|sqrt()| and \verb|inv_sqrt()| & 10 \\
-\verb|log()|, \verb|log2()| and thier derivatives & 25 \\
+\verb|log()|, \verb|log2()| and other logarithmic functions & 25 \\
 \verb|exp()| and \verb|pow()| & 25 \\
 \verb|sin()|, \verb|cos()| and all other trigonometric functions & 25 \\
+\verb|frexp()| & 2 \\
+\verb|fmod()| & 18 \\
 \hline
 \end{tabular}
 \label{tab:cost_of_math_functions}
 \end{table}
 
+
 %-----------------------------------------------------------------------------
 \subsection{User-defined functions}
 
-All user-defined functions in the source code are instrumented with one or more underscore '\verb|_|' symbols appended to their names. The WMC tool will count the number of arguments from the function call and insert a wrapper macro at the top of the source file, after the first \verb|#include| section. See the example below.
+All user-defined functions in the source code are instrumented with one or more underscore '\verb|_|' symbols appended to their names. The WMC tool counts the number of arguments in function calls and inserts a wrapper macro at the top of the source file, after the first \verb|#include| section. See the example below.
 
 \begin{Verbatim}[fontsize=\small]
 #include <stdlib.h>
@@ -266,6 +274,49 @@ \subsection{User-defined functions}
 
 The WMC tool appends an additional underscore symbol '\verb|_|' in each variant of the user-defined function call. This means that if a function is defined to accept a variable number of arguments the first function call will have one underscore symbol '\verb|_|' appended to its name, the second function call will have two underscore symbols '\verb|__|' appended to its name and so on. The WMC tool supports up to 10 variants (different number of arguments) of the same function.
 
+The WMC tool may encounter difficulties in properly instrumenting one-line functions that are defined as follows
+
+\begin{Verbatim}[fontsize=\small]
+    float calculate_vector_norm( float vec[], int_16t size ) { return sum2_f(vec, size); }
+\end{Verbatim}
+
+It's advisable to define the function body on separate lines, like this
+
+\begin{Verbatim}[fontsize=\small]
+    float calculate_vector_norm( float vec[], int_16t size ) 
+    {
+        return sum2_f(vec, size); 
+    }
+\end{Verbatim}
+
+In a local function, all declared variables should be positioned at the start of the function body. Additionally, the initialization of local variables should be segregated from their declaration. For instance, consider the following code snippet, where the WMC tool inserts \verb|$("M")| before \verb|int ch = 0;|
+
+\begin{Verbatim}[fontsize=\small]
+    int free_structs(MY_STRUCT_INFO *ptr_dec)
+    {   
+        if (ptr_dec == NULL) return -1;
+    
+$("M")  int ch = 0;
+
+        for (ch = 0; ch < n_channels; ch++) 
+        {
+            struct_free(&decoder->cfg[ch]->fft_buf);
+            struct_free(&decoder->cfg[ch]->ifft_buf);
+        }
+
+        return 0;
+    }
+\end{Verbatim}
+
+Compiling the instrumented code provided above results in the following error (gcc):
+
+\begin{Verbatim}[fontsize=\small]
+lib_thj/struct_f.c:373:17: error: expected expression before ‘int’
+  125 | $("M")          int ch = 0;
+      |                 ^~~
+\end{Verbatim}
+
+
 %-----------------------------------------------------------------------------
 \subsection{Operators}
 
@@ -361,7 +412,7 @@ \subsection{Operators}
 %-----------------------------------------------------------------------------
 \subsection{Manual instrumentation}
 
-It was already shown in the previous section in this document that it is possible to avoid the automatic instrumentation by encapsulating parts of the source code with
+As demonstrated in the preceding section of this document, it's possible to bypass automatic instrumentation by encapsulating sections of the source code with
 
 \begin{Verbatim}[fontsize=\small]
 #define WMC_TOOL_SKIP
@@ -435,7 +486,7 @@ \subsection{Complexity of functions and functional blocks}
 }
 \end{Verbatim}
 
-The string inside each \verb|push_wmops()| macro shall be unique in the entire project and reflect what is being measured. It does not need to be the same as the name of the function in which it is used. The macro \verb|pop_wmops()| must \textbf{ALWAYS} be used to terminate the complexity counters associated with each \verb|push_wmops()| macro. The WMC tool supports nesting of these macro pairs. Note, that the \verb|push_wmops()| and the \verb|pop_wmops()| macros do not need to be encapsulated within the \verb|#define WMC_TOOL_SKIP| \ldots \verb|#undef WMC_TOOL_SKIP| macro pairs.
+The \verb|push_wmops()| macro must be placed after the declaration of local variables. The string inside each \verb|push_wmops()| macro shall be unique in the entire project and reflect what is being measured. It does not need to be the same as the name of the function in which it is used. The macro \verb|pop_wmops()| must \textbf{ALWAYS} be used to terminate the complexity counters associated with each \verb|push_wmops()| macro. The WMC tool supports nesting of these macro pairs. Note, that the \verb|push_wmops()| and the \verb|pop_wmops()| macros do not need to be encapsulated within the \verb|#define WMC_TOOL_SKIP| \ldots \verb|#undef WMC_TOOL_SKIP| macro pairs.
 
 %-----------------------------------------------------------------------------
 \subsection{Printing statistics about computational complexity}

doc/manual/wmc_tool.tex

@@ -11,3 +11,9 @@ add_test(bs1770demo2-verify ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/signal-diff -equiv
 
 add_test(bs1770demo3 ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/bs1770demo -lev -16 -conf 11L000 test_data/sine_noise_test.pcm test_data/sine_noise_test.16LKFS.11L000.test.pcm)
 add_test(bs1770demo3-verify ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/signal-diff -equiv 159 -q test_data/sine_noise_test.16LKFS.11L000.test.pcm test_data/sine_noise_test.16LKFS.11L000.pcm)
+
+add_test(bs1770demo4 ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/bs1770demo -lev -26 test_data/sine_ramp.pcm test_data/sine_ramp.26LKFS.test.pcm)
+add_test(bs1770demo4-verify ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/signal-diff -equiv 115 -q test_data/sine_ramp.26LKFS.test.pcm test_data/sine_ramp.26LKFS.pcm)
+
+add_test(bs1770demo5 ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/bs1770demo -rms -lev -26 test_data/sine_ramp.pcm test_data/sine_ramp.26LKFSrms.test.pcm)
+add_test(bs1770demo5-verify ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/signal-diff -equiv 206 -q test_data/sine_ramp.26LKFSrms.test.pcm test_data/sine_ramp.26LKFSrms.pcm)