Silent truncation of varchar data type in freetds 1.4.26 and higher

[dishdj-2606]

Hi Team,

Could you please assist us with the following query?

We are in the process of migrating our application from FreeTDS version 0.91 on RHEL7 to FreeTDS version 1.4.26 on RHEL8.
During our testing of loading the data using function bcp_exec defined in bcp.c, we encountered a situation where if the data in the input file exceeds the size of the corresponding table column, FreeTDS 1.4.26 truncates the data and loads it into the column without generating any error or warning.
In contrast, with FreeTDS 0.91 on RHEL7, we received the below database error in the same scenario.

SYBASE ERROR #20049 (severity 4 oserrno 0)

Data conversion resulted in overflow
operating system message: SuccessFreeTDS: db-lib: exiting because client error handler returned INT_EXIT for msgno 20049

Could you please advise on how we can configure FreeTDS 1.4.26 to produce an error message similar to the one we received with the previous version?

Please find attached debugging logs.

Could you please look into this on priority.

Logs are attached here.

freetds_rh8.log

[freddy77]

That does not sound really great indeed. There should be no configuration for this. The only thing that could make a difference is the callback for error handling.
Can you post some code snippet?

[dishdj-2606]

Hi ,

I am not a C programmer, but I would like to explain the flow we have for bulk loading data

We have a Perl script that invokes the C binary freetdsbulkcopycall, which is built from the sybasbulkcopycall C code which invokes freetds bcp_exec.
Below is the relevant code snippet:

if (bcp_exec(dbproc, &rows) == FAIL) {
printf("bcp_exec returned FAIL;\n");
printf(" number of rows read into table: %ld\n", (long)rows);
exit(EXIT_FAILURE);
} else {
printf(" successful write of %ld rows into table\n", (long)rows);
}

In this flow, the entire row of data is passed as a parameter to bcp_exec. When it processes the host file in the _bcp_read_hostfile function, it convert column for input to a table using _bcp_convert_in function, as shown in the following snippet from FreeTDS version 1.4.26:

rc = _bcp_convert_in(dbproc, hostcol->datatype, (const TDS_CHAR*) coldata, collen,
desttype, bcpcol->bcp_column_data);

The _bcp_convert_in function calls tds_convert from convert.c, which then calls tds_convert_char. This function truncates the data if the size exceeds the column size. Here’s a relevant excerpt from the code:

case TDS_CONVERT_CHAR:
memcpy(cr->cc.c, src, srclen < cr->cc.len ? srclen : cr->cc.len);
return srclen;

case CASE_ALL_CHAR:
cr->c = tds_new(TDS_CHAR, srclen + 1);
test_alloc(cr->c);
memcpy(cr->c, src, srclen);
cr->c[srclen] = 0;
return srclen;
break;

In contrast, in the older version of FreeTDS (0.91), the flow reads the host file and calls dbconvert to verify the data against the column size:

• FIXME bcpcol->bcp_column_data->data && bcpcol->column_size ??
• It seems a buffer overflow waiting...
  */
  bcpcol->bcp_column_data->datalen = dbconvert(dbproc, hostcol->datatype, (const BYTE *) coldata, collen, desttype,
  bcpcol->bcp_column_data->data, bcpcol->column_size);

This highlights a significant difference in how data conversion and size checks are handled between the two versions.

I have previously shared the logs from loading data using FreeTDS version 1.4.26. Attached are the logs for the data loading process with FreeTDS version 0.91

freetds_rh7.log

Hope this information is helpful for proceeding with further actions.

Regards,
Disha

[freddy77]

It looks like the logs for RH8 are truncated.

[dishdj-2606]

Yes, the logs freetds_rh8.log are generated through freetds version 1.4.26 and freetds_rh7.log are generated using freetds version 0.9
Is this the default behavior of bcp to silently truncate the data in freetds higher version like 1.4.26?
What would be the solution to display the error instead of silent truncation of records?

[freddy77]

Yes, the logs freetds_rh8.log are generated through freetds version 1.4.26 and freetds_rh7.log are generated using freetds version 0.9 Is this the default behavior of bcp to silently truncate the data in freetds higher version like 1.4.26? What would be the solution to display the error instead of silent truncation of records?

No, I said that the rh8 logs are truncated, that is freetds_rh8.log file

[dishdj-2606]

Hi,
We apologize for inconvenience this may have caused.
Please find attached complete logs

freetds_rh8_1.log