feat: Migrate to bitbucket

[sigalikanevsky]

No description provided.

[gemini-code-assist]

Code Review

This pull request introduces support for column-specific character set decoding, a heartbeat mechanism to maintain connection liveness, and configurable connection retry intervals. However, several critical issues must be addressed: the table charset retrieval logic breaks when regular expressions are used in table filters, and calling it before filter initialization causes highly inefficient database queries. Additionally, a potential out-of-bounds panic exists when slicing hostname runes using byte lengths. Finally, several data corruption risks were identified, including the unintended sanitization of formatting characters (newlines/tabs) in text columns, the forced string conversion of binary BLOB data, and the mutation of user data via smart quote normalization. A test query mismatch in TestGenerateCharsetQuery also needs to be corrected.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

GetColumnsCharsets attempts to split tableRegex by . and expects exactly 2 parts, both of which must be safe identifiers. However, IncludeTableRegex contains regular expressions (e.g., .*\\.canal_test), not literal table names. Splitting a regex by . will produce incorrect parts, and regex characters like * or \\ will fail the isSafeIdentifier check. This completely breaks NewCanal initialization whenever any standard regex is used in IncludeTableRegex.

Additionally, storing the column charsets in c.cfg.ColumnCharset using the tableRegex string as the key (e.g., ".*\\.canal_test") will cause lookups in replication/row_event.go to fail, because the lookup is performed using the actual table name (e.g., "test.canal_test").

To fix both issues, query all tables from information_schema.TABLES first, filter them using the already compiled c.checkTableMatch(key) helper, and then query the column charsets only for the matching tables using the actual db.table name as the map key.

	// 1. Query all tables from information_schema
	rows, err := db.QueryContext(c.ctx, "SELECT TABLE_SCHEMA, TABLE_NAME FROM information_schema.TABLES")
	if err != nil {
		return fmt.Errorf("failed to query tables: %w", err)
	}
	defer rows.Close()

	var matchingTables []string
	for rows.Next() {
		var dbName, tableName string
		if err := rows.Scan(&dbName, &tableName); err != nil {
			return fmt.Errorf("failed to scan table: %w", err)
		}
		key := fmt.Sprintf("%s.%s", dbName, tableName)
		if c.checkTableMatch(key) {
			matchingTables = append(matchingTables, key)
		}
	}
	if err := rows.Err(); err != nil {
		return fmt.Errorf("error reading tables: %w", err)
	}
	rows.Close()

	// 2. Query column charsets for each matching table
	query, err := c.GenerateCharsetQuery()
	if err != nil {
		return fmt.Errorf("failed to generate charset query: %w", err)
	}

	for _, tableKey := range matchingTables {
		parts := strings.Split(tableKey, ".")
		dbName, tableName := parts[0], parts[1]

		c.cfg.ColumnCharset[tableKey] = make(map[int]string)
		colRows, err := db.QueryContext(c.ctx, query, dbName, tableName)
		if err != nil {
			return fmt.Errorf("failed to query column charsets for %s: %w", tableKey, err)
		}
		
		for colRows.Next() {
			var ordinal int
			var charset, columnName sql.NullString
			if err := colRows.Scan(&ordinal, &charset, &columnName); err != nil {
				colRows.Close()
				return fmt.Errorf("failed to scan column charset for %s: %w", tableKey, err)
			}
			charsetValue := "utf8mb4"
			if charset.Valid && charset.String != "" {
				charsetValue = charset.String
			}
			c.cfg.ColumnCharset[tableKey][ordinal] = charsetValue
		}
		colRows.Close()
	}

[gemini-code-assist]

The code checks len(hostname) > maxHostNameLength (which checks the number of bytes), but then slices the runes slice up to maxHostNameLength. If hostname contains multi-byte UTF-8 characters, the number of runes will be less than the number of bytes. If len(hostname) > 60 but len(runes) < 60, slicing runes[0:60] will cause a runtime out-of-bounds panic. Convert to runes first, and then check the length of the runes slice before slicing.

Suggested change

	if len(hostname) > maxHostNameLength {
	runes := []rune(hostname)
	hostname = string(runes[0:maxHostNameLength])
	}
	runes := []rune(hostname)
	if len(runes) > maxHostNameLength {
	hostname = string(runes[0:maxHostNameLength])
	}

[gemini-code-assist]

sanitizeNonPrintable replaces all characters less than 0x20 with spaces. This includes standard formatting characters like newlines (\n / 0x0A), carriage returns (\r / 0x0D), and tabs (\t / 0x09). Applying this to all decoded strings will silently corrupt the data of all VARCHAR, CHAR, TEXT, and JSON columns by replacing all newlines and tabs with spaces. A replication tool must preserve the exact content of the database. Please remove sanitizeNonPrintable entirely.

func decodeStringByCharSet(data []byte, charset string, length int) (v string, n int) {
	enc, err := getDecoderByCharsetName(charset)
	if err != nil {
		log.Errorf(err.Error())
		return decodeString(data, length)
	}
	if enc == nil {
		log.Warnf("Falling back to default decoding for charset: %s", charset)
		return decodeString(data, length)
	}
	return decodeStringWithEncoder(data, length, enc)
}

[gemini-code-assist]

In NewCanal, c.GetColumnsCharsets() is called before the table filters (c.includeTableRegex, c.excludeTableRegex, and c.tableMatchCache) are initialized. Because c.tableMatchCache is nil at this point, c.checkTableMatch() will always return true for every table in the database. This causes GetColumnsCharsets to fetch column charsets for all tables in the database, which is extremely inefficient. Please move the table filter initialization block above the GetColumnsCharsets() call.

[gemini-code-assist]

For MYSQL_TYPE_BLOB, the code automatically calls convertToString(v) which converts the []byte to a string and sanitizes it. However, BLOB columns are designed to store arbitrary binary data (e.g., images, encrypted data). Converting binary data to a UTF-8 string and replacing non-printable bytes with spaces will completely corrupt any binary data stored in BLOB columns. Only convert MYSQL_TYPE_BLOB to string if the column is actually a text column (i.e., its charset is not "binary").

Suggested change

	case MYSQL_TYPE_BLOB:
	v, n, err = decodeBlob(data, meta)
	newValue, ok := convertToString(v)
	if ok {
	v = newValue
	}
	case MYSQL_TYPE_BLOB:
	v, n, err = decodeBlob(data, meta)
	if charset != "binary" {
	newValue, ok := convertToString(v)
	if ok {
	v = newValue
	}
	}

[gemini-code-assist]

normalizeSmartQuotes replaces smart quotes (‘, ’, “, ”) with standard ASCII quotes (', "). A database replication tool should be a faithful, lossless replica of the source database and should not mutate user data. Please avoid mutating user data; remove normalizeSmartQuotes or make it strictly optional/configurable.

[gemini-code-assist]

The expected query in TestGenerateCharsetQuery does not match the actual query returned by GenerateCharsetQuery() in canal.go (which contains COALESCE and ELSE clauses). This mismatch will cause the test to fail. Please update the expected query to match the actual implementation.

	expected := "\n\t	SELECT \n\t\t   c.ORDINAL_POSITION,\n\t\t   COALESCE(\n\t\t\t  CASE \n\t\t\t\t WHEN c.CHARACTER_SET_NAME IS NOT NULL THEN c.CHARACTER_SET_NAME\n\t\t\t\t WHEN c.DATA_TYPE IN ('binary','varbinary','tinyblob','blob','mediumblob','longblob') THEN col.CHARACTER_SET_NAME\n\t\t\t\t ELSE col.CHARACTER_SET_NAME\n\t\t\t  END,\n\t\t\t  'utf8mb4'\n\t\t   ) AS CHARACTER_SET_NAME,\n\t\t   c.COLUMN_NAME\n\t\tFROM \n\t\t   information_schema.COLUMNS c\n\t\tLEFT JOIN information_schema.TABLES t\n\t\t   ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME\n\t\tLEFT JOIN information_schema.COLLATIONS col\n\t\t   ON col.COLLATION_NAME = t.TABLE_COLLATION\n\t\tWHERE \n\t\t   c.TABLE_SCHEMA = ?\n\t\t   AND c.TABLE_NAME = ?\n\t\t   AND (c.CHARACTER_SET_NAME IS NOT NULL \n\t\t\t\tOR c.DATA_TYPE IN ('binary','varbinary','tinyblob','blob','mediumblob','longblob')\n\t\t\t\tOR c.DATA_TYPE IN ('varchar','char','text','tinytext','mediumtext','longtext'));\n\t\t"

[lance6716]

hi @sigalikanevsky did you wrongly create this PR? Seems it's your highly customized fork

[dveeden]

hi @sigalikanevsky did you wrongly create this PR? Seems it's your highly customized fork

Yes, this PR also doesn't seem to make sense to me.

For the PR title: We could setup a mirror on bitbucket and/or any other git hosting as long as it doesn't interfere too much with the current workflow.