[Feature] Default Data Type

[tpcgold]

Hi there, it would be great to have the possibility to pass the data type for all fields to the deduce_schema function.
It's not that difficult to implement and could also just include STRING (as it makes the most sense)

e.g.:
generator.deduce_schema(input_data=data1, all_fields_as_string=True)

Would return RECORD (REPEATED or STRUCT), or STRING for all other values in the Schema

This is quite beneficial if unknown and repeatedly changing data is loaded and processed via a Staging area in BigQuery

[bxparks]

I don't think I understand your request. It sounds like this feature is provided by quoted_values_are_strings parameter in the constructor for SchemaGenerator. Maybe you can post some example data, and the schema that you expect from it.

[tpcgold]

Example:

from bigquery_schema_generator.generate_schema import SchemaGenerator
import sys
import json

test = {
    "a": "123",
    "b": 123,
    "c": 456,
    "values": [{
        "percentage": 3,
        "values": [{
            "a": "20220101",
            "b": 100,
        }]
    }]
}

generator = SchemaGenerator(input_format='dict', infer_mode='NULLABLE')

schema_map, error_logs = generator.deduce_schema(input_data=[test])
schema = generator.flatten_schema(schema_map)
json.dump(schema, sys.stdout, indent=2)

would result in:

[
  {
    "mode": "NULLABLE",
    "name": "a",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "b",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "c",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "mode": "NULLABLE",
        "name": "percentage",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "mode": "NULLABLE",
            "name": "a",
            "type": "STRING"
          },
          {
            "mode": "NULLABLE",
            "name": "b",
            "type": "STRING"
          }
        ],
        "mode": "REPEATED",
        "name": "values",
        "type": "RECORD"
      }
    ],
    "mode": "REPEATED",
    "name": "values",
    "type": "RECORD"
  }
]

this is quite practical as stated in the request

[bxparks]

I see, so assume every scalar field is a STRING even though the JSON type is an integer or something else.

I thought that bq load would barf on that schema, but it looks like it will happily ingest the data file and map every field to a string, even if the actual quantity is an integer, bool, or float. Very interesting. I agree that this could be a useful feature for some people.

I think the flag should be something like --all_values_are_strings, to be consistent with the --quoted_values_are_strings flag, and if both are given, the --all_values_are_strings takes precedence. (I think it is slightly easier to implement this rule than "the last flag wins" because of how the argparse library works).

Do you want to send me a PR?