How to control the shard size for reading JSON files

[coolderli]

How to control the shard size for reading JSON files? I have a 2.7TB JSON file, but only one shard. Thanks.

* TabularScan:
|   Num Scan Tasks = 1
|   Estimated Scan Bytes = 2864423480835
|   Clustering spec = { Num partitions = 1 }
|   Pushdowns: {filter: is_null(get(get(col(metadata))))}

[universalmind303]

Are you able share more information about your data? is this a local JSON file? or is this in an object store like S3?

We do have a chunk_size method on the read_json function that you could try setting to change the number of chunks

[coolderli]

@universalmind303 It's a local JSON file. I have already set the chunk size. The daft version is 0.4.16.

* Filter: is_null(get(get(col(metadata))))
|
* GlobScanOperator
|   Glob paths = [/mnt/bos-text/raw_data/worldcat.jsonl]
|   Chunk size = 1073741824
|   IO config = S3 config = { Max connections = 8, Retry initial backoff ms = 1000,
|     Connect timeout ms = 30000, Read timeout ms = 30000, Max retries = 25, Retry
|     mode = adaptive, Anonymous = false, Use SSL = true, Verify SSL = true, Check
|     hostname SSL = true, Requester pays = false, Force Virtual Addressing = false },
|     Azure config = { Use Fabric Endpoint = false, Anonymous = false, Use SSL =
|     true }, GCS config = { Anonymous = false, Max connections = 8, Retry initial
|     backoff ms = 1000, Connect timeout ms = 30000, Read timeout ms = 30000, Max
|     retries = 5 }, HTTP config = { User agent = daft/0.0.1, Retry initial backoff ms
|     = 1000, Connect timeout ms = 30000, Read timeout ms = 30000, Max retries = 5 }
|   Use multithreading = true
|   File schema = aacid#Utf8, metadata#Struct([Field { name: "oclc_number", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} }, Field
|     { name: "from_filenames", dtype: List(Utf8), metadata: {} }, Field { name:
|     "record", dtype: Struct([Field { name: "oclcNumber", dtype: Utf8, metadata:
|     {} }, Field { name: "title", dtype: Utf8, metadata: {} }, Field { name:
|     "titleInfo", dtype: Struct([Field { name: "text", dtype: Utf8, metadata:
|     {} }]), metadata: {} }, Field { name: "creator", dtype: Utf8, metadata: {} },
|     Field { name: "generalFormat", dtype: Utf8, metadata: {} }, Field { name:
|     "specificFormat", dtype: Utf8, metadata: {} }, Field { name: "edition",
|     dtype: Utf8, metadata: {} }, Field { name: "totalEditions", dtype: Int64,
|     metadata: {} }, Field { name: "publisher", dtype: Utf8, metadata: {} }, Field
|     { name: "publisherName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "publicationPlace", dtype:
|     Utf8, metadata: {} }, Field { name: "publicationDate", dtype: Utf8, metadata:
|     {} }, Field { name: "catalogingLanguage", dtype: Utf8, metadata: {} }, Field
|     { name: "physicalDescription", dtype: Utf8, metadata: {} }, Field { name:
|     "series", dtype: Utf8, metadata: {} }, Field { name: "languageNotes", dtype:
|     List(Utf8), metadata: {} }, Field { name: "subjectsText", dtype: List(Utf8),
|     metadata: {} }, Field { name: "dissertationInfo", dtype: Struct([Field
|     { name: "institution", dtype: Utf8, metadata: {} }, Field { name: "year",
|     dtype: Utf8, metadata: {} }]), metadata: {} }, Field { name: "genre", dtype:
|     Utf8, metadata: {} }, Field { name: "generalNotes", dtype: List(Struct([Field
|     { name: "text", dtype: Utf8, metadata: {} }, Field { name: "local", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "contentNotes", dtype:
|     Struct([Field { name: "text", dtype: List(Utf8), metadata: {} }]), metadata:
|     {} }, Field { name: "peerReviewed", dtype: Utf8, metadata: {} }, Field
|     { name: "additionalPhysicalFormEntries", dtype: List(Struct([Field { name:
|     "displayConstant", dtype: Utf8, metadata: {} }, Field { name: "titles",
|     dtype: List(Utf8), metadata: {} }, Field { name: "recordControlOclcNumbers",
|     dtype: List(Utf8), metadata: {} }, Field { name: "mainEntryHeadings", dtype:
|     List(Utf8), metadata: {} }, Field { name: "uniformTitle", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "digitalAccessAndLocations", dtype:
|     List(Struct([Field { name: "uri", dtype: Utf8, metadata: {} }, Field { name:
|     "linkText", dtype: Utf8, metadata: {} }, Field { name: "materialSpecified",
|     dtype: Utf8, metadata: {} }, Field { name: "publicNote", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "abstract", dtype: Utf8,
|     metadata: {} }, Field { name: "otherFormats", dtype: List(Struct([Field { name:
|     "oclcNumber", dtype: Utf8, metadata: {} }, Field { name: "generalFormat",
|     dtype: Utf8, metadata: {} }, Field { name: "specificFormat", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "isbns", dtype: List(Utf8),
|     metadata: {} }, Field { name: "isbn13", dtype: Utf8, metadata: {} }, Field
|     { name: "openAccessLinks", dtype: List(Struct([Field { name: "url", dtype:
|     Utf8, metadata: {} }, Field { name: "source", dtype: Utf8, metadata: {} },
|     Field { name: "contentSource", dtype: Utf8, metadata: {} }, Field { name:
|     "materialSpecified", dtype: Utf8, metadata: {} }])), metadata: {} }, Field
|     { name: "contributors", dtype: List(Struct([Field { name: "firstName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata: {} },
|     Field { name: "secondName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "isPrimary", dtype: Boolean,
|     metadata: {} }, Field { name: "relatorCodes", dtype: List(Utf8), metadata:
|     {} }, Field { name: "contributorNotes", dtype: Struct([Field { name: "date",
|     dtype: List(Utf8), metadata: {} }, Field { name: "note", dtype: List(Utf8),
|     metadata: {} }]), metadata: {} }, Field { name: "nonPersonName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata:
|     {} }, Field { name: "includes", dtype: List(Struct([Field { name: "title",
|     dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name: "uris", dtype:
|     List(Struct([Field { name: "type", dtype: Utf8, metadata: {} }, Field { name:
|     "uri", dtype: Utf8, metadata: {} }])), metadata: {} }])), metadata: {} },
|     Field { name: "machineReadableDate", dtype: Utf8, metadata: {} }, Field
|     { name: "seriesVolumes", dtype: List(Utf8), metadata: {} }, Field { name:
|     "mergedOclcNumbers", dtype: List(Utf8), metadata: {} }, Field { name:
|     "date", dtype: Utf8, metadata: {} }, Field { name: "language", dtype: Utf8,
|     metadata: {} }, Field { name: "subjects", dtype: List(Utf8), metadata: {} },
|     Field { name: "summaries", dtype: List(Utf8), metadata: {} }, Field { name:
|     "summary", dtype: Utf8, metadata: {} }, Field { name: "summariesText",
|     dtype: List(Utf8), metadata: {} }, Field { name: "oclc_number", dtype: Int64,
|     metadata: {} }, Field { name: "total_holding_count", dtype: Int64, metadata:
|     {} }, Field { name: "total_editions", dtype: Int64, metadata: {} }, Field
|     { name: "abstractsText", dtype: List(Utf8), metadata: {} }, Field { name:
|     "totalHoldingCount", dtype: Int64, metadata: {} }, Field { name: "holdings",
|     dtype: List(Int64), metadata: {} }, Field { name: "numPublicLibraries", dtype:
|     Int64, metadata: {} }, Field { name: "authors", dtype: List(Struct([Field
|     { name: "firstNameObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "flipNameOrder", dtype: Boolean,
|     metadata: {} }, Field { name: "lastNameObject", dtype: Struct([Field { name:
|     "", dtype: Null, metadata: {} }]), metadata: {} }, Field { name: "notes",
|     dtype: Utf8, metadata: {} }, Field { name: "primary", dtype: Boolean, metadata:
|     {} }, Field { name: "relatorList", dtype: Struct([Field { name: "relators",
|     dtype: List(Struct([Field { name: "code", dtype: Utf8, metadata: {} }, Field
|     { name: "term", dtype: Utf8, metadata: {} }])), metadata: {} }]), metadata:
|     {} }, Field { name: "subFieldsQueryString", dtype: Utf8, metadata: {} }, Field
|     { name: "type", dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name:
|     "contentsObjects", dtype: List(Struct([Field { name: "note", dtype: Utf8,
|     metadata: {} }, Field { name: "noteObject", dtype: Struct([Field { name: "data",
|     dtype: Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }])), metadata: {} }, Field { name: "defaultCoverArtUrl",
|     dtype: Utf8, metadata: {} }, Field { name: "digitalGraphicRepresentation",
|     dtype: Utf8, metadata: {} }, Field { name: "disableAuthorLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype: Boolean,
|     metadata: {} }, Field { name: "editionId", dtype: Utf8, metadata: {} },
|     Field { name: "editionSingletonEdition", dtype: Boolean, metadata: {} },
|     Field { name: "enhancedCollectionName", dtype: Utf8, metadata: {} }, Field
|     { name: "genreObjects", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }, Field { name: "local", dtype: Boolean, metadata: {} }])),
|     metadata: {} }, Field { name: "genres", dtype: List(Utf8), metadata: {} },
|     Field { name: "heldByLevel", dtype: Int64, metadata: {} }, Field { name:
|     "highlightedRecord", dtype: Struct([Field { name: "disableAuthorLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype:
|     Boolean, metadata: {} }, Field { name: "enhancedCollectionName", dtype:
|     Utf8, metadata: {} }, Field { name: "heldByLevel", dtype: Int64, metadata:
|     {} }, Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field
|     { name: "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field
|     { name: "numberOfEditionIds", dtype: Int64, metadata: {} }, Field { name:
|     "numberOfOtherEditions", dtype: Int64, metadata: {} }, Field { name:
|     "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name: "titleObject",
|     dtype: Struct([Field { name: "", dtype: Null, metadata: {} }]), metadata:
|     {} }]), metadata: {} }, Field { name: "itemType", dtype: Utf8, metadata: {} },
|     Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field { name:
|     "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field { name:
|     "lcNumber", dtype: Utf8, metadata: {} }, Field { name: "masterCallNumber",
|     dtype: Utf8, metadata: {} }, Field { name: "mediumCoverArtUrl", dtype: Utf8,
|     metadata: {} }, Field { name: "musicalPresentationStatement", dtype: Utf8,
|     metadata: {} }, Field { name: "numberOfEditionIds", dtype: Int64, metadata:
|     {} }, Field { name: "numberOfOtherEditions", dtype: Int64, metadata: {} },
|     Field { name: "openUrlContextObject", dtype: Utf8, metadata: {} }, Field
|     { name: "publishers", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "remoteDatabase", dtype:
|     Boolean, metadata: {} }, Field { name: "source", dtype: Utf8, metadata:
|     {} }, Field { name: "sourceCollection", dtype: Utf8, metadata: {} }, Field
|     { name: "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name:
|     "subjectGroups", dtype: List(Struct([Field { name: "bibSubjects", dtype:
|     List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field { name:
|     "local", dtype: Boolean, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} },
|     Field { name: "unifiedData", dtype: Struct([Field { name: "data", dtype:
|     Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }, Field { name: "otherSource", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "id", dtype: Utf8, metadata: {} },
|     Field { name: "isPromoted", dtype: Boolean, metadata: {} }, Field { name:
|     "label", dtype: Utf8, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "summariesObjectList",
|     dtype: List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field
|     { name: "private", dtype: Boolean, metadata: {} }])), metadata: {} }, Field
|     { name: "titleObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "workCount", dtype: Int64,
|     metadata: {} }, Field { name: "workId", dtype: Utf8, metadata: {} }, Field
|     { name: "workSingletonIndicator", dtype: Boolean, metadata: {} }, Field { name:
|     "workSingletonWork", dtype: Boolean, metadata: {} }, Field { name: "not_found",
|     dtype: Int64, metadata: {} }]), metadata: {} }, Field { name: "html", dtype:
|     Utf8, metadata: {} }])
|   Partitioning keys = []
|   Output schema = aacid#Utf8, metadata#Struct([Field { name: "oclc_number",
|     dtype: Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata:
|     {} }, Field { name: "from_filenames", dtype: List(Utf8), metadata: {} },
|     Field { name: "record", dtype: Struct([Field { name: "oclcNumber", dtype:
|     Utf8, metadata: {} }, Field { name: "title", dtype: Utf8, metadata: {} },
|     Field { name: "titleInfo", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "creator", dtype: Utf8,
|     metadata: {} }, Field { name: "generalFormat", dtype: Utf8, metadata: {} },
|     Field { name: "specificFormat", dtype: Utf8, metadata: {} }, Field { name:
|     "edition", dtype: Utf8, metadata: {} }, Field { name: "totalEditions", dtype:
|     Int64, metadata: {} }, Field { name: "publisher", dtype: Utf8, metadata: {} },
|     Field { name: "publisherName", dtype: Struct([Field { name: "text", dtype:
|     Utf8, metadata: {} }]), metadata: {} }, Field { name: "publicationPlace", dtype:
|     Utf8, metadata: {} }, Field { name: "publicationDate", dtype: Utf8, metadata:
|     {} }, Field { name: "catalogingLanguage", dtype: Utf8, metadata: {} }, Field
|     { name: "physicalDescription", dtype: Utf8, metadata: {} }, Field { name:
|     "series", dtype: Utf8, metadata: {} }, Field { name: "languageNotes", dtype:
|     List(Utf8), metadata: {} }, Field { name: "subjectsText", dtype: List(Utf8),
|     metadata: {} }, Field { name: "dissertationInfo", dtype: Struct([Field
|     { name: "institution", dtype: Utf8, metadata: {} }, Field { name: "year",
|     dtype: Utf8, metadata: {} }]), metadata: {} }, Field { name: "genre", dtype:
|     Utf8, metadata: {} }, Field { name: "generalNotes", dtype: List(Struct([Field
|     { name: "text", dtype: Utf8, metadata: {} }, Field { name: "local", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "contentNotes", dtype:
|     Struct([Field { name: "text", dtype: List(Utf8), metadata: {} }]), metadata:
|     {} }, Field { name: "peerReviewed", dtype: Utf8, metadata: {} }, Field
|     { name: "additionalPhysicalFormEntries", dtype: List(Struct([Field { name:
|     "displayConstant", dtype: Utf8, metadata: {} }, Field { name: "titles",
|     dtype: List(Utf8), metadata: {} }, Field { name: "recordControlOclcNumbers",
|     dtype: List(Utf8), metadata: {} }, Field { name: "mainEntryHeadings", dtype:
|     List(Utf8), metadata: {} }, Field { name: "uniformTitle", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "digitalAccessAndLocations", dtype:
|     List(Struct([Field { name: "uri", dtype: Utf8, metadata: {} }, Field { name:
|     "linkText", dtype: Utf8, metadata: {} }, Field { name: "materialSpecified",
|     dtype: Utf8, metadata: {} }, Field { name: "publicNote", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "abstract", dtype: Utf8,
|     metadata: {} }, Field { name: "otherFormats", dtype: List(Struct([Field { name:
|     "oclcNumber", dtype: Utf8, metadata: {} }, Field { name: "generalFormat",
|     dtype: Utf8, metadata: {} }, Field { name: "specificFormat", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "isbns", dtype: List(Utf8),
|     metadata: {} }, Field { name: "isbn13", dtype: Utf8, metadata: {} }, Field
|     { name: "openAccessLinks", dtype: List(Struct([Field { name: "url", dtype:
|     Utf8, metadata: {} }, Field { name: "source", dtype: Utf8, metadata: {} },
|     Field { name: "contentSource", dtype: Utf8, metadata: {} }, Field { name:
|     "materialSpecified", dtype: Utf8, metadata: {} }])), metadata: {} }, Field
|     { name: "contributors", dtype: List(Struct([Field { name: "firstName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata: {} },
|     Field { name: "secondName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "isPrimary", dtype: Boolean,
|     metadata: {} }, Field { name: "relatorCodes", dtype: List(Utf8), metadata:
|     {} }, Field { name: "contributorNotes", dtype: Struct([Field { name: "date",
|     dtype: List(Utf8), metadata: {} }, Field { name: "note", dtype: List(Utf8),
|     metadata: {} }]), metadata: {} }, Field { name: "nonPersonName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata:
|     {} }, Field { name: "includes", dtype: List(Struct([Field { name: "title",
|     dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name: "uris", dtype:
|     List(Struct([Field { name: "type", dtype: Utf8, metadata: {} }, Field { name:
|     "uri", dtype: Utf8, metadata: {} }])), metadata: {} }])), metadata: {} },
|     Field { name: "machineReadableDate", dtype: Utf8, metadata: {} }, Field
|     { name: "seriesVolumes", dtype: List(Utf8), metadata: {} }, Field { name:
|     "mergedOclcNumbers", dtype: List(Utf8), metadata: {} }, Field { name:
|     "date", dtype: Utf8, metadata: {} }, Field { name: "language", dtype: Utf8,
|     metadata: {} }, Field { name: "subjects", dtype: List(Utf8), metadata: {} },
|     Field { name: "summaries", dtype: List(Utf8), metadata: {} }, Field { name:
|     "summary", dtype: Utf8, metadata: {} }, Field { name: "summariesText",
|     dtype: List(Utf8), metadata: {} }, Field { name: "oclc_number", dtype: Int64,
|     metadata: {} }, Field { name: "total_holding_count", dtype: Int64, metadata:
|     {} }, Field { name: "total_editions", dtype: Int64, metadata: {} }, Field
|     { name: "abstractsText", dtype: List(Utf8), metadata: {} }, Field { name:
|     "totalHoldingCount", dtype: Int64, metadata: {} }, Field { name: "holdings",
|     dtype: List(Int64), metadata: {} }, Field { name: "numPublicLibraries", dtype:
|     Int64, metadata: {} }, Field { name: "authors", dtype: List(Struct([Field
|     { name: "firstNameObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "flipNameOrder", dtype: Boolean,
|     metadata: {} }, Field { name: "lastNameObject", dtype: Struct([Field { name:
|     "", dtype: Null, metadata: {} }]), metadata: {} }, Field { name: "notes",
|     dtype: Utf8, metadata: {} }, Field { name: "primary", dtype: Boolean, metadata:
|     {} }, Field { name: "relatorList", dtype: Struct([Field { name: "relators",
|     dtype: List(Struct([Field { name: "code", dtype: Utf8, metadata: {} }, Field
|     { name: "term", dtype: Utf8, metadata: {} }])), metadata: {} }]), metadata:
|     {} }, Field { name: "subFieldsQueryString", dtype: Utf8, metadata: {} }, Field
|     { name: "type", dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name:
|     "contentsObjects", dtype: List(Struct([Field { name: "note", dtype: Utf8,
|     metadata: {} }, Field { name: "noteObject", dtype: Struct([Field { name: "data",
|     dtype: Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }])), metadata: {} }, Field { name: "defaultCoverArtUrl",
|     dtype: Utf8, metadata: {} }, Field { name: "digitalGraphicRepresentation",
|     dtype: Utf8, metadata: {} }, Field { name: "disableAuthorLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype: Boolean,
|     metadata: {} }, Field { name: "editionId", dtype: Utf8, metadata: {} },
|     Field { name: "editionSingletonEdition", dtype: Boolean, metadata: {} },
|     Field { name: "enhancedCollectionName", dtype: Utf8, metadata: {} }, Field
|     { name: "genreObjects", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }, Field { name: "local", dtype: Boolean, metadata: {} }])),
|     metadata: {} }, Field { name: "genres", dtype: List(Utf8), metadata: {} },
|     Field { name: "heldByLevel", dtype: Int64, metadata: {} }, Field { name:
|     "highlightedRecord", dtype: Struct([Field { name: "disableAuthorLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype:
|     Boolean, metadata: {} }, Field { name: "enhancedCollectionName", dtype:
|     Utf8, metadata: {} }, Field { name: "heldByLevel", dtype: Int64, metadata:
|     {} }, Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field
|     { name: "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field
|     { name: "numberOfEditionIds", dtype: Int64, metadata: {} }, Field { name:
|     "numberOfOtherEditions", dtype: Int64, metadata: {} }, Field { name:
|     "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name: "titleObject",
|     dtype: Struct([Field { name: "", dtype: Null, metadata: {} }]), metadata:
|     {} }]), metadata: {} }, Field { name: "itemType", dtype: Utf8, metadata: {} },
|     Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field { name:
|     "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field { name:
|     "lcNumber", dtype: Utf8, metadata: {} }, Field { name: "masterCallNumber",
|     dtype: Utf8, metadata: {} }, Field { name: "mediumCoverArtUrl", dtype: Utf8,
|     metadata: {} }, Field { name: "musicalPresentationStatement", dtype: Utf8,
|     metadata: {} }, Field { name: "numberOfEditionIds", dtype: Int64, metadata:
|     {} }, Field { name: "numberOfOtherEditions", dtype: Int64, metadata: {} },
|     Field { name: "openUrlContextObject", dtype: Utf8, metadata: {} }, Field
|     { name: "publishers", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "remoteDatabase", dtype:
|     Boolean, metadata: {} }, Field { name: "source", dtype: Utf8, metadata:
|     {} }, Field { name: "sourceCollection", dtype: Utf8, metadata: {} }, Field
|     { name: "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name:
|     "subjectGroups", dtype: List(Struct([Field { name: "bibSubjects", dtype:
|     List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field { name:
|     "local", dtype: Boolean, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} },
|     Field { name: "unifiedData", dtype: Struct([Field { name: "data", dtype:
|     Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }, Field { name: "otherSource", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "id", dtype: Utf8, metadata: {} },
|     Field { name: "isPromoted", dtype: Boolean, metadata: {} }, Field { name:
|     "label", dtype: Utf8, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "summariesObjectList",
|     dtype: List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field
|     { name: "private", dtype: Boolean, metadata: {} }])), metadata: {} }, Field
|     { name: "titleObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "workCount", dtype: Int64,
|     metadata: {} }, Field { name: "workId", dtype: Utf8, metadata: {} }, Field
|     { name: "workSingletonIndicator", dtype: Boolean, metadata: {} }, Field { name:
|     "workSingletonWork", dtype: Boolean, metadata: {} }, Field { name: "not_found",
|     dtype: Int64, metadata: {} }]), metadata: {} }, Field { name: "html", dtype:
|     Utf8, metadata: {} }])


== Optimized Logical Plan ==

* Num Scan Tasks = 1
|   File schema = aacid#Utf8, metadata#Struct([Field { name: "oclc_number", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} }, Field
|     { name: "from_filenames", dtype: List(Utf8), metadata: {} }, Field { name:
|     "record", dtype: Struct([Field { name: "oclcNumber", dtype: Utf8, metadata:
|     {} }, Field { name: "title", dtype: Utf8, metadata: {} }, Field { name:
|     "titleInfo", dtype: Struct([Field { name: "text", dtype: Utf8, metadata:
|     {} }]), metadata: {} }, Field { name: "creator", dtype: Utf8, metadata: {} },
|     Field { name: "generalFormat", dtype: Utf8, metadata: {} }, Field { name:
|     "specificFormat", dtype: Utf8, metadata: {} }, Field { name: "edition",
|     dtype: Utf8, metadata: {} }, Field { name: "totalEditions", dtype: Int64,
|     metadata: {} }, Field { name: "publisher", dtype: Utf8, metadata: {} }, Field
|     { name: "publisherName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "publicationPlace", dtype:
|     Utf8, metadata: {} }, Field { name: "publicationDate", dtype: Utf8, metadata:
|     {} }, Field { name: "catalogingLanguage", dtype: Utf8, metadata: {} }, Field
|     { name: "physicalDescription", dtype: Utf8, metadata: {} }, Field { name:
|     "series", dtype: Utf8, metadata: {} }, Field { name: "languageNotes", dtype:
|     List(Utf8), metadata: {} }, Field { name: "subjectsText", dtype: List(Utf8),
|     metadata: {} }, Field { name: "dissertationInfo", dtype: Struct([Field
|     { name: "institution", dtype: Utf8, metadata: {} }, Field { name: "year",
|     dtype: Utf8, metadata: {} }]), metadata: {} }, Field { name: "genre", dtype:
|     Utf8, metadata: {} }, Field { name: "generalNotes", dtype: List(Struct([Field
|     { name: "text", dtype: Utf8, metadata: {} }, Field { name: "local", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "contentNotes", dtype:
|     Struct([Field { name: "text", dtype: List(Utf8), metadata: {} }]), metadata:
|     {} }, Field { name: "peerReviewed", dtype: Utf8, metadata: {} }, Field
|     { name: "additionalPhysicalFormEntries", dtype: List(Struct([Field { name:
|     "displayConstant", dtype: Utf8, metadata: {} }, Field { name: "titles",
|     dtype: List(Utf8), metadata: {} }, Field { name: "recordControlOclcNumbers",
|     dtype: List(Utf8), metadata: {} }, Field { name: "mainEntryHeadings", dtype:
|     List(Utf8), metadata: {} }, Field { name: "uniformTitle", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "digitalAccessAndLocations", dtype:
|     List(Struct([Field { name: "uri", dtype: Utf8, metadata: {} }, Field { name:
|     "linkText", dtype: Utf8, metadata: {} }, Field { name: "materialSpecified",
|     dtype: Utf8, metadata: {} }, Field { name: "publicNote", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "abstract", dtype: Utf8,
|     metadata: {} }, Field { name: "otherFormats", dtype: List(Struct([Field { name:
|     "oclcNumber", dtype: Utf8, metadata: {} }, Field { name: "generalFormat",
|     dtype: Utf8, metadata: {} }, Field { name: "specificFormat", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "isbns", dtype: List(Utf8),
|     metadata: {} }, Field { name: "isbn13", dtype: Utf8, metadata: {} }, Field
|     { name: "openAccessLinks", dtype: List(Struct([Field { name: "url", dtype:
|     Utf8, metadata: {} }, Field { name: "source", dtype: Utf8, metadata: {} },
|     Field { name: "contentSource", dtype: Utf8, metadata: {} }, Field { name:
|     "materialSpecified", dtype: Utf8, metadata: {} }])), metadata: {} }, Field
|     { name: "contributors", dtype: List(Struct([Field { name: "firstName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata: {} },
|     Field { name: "secondName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "isPrimary", dtype: Boolean,
|     metadata: {} }, Field { name: "relatorCodes", dtype: List(Utf8), metadata:
|     {} }, Field { name: "contributorNotes", dtype: Struct([Field { name: "date",
|     dtype: List(Utf8), metadata: {} }, Field { name: "note", dtype: List(Utf8),
|     metadata: {} }]), metadata: {} }, Field { name: "nonPersonName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata:
|     {} }, Field { name: "includes", dtype: List(Struct([Field { name: "title",
|     dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name: "uris", dtype:
|     List(Struct([Field { name: "type", dtype: Utf8, metadata: {} }, Field { name:
|     "uri", dtype: Utf8, metadata: {} }])), metadata: {} }])), metadata: {} },
|     Field { name: "machineReadableDate", dtype: Utf8, metadata: {} }, Field
|     { name: "seriesVolumes", dtype: List(Utf8), metadata: {} }, Field { name:
|     "mergedOclcNumbers", dtype: List(Utf8), metadata: {} }, Field { name:
|     "date", dtype: Utf8, metadata: {} }, Field { name: "language", dtype: Utf8,
|     metadata: {} }, Field { name: "subjects", dtype: List(Utf8), metadata: {} },
|     Field { name: "summaries", dtype: List(Utf8), metadata: {} }, Field { name:
|     "summary", dtype: Utf8, metadata: {} }, Field { name: "summariesText",
|     dtype: List(Utf8), metadata: {} }, Field { name: "oclc_number", dtype: Int64,
|     metadata: {} }, Field { name: "total_holding_count", dtype: Int64, metadata:
|     {} }, Field { name: "total_editions", dtype: Int64, metadata: {} }, Field
|     { name: "abstractsText", dtype: List(Utf8), metadata: {} }, Field { name:
|     "totalHoldingCount", dtype: Int64, metadata: {} }, Field { name: "holdings",
|     dtype: List(Int64), metadata: {} }, Field { name: "numPublicLibraries", dtype:
|     Int64, metadata: {} }, Field { name: "authors", dtype: List(Struct([Field
|     { name: "firstNameObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "flipNameOrder", dtype: Boolean,
|     metadata: {} }, Field { name: "lastNameObject", dtype: Struct([Field { name:
|     "", dtype: Null, metadata: {} }]), metadata: {} }, Field { name: "notes",
|     dtype: Utf8, metadata: {} }, Field { name: "primary", dtype: Boolean, metadata:
|     {} }, Field { name: "relatorList", dtype: Struct([Field { name: "relators",
|     dtype: List(Struct([Field { name: "code", dtype: Utf8, metadata: {} }, Field
|     { name: "term", dtype: Utf8, metadata: {} }])), metadata: {} }]), metadata:
|     {} }, Field { name: "subFieldsQueryString", dtype: Utf8, metadata: {} }, Field
|     { name: "type", dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name:
|     "contentsObjects", dtype: List(Struct([Field { name: "note", dtype: Utf8,
|     metadata: {} }, Field { name: "noteObject", dtype: Struct([Field { name: "data",
|     dtype: Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }])), metadata: {} }, Field { name: "defaultCoverArtUrl",
|     dtype: Utf8, metadata: {} }, Field { name: "digitalGraphicRepresentation",
|     dtype: Utf8, metadata: {} }, Field { name: "disableAuthorLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype: Boolean,
|     metadata: {} }, Field { name: "editionId", dtype: Utf8, metadata: {} },
|     Field { name: "editionSingletonEdition", dtype: Boolean, metadata: {} },
|     Field { name: "enhancedCollectionName", dtype: Utf8, metadata: {} }, Field
|     { name: "genreObjects", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }, Field { name: "local", dtype: Boolean, metadata: {} }])),
|     metadata: {} }, Field { name: "genres", dtype: List(Utf8), metadata: {} },
|     Field { name: "heldByLevel", dtype: Int64, metadata: {} }, Field { name:
|     "highlightedRecord", dtype: Struct([Field { name: "disableAuthorLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype:
|     Boolean, metadata: {} }, Field { name: "enhancedCollectionName", dtype:
|     Utf8, metadata: {} }, Field { name: "heldByLevel", dtype: Int64, metadata:
|     {} }, Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field
|     { name: "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field
|     { name: "numberOfEditionIds", dtype: Int64, metadata: {} }, Field { name:
|     "numberOfOtherEditions", dtype: Int64, metadata: {} }, Field { name:
|     "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name: "titleObject",
|     dtype: Struct([Field { name: "", dtype: Null, metadata: {} }]), metadata:
|     {} }]), metadata: {} }, Field { name: "itemType", dtype: Utf8, metadata: {} },
|     Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field { name:
|     "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field { name:
|     "lcNumber", dtype: Utf8, metadata: {} }, Field { name: "masterCallNumber",
|     dtype: Utf8, metadata: {} }, Field { name: "mediumCoverArtUrl", dtype: Utf8,
|     metadata: {} }, Field { name: "musicalPresentationStatement", dtype: Utf8,
|     metadata: {} }, Field { name: "numberOfEditionIds", dtype: Int64, metadata:
|     {} }, Field { name: "numberOfOtherEditions", dtype: Int64, metadata: {} },
|     Field { name: "openUrlContextObject", dtype: Utf8, metadata: {} }, Field
|     { name: "publishers", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "remoteDatabase", dtype:
|     Boolean, metadata: {} }, Field { name: "source", dtype: Utf8, metadata:
|     {} }, Field { name: "sourceCollection", dtype: Utf8, metadata: {} }, Field
|     { name: "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name:
|     "subjectGroups", dtype: List(Struct([Field { name: "bibSubjects", dtype:
|     List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field { name:
|     "local", dtype: Boolean, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} },
|     Field { name: "unifiedData", dtype: Struct([Field { name: "data", dtype:
|     Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }, Field { name: "otherSource", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "id", dtype: Utf8, metadata: {} },
|     Field { name: "isPromoted", dtype: Boolean, metadata: {} }, Field { name:
|     "label", dtype: Utf8, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "summariesObjectList",
|     dtype: List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field
|     { name: "private", dtype: Boolean, metadata: {} }])), metadata: {} }, Field
|     { name: "titleObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "workCount", dtype: Int64,
|     metadata: {} }, Field { name: "workId", dtype: Utf8, metadata: {} }, Field
|     { name: "workSingletonIndicator", dtype: Boolean, metadata: {} }, Field { name:
|     "workSingletonWork", dtype: Boolean, metadata: {} }, Field { name: "not_found",
|     dtype: Int64, metadata: {} }]), metadata: {} }, Field { name: "html", dtype:
|     Utf8, metadata: {} }])
|   Partitioning keys = []
|   Filter pushdown = is_null(get(get(col(metadata))))
|   Output schema = aacid#Utf8, metadata#Struct([Field { name: "oclc_number",
|     dtype: Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata:
|     {} }, Field { name: "from_filenames", dtype: List(Utf8), metadata: {} },
|     Field { name: "record", dtype: Struct([Field { name: "oclcNumber", dtype:
|     Utf8, metadata: {} }, Field { name: "title", dtype: Utf8, metadata: {} },
|     Field { name: "titleInfo", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "creator", dtype: Utf8,
|     metadata: {} }, Field { name: "generalFormat", dtype: Utf8, metadata: {} },
|     Field { name: "specificFormat", dtype: Utf8, metadata: {} }, Field { name:
|     "edition", dtype: Utf8, metadata: {} }, Field { name: "totalEditions", dtype:
|     Int64, metadata: {} }, Field { name: "publisher", dtype: Utf8, metadata: {} },
|     Field { name: "publisherName", dtype: Struct([Field { name: "text", dtype:
|     Utf8, metadata: {} }]), metadata: {} }, Field { name: "publicationPlace", dtype:
|     Utf8, metadata: {} }, Field { name: "publicationDate", dtype: Utf8, metadata:
|     {} }, Field { name: "catalogingLanguage", dtype: Utf8, metadata: {} }, Field
|     { name: "physicalDescription", dtype: Utf8, metadata: {} }, Field { name:
|     "series", dtype: Utf8, metadata: {} }, Field { name: "languageNotes", dtype:
|     List(Utf8), metadata: {} }, Field { name: "subjectsText", dtype: List(Utf8),
|     metadata: {} }, Field { name: "dissertationInfo", dtype: Struct([Field
|     { name: "institution", dtype: Utf8, metadata: {} }, Field { name: "year",
|     dtype: Utf8, metadata: {} }]), metadata: {} }, Field { name: "genre", dtype:
|     Utf8, metadata: {} }, Field { name: "generalNotes", dtype: List(Struct([Field
|     { name: "text", dtype: Utf8, metadata: {} }, Field { name: "local", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "contentNotes", dtype:
|     Struct([Field { name: "text", dtype: List(Utf8), metadata: {} }]), metadata:
|     {} }, Field { name: "peerReviewed", dtype: Utf8, metadata: {} }, Field
|     { name: "additionalPhysicalFormEntries", dtype: List(Struct([Field { name:
|     "displayConstant", dtype: Utf8, metadata: {} }, Field { name: "titles",
|     dtype: List(Utf8), metadata: {} }, Field { name: "recordControlOclcNumbers",
|     dtype: List(Utf8), metadata: {} }, Field { name: "mainEntryHeadings", dtype:
|     List(Utf8), metadata: {} }, Field { name: "uniformTitle", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "digitalAccessAndLocations", dtype:
|     List(Struct([Field { name: "uri", dtype: Utf8, metadata: {} }, Field { name:
|     "linkText", dtype: Utf8, metadata: {} }, Field { name: "materialSpecified",
|     dtype: Utf8, metadata: {} }, Field { name: "publicNote", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "abstract", dtype: Utf8,
|     metadata: {} }, Field { name: "otherFormats", dtype: List(Struct([Field { name:
|     "oclcNumber", dtype: Utf8, metadata: {} }, Field { name: "generalFormat",
|     dtype: Utf8, metadata: {} }, Field { name: "specificFormat", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "isbns", dtype: List(Utf8),
|     metadata: {} }, Field { name: "isbn13", dtype: Utf8, metadata: {} }, Field
|     { name: "openAccessLinks", dtype: List(Struct([Field { name: "url", dtype:
|     Utf8, metadata: {} }, Field { name: "source", dtype: Utf8, metadata: {} },
|     Field { name: "contentSource", dtype: Utf8, metadata: {} }, Field { name:
|     "materialSpecified", dtype: Utf8, metadata: {} }])), metadata: {} }, Field
|     { name: "contributors", dtype: List(Struct([Field { name: "firstName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata: {} },
|     Field { name: "secondName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "isPrimary", dtype: Boolean,
|     metadata: {} }, Field { name: "relatorCodes", dtype: List(Utf8), metadata:
|     {} }, Field { name: "contributorNotes", dtype: Struct([Field { name: "date",
|     dtype: List(Utf8), metadata: {} }, Field { name: "note", dtype: List(Utf8),
|     metadata: {} }]), metadata: {} }, Field { name: "nonPersonName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata:
|     {} }, Field { name: "includes", dtype: List(Struct([Field { name: "title",
|     dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name: "uris", dtype:
|     List(Struct([Field { name: "type", dtype: Utf8, metadata: {} }, Field { name:
|     "uri", dtype: Utf8, metadata: {} }])), metadata: {} }])), metadata: {} },
|     Field { name: "machineReadableDate", dtype: Utf8, metadata: {} }, Field
|     { name: "seriesVolumes", dtype: List(Utf8), metadata: {} }, Field { name:
|     "mergedOclcNumbers", dtype: List(Utf8), metadata: {} }, Field { name:
|     "date", dtype: Utf8, metadata: {} }, Field { name: "language", dtype: Utf8,
|     metadata: {} }, Field { name: "subjects", dtype: List(Utf8), metadata: {} },
|     Field { name: "summaries", dtype: List(Utf8), metadata: {} }, Field { name:
|     "summary", dtype: Utf8, metadata: {} }, Field { name: "summariesText",
|     dtype: List(Utf8), metadata: {} }, Field { name: "oclc_number", dtype: Int64,
|     metadata: {} }, Field { name: "total_holding_count", dtype: Int64, metadata:
|     {} }, Field { name: "total_editions", dtype: Int64, metadata: {} }, Field
|     { name: "abstractsText", dtype: List(Utf8), metadata: {} }, Field { name:
|     "totalHoldingCount", dtype: Int64, metadata: {} }, Field { name: "holdings",
|     dtype: List(Int64), metadata: {} }, Field { name: "numPublicLibraries", dtype:
|     Int64, metadata: {} }, Field { name: "authors", dtype: List(Struct([Field
|     { name: "firstNameObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "flipNameOrder", dtype: Boolean,
|     metadata: {} }, Field { name: "lastNameObject", dtype: Struct([Field { name:
|     "", dtype: Null, metadata: {} }]), metadata: {} }, Field { name: "notes",
|     dtype: Utf8, metadata: {} }, Field { name: "primary", dtype: Boolean, metadata:
|     {} }, Field { name: "relatorList", dtype: Struct([Field { name: "relators",
|     dtype: List(Struct([Field { name: "code", dtype: Utf8, metadata: {} }, Field
|     { name: "term", dtype: Utf8, metadata: {} }])), metadata: {} }]), metadata:
|     {} }, Field { name: "subFieldsQueryString", dtype: Utf8, metadata: {} }, Field
|     { name: "type", dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name:
|     "contentsObjects", dtype: List(Struct([Field { name: "note", dtype: Utf8,
|     metadata: {} }, Field { name: "noteObject", dtype: Struct([Field { name: "data",
|     dtype: Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }])), metadata: {} }, Field { name: "defaultCoverArtUrl",
|     dtype: Utf8, metadata: {} }, Field { name: "digitalGraphicRepresentation",
|     dtype: Utf8, metadata: {} }, Field { name: "disableAuthorLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype: Boolean,
|     metadata: {} }, Field { name: "editionId", dtype: Utf8, metadata: {} },
|     Field { name: "editionSingletonEdition", dtype: Boolean, metadata: {} },
|     Field { name: "enhancedCollectionName", dtype: Utf8, metadata: {} }, Field
|     { name: "genreObjects", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }, Field { name: "local", dtype: Boolean, metadata: {} }])),
|     metadata: {} }, Field { name: "genres", dtype: List(Utf8), metadata: {} },
|     Field { name: "heldByLevel", dtype: Int64, metadata: {} }, Field { name:
|     "highlightedRecord", dtype: Struct([Field { name: "disableAuthorLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype:
|     Boolean, metadata: {} }, Field { name: "enhancedCollectionName", dtype:
|     Utf8, metadata: {} }, Field { name: "heldByLevel", dtype: Int64, metadata:
|     {} }, Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field
|     { name: "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field
|     { name: "numberOfEditionIds", dtype: Int64, metadata: {} }, Field { name:
|     "numberOfOtherEditions", dtype: Int64, metadata: {} }, Field { name:
|     "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name: "titleObject",
|     dtype: Struct([Field { name: "", dtype: Null, metadata: {} }]), metadata:
|     {} }]), metadata: {} }, Field { name: "itemType", dtype: Utf8, metadata: {} },
|     Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field { name:
|     "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field { name:
|     "lcNumber", dtype: Utf8, metadata: {} }, Field { name: "masterCallNumber",
|     dtype: Utf8, metadata: {} }, Field { name: "mediumCoverArtUrl", dtype: Utf8,
|     metadata: {} }, Field { name: "musicalPresentationStatement", dtype: Utf8,
|     metadata: {} }, Field { name: "numberOfEditionIds", dtype: Int64, metadata:
|     {} }, Field { name: "numberOfOtherEditions", dtype: Int64, metadata: {} },
|     Field { name: "openUrlContextObject", dtype: Utf8, metadata: {} }, Field
|     { name: "publishers", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "remoteDatabase", dtype:
|     Boolean, metadata: {} }, Field { name: "source", dtype: Utf8, metadata:
|     {} }, Field { name: "sourceCollection", dtype: Utf8, metadata: {} }, Field
|     { name: "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name:
|     "subjectGroups", dtype: List(Struct([Field { name: "bibSubjects", dtype:
|     List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field { name:
|     "local", dtype: Boolean, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} },
|     Field { name: "unifiedData", dtype: Struct([Field { name: "data", dtype:
|     Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }, Field { name: "otherSource", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "id", dtype: Utf8, metadata: {} },
|     Field { name: "isPromoted", dtype: Boolean, metadata: {} }, Field { name:
|     "label", dtype: Utf8, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "summariesObjectList",
|     dtype: List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field
|     { name: "private", dtype: Boolean, metadata: {} }])), metadata: {} }, Field
|     { name: "titleObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "workCount", dtype: Int64,
|     metadata: {} }, Field { name: "workId", dtype: Utf8, metadata: {} }, Field
|     { name: "workSingletonIndicator", dtype: Boolean, metadata: {} }, Field { name:
|     "workSingletonWork", dtype: Boolean, metadata: {} }, Field { name: "not_found",
|     dtype: Int64, metadata: {} }]), metadata: {} }, Field { name: "html", dtype:
|     Utf8, metadata: {} }])
|   Stats = { Approx num rows = 7,402,758, Approx size bytes = 66.69 GiB,
|     Accumulated selectivity = 0.05 }


== Physical Plan ==

* TabularScan:
|   Num Scan Tasks = 1
|   Estimated Scan Bytes = 2864423480835
|   Clustering spec = { Num partitions = 1 }
|   Pushdowns: {filter: is_null(get(get(col(metadata))))}
|   Schema: {aacid#Utf8, metadata#Struct([Field { name: "oclc_number", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} }, Field
|     { name: "from_filenames", dtype: List(Utf8), metadata: {} }, Field { name:
|     "record", dtype: Struct([Field { name: "oclcNumber", dtype: Utf8, metadata:
|     {} }, Field { name: "title", dtype: Utf8, metadata: {} }, Field { name:
|     "titleInfo", dtype: Struct([Field { name: "text", dtype: Utf8, metadata:
|     {} }]), metadata: {} }, Field { name: "creator", dtype: Utf8, metadata: {} },
|     Field { name: "generalFormat", dtype: Utf8, metadata: {} }, Field { name:
|     "specificFormat", dtype: Utf8, metadata: {} }, Field { name: "edition",
|     dtype: Utf8, metadata: {} }, Field { name: "totalEditions", dtype: Int64,
|     metadata: {} }, Field { name: "publisher", dtype: Utf8, metadata: {} }, Field
|     { name: "publisherName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "publicationPlace", dtype:
|     Utf8, metadata: {} }, Field { name: "publicationDate", dtype: Utf8, metadata:
|     {} }, Field { name: "catalogingLanguage", dtype: Utf8, metadata: {} }, Field
|     { name: "physicalDescription", dtype: Utf8, metadata: {} }, Field { name:
|     "series", dtype: Utf8, metadata: {} }, Field { name: "languageNotes", dtype:
|     List(Utf8), metadata: {} }, Field { name: "subjectsText", dtype: List(Utf8),
|     metadata: {} }, Field { name: "dissertationInfo", dtype: Struct([Field
|     { name: "institution", dtype: Utf8, metadata: {} }, Field { name: "year",
|     dtype: Utf8, metadata: {} }]), metadata: {} }, Field { name: "genre", dtype:
|     Utf8, metadata: {} }, Field { name: "generalNotes", dtype: List(Struct([Field
|     { name: "text", dtype: Utf8, metadata: {} }, Field { name: "local", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "contentNotes", dtype:
|     Struct([Field { name: "text", dtype: List(Utf8), metadata: {} }]), metadata:
|     {} }, Field { name: "peerReviewed", dtype: Utf8, metadata: {} }, Field
|     { name: "additionalPhysicalFormEntries", dtype: List(Struct([Field { name:
|     "displayConstant", dtype: Utf8, metadata: {} }, Field { name: "titles",
|     dtype: List(Utf8), metadata: {} }, Field { name: "recordControlOclcNumbers",
|     dtype: List(Utf8), metadata: {} }, Field { name: "mainEntryHeadings", dtype:
|     List(Utf8), metadata: {} }, Field { name: "uniformTitle", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "digitalAccessAndLocations", dtype:
|     List(Struct([Field { name: "uri", dtype: Utf8, metadata: {} }, Field { name:
|     "linkText", dtype: Utf8, metadata: {} }, Field { name: "materialSpecified",
|     dtype: Utf8, metadata: {} }, Field { name: "publicNote", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "abstract", dtype: Utf8,
|     metadata: {} }, Field { name: "otherFormats", dtype: List(Struct([Field { name:
|     "oclcNumber", dtype: Utf8, metadata: {} }, Field { name: "generalFormat",
|     dtype: Utf8, metadata: {} }, Field { name: "specificFormat", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "isbns", dtype: List(Utf8),
|     metadata: {} }, Field { name: "isbn13", dtype: Utf8, metadata: {} }, Field
|     { name: "openAccessLinks", dtype: List(Struct([Field { name: "url", dtype:
|     Utf8, metadata: {} }, Field { name: "source", dtype: Utf8, metadata: {} },
|     Field { name: "contentSource", dtype: Utf8, metadata: {} }, Field { name:
|     "materialSpecified", dtype: Utf8, metadata: {} }])), metadata: {} }, Field
|     { name: "contributors", dtype: List(Struct([Field { name: "firstName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata: {} },
|     Field { name: "secondName", dtype: Struct([Field { name: "text", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "isPrimary", dtype: Boolean,
|     metadata: {} }, Field { name: "relatorCodes", dtype: List(Utf8), metadata:
|     {} }, Field { name: "contributorNotes", dtype: Struct([Field { name: "date",
|     dtype: List(Utf8), metadata: {} }, Field { name: "note", dtype: List(Utf8),
|     metadata: {} }]), metadata: {} }, Field { name: "nonPersonName", dtype:
|     Struct([Field { name: "text", dtype: Utf8, metadata: {} }]), metadata:
|     {} }, Field { name: "includes", dtype: List(Struct([Field { name: "title",
|     dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name: "uris", dtype:
|     List(Struct([Field { name: "type", dtype: Utf8, metadata: {} }, Field { name:
|     "uri", dtype: Utf8, metadata: {} }])), metadata: {} }])), metadata: {} },
|     Field { name: "machineReadableDate", dtype: Utf8, metadata: {} }, Field
|     { name: "seriesVolumes", dtype: List(Utf8), metadata: {} }, Field { name:
|     "mergedOclcNumbers", dtype: List(Utf8), metadata: {} }, Field { name:
|     "date", dtype: Utf8, metadata: {} }, Field { name: "language", dtype: Utf8,
|     metadata: {} }, Field { name: "subjects", dtype: List(Utf8), metadata: {} },
|     Field { name: "summaries", dtype: List(Utf8), metadata: {} }, Field { name:
|     "summary", dtype: Utf8, metadata: {} }, Field { name: "summariesText",
|     dtype: List(Utf8), metadata: {} }, Field { name: "oclc_number", dtype: Int64,
|     metadata: {} }, Field { name: "total_holding_count", dtype: Int64, metadata:
|     {} }, Field { name: "total_editions", dtype: Int64, metadata: {} }, Field
|     { name: "abstractsText", dtype: List(Utf8), metadata: {} }, Field { name:
|     "totalHoldingCount", dtype: Int64, metadata: {} }, Field { name: "holdings",
|     dtype: List(Int64), metadata: {} }, Field { name: "numPublicLibraries", dtype:
|     Int64, metadata: {} }, Field { name: "authors", dtype: List(Struct([Field
|     { name: "firstNameObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "flipNameOrder", dtype: Boolean,
|     metadata: {} }, Field { name: "lastNameObject", dtype: Struct([Field { name:
|     "", dtype: Null, metadata: {} }]), metadata: {} }, Field { name: "notes",
|     dtype: Utf8, metadata: {} }, Field { name: "primary", dtype: Boolean, metadata:
|     {} }, Field { name: "relatorList", dtype: Struct([Field { name: "relators",
|     dtype: List(Struct([Field { name: "code", dtype: Utf8, metadata: {} }, Field
|     { name: "term", dtype: Utf8, metadata: {} }])), metadata: {} }]), metadata:
|     {} }, Field { name: "subFieldsQueryString", dtype: Utf8, metadata: {} }, Field
|     { name: "type", dtype: Utf8, metadata: {} }])), metadata: {} }, Field { name:
|     "contentsObjects", dtype: List(Struct([Field { name: "note", dtype: Utf8,
|     metadata: {} }, Field { name: "noteObject", dtype: Struct([Field { name: "data",
|     dtype: Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }])), metadata: {} }, Field { name: "defaultCoverArtUrl",
|     dtype: Utf8, metadata: {} }, Field { name: "digitalGraphicRepresentation",
|     dtype: Utf8, metadata: {} }, Field { name: "disableAuthorLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks", dtype:
|     Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype: Boolean,
|     metadata: {} }, Field { name: "editionId", dtype: Utf8, metadata: {} },
|     Field { name: "editionSingletonEdition", dtype: Boolean, metadata: {} },
|     Field { name: "enhancedCollectionName", dtype: Utf8, metadata: {} }, Field
|     { name: "genreObjects", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }, Field { name: "local", dtype: Boolean, metadata: {} }])),
|     metadata: {} }, Field { name: "genres", dtype: List(Utf8), metadata: {} },
|     Field { name: "heldByLevel", dtype: Int64, metadata: {} }, Field { name:
|     "highlightedRecord", dtype: Struct([Field { name: "disableAuthorLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayCopyAndPasteCitations",
|     dtype: Boolean, metadata: {} }, Field { name: "displayDeepOpacLinks",
|     dtype: Boolean, metadata: {} }, Field { name: "displayOpacLink", dtype:
|     Boolean, metadata: {} }, Field { name: "enhancedCollectionName", dtype:
|     Utf8, metadata: {} }, Field { name: "heldByLevel", dtype: Int64, metadata:
|     {} }, Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field
|     { name: "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field
|     { name: "numberOfEditionIds", dtype: Int64, metadata: {} }, Field { name:
|     "numberOfOtherEditions", dtype: Int64, metadata: {} }, Field { name:
|     "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name: "titleObject",
|     dtype: Struct([Field { name: "", dtype: Null, metadata: {} }]), metadata:
|     {} }]), metadata: {} }, Field { name: "itemType", dtype: Utf8, metadata: {} },
|     Field { name: "itemTypeDisplay", dtype: Utf8, metadata: {} }, Field { name:
|     "labelAsUniqueIdentifier", dtype: Boolean, metadata: {} }, Field { name:
|     "lcNumber", dtype: Utf8, metadata: {} }, Field { name: "masterCallNumber",
|     dtype: Utf8, metadata: {} }, Field { name: "mediumCoverArtUrl", dtype: Utf8,
|     metadata: {} }, Field { name: "musicalPresentationStatement", dtype: Utf8,
|     metadata: {} }, Field { name: "numberOfEditionIds", dtype: Int64, metadata:
|     {} }, Field { name: "numberOfOtherEditions", dtype: Int64, metadata: {} },
|     Field { name: "openUrlContextObject", dtype: Utf8, metadata: {} }, Field
|     { name: "publishers", dtype: List(Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }])), metadata: {} }, Field { name: "remoteDatabase", dtype:
|     Boolean, metadata: {} }, Field { name: "source", dtype: Utf8, metadata:
|     {} }, Field { name: "sourceCollection", dtype: Utf8, metadata: {} }, Field
|     { name: "staffILLRequestUrl", dtype: Utf8, metadata: {} }, Field { name:
|     "subjectGroups", dtype: List(Struct([Field { name: "bibSubjects", dtype:
|     List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field { name:
|     "local", dtype: Boolean, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }, Field { name: "type", dtype: Utf8, metadata: {} },
|     Field { name: "unifiedData", dtype: Struct([Field { name: "data", dtype:
|     Utf8, metadata: {} }, Field { name: "private", dtype: Boolean, metadata:
|     {} }]), metadata: {} }, Field { name: "otherSource", dtype: Utf8, metadata:
|     {} }])), metadata: {} }, Field { name: "id", dtype: Utf8, metadata: {} },
|     Field { name: "isPromoted", dtype: Boolean, metadata: {} }, Field { name:
|     "label", dtype: Utf8, metadata: {} }, Field { name: "thesaurusType", dtype:
|     Utf8, metadata: {} }])), metadata: {} }, Field { name: "summariesObjectList",
|     dtype: List(Struct([Field { name: "data", dtype: Utf8, metadata: {} }, Field
|     { name: "private", dtype: Boolean, metadata: {} }])), metadata: {} }, Field
|     { name: "titleObject", dtype: Struct([Field { name: "data", dtype: Utf8,
|     metadata: {} }]), metadata: {} }, Field { name: "workCount", dtype: Int64,
|     metadata: {} }, Field { name: "workId", dtype: Utf8, metadata: {} }, Field
|     { name: "workSingletonIndicator", dtype: Boolean, metadata: {} }, Field { name:
|     "workSingletonWork", dtype: Boolean, metadata: {} }, Field { name: "not_found",
|     dtype: Int64, metadata: {} }]), metadata: {} }, Field { name: "html", dtype:
|     Utf8, metadata: {} }])}
|   Scan Tasks: [
|   {File {file:///mnt/bos-text/raw_data/worldcat.jsonl}}
|   ]

[universalmind303]

So after some investigation, it appears that we always concat json chunks back into a single chunk when reading, as seen here.

I'll go ahead and convert this discussion to an issue and we'll get it taken care of!