pandoc --citeproc throws parse exception error on pandoc generated bibliography

[amunn]

I'm using pandoc to generate an html bibliography from a .bib file. It's a two step process:

pandoc lab-publications.bib -s -f biblatex -t markdown -o lab-pubs.txt
pandoc --citeproc --csl apa-cv lab-pubs.txt -o tmp-bib.html

This was working fine, but suddenly has stopped working, and I get the following error on running the second step:

YAML parse exception at line 0, column 0:
control characters are not allowed

I'm assuming that there's something in the .bib file so that the output of the first step is ill-formed, but that output looks ok AFAICS. I don't really know how to go about debugging this, short of trimming the bib entries one by one. It's also strange that the the first step always works, so I would have thought that the output would be valid YAML, since pandoc generated it.

Any suggestions on how to solve this?

The actual .bib file can be found here.

[amunn]

So it turns out that one of the imported .bib entries had the character â�� in it, which was fine for step 1 but bad for step 2. It would be nice if such characters could be flagged in the conversion from bibtex to YAML.

[jgm]

Can you create a YAML from just that one bib entry, so I can look at the generated YAML?

[amunn]

Here's the .bib file entry: (this is what I get from pasting the DOI into BibDesk, which retrieves the entry automatically)

@article{Miller_2012,
	author = {Miller, Karen L. and Schmitt, Cristina},
	date-added = {2024-09-04 12:21:52 -0400},
	date-modified = {2024-09-04 12:21:52 -0400},
	doi = {10.1080/10489223.2012.685026},
	issn = {1532-7817},
	journal = {Language Acquisition},
	month = jun,
	number = {3},
	pages = {223�261},
	publisher = {Informa UK Limited},
	title = {Variable Input and the Acquisition of Plural Morphology},
	url = {http://dx.doi.org/10.1080/10489223.2012.685026},
	volume = {19},
	year = {2012},
	bdsk-url-1 = {http://dx.doi.org/10.1080/10489223.2012.685026}}

And here's the generated YAML:

---
nocite: "[@*]"
references:
- author:
  - family: Miller
    given: Karen L.
  - family: Schmitt
    given: Cristina
  container-title: Language Acquisition
  doi: 10.1080/10489223.2012.685026
  id: Miller_2012
  issn: 1532-7817
  issue: 3
  issued: 2012-06
  page: 223�261
  publisher: Informa UK Limited
  title: Variable input and the acquisition of plural morphology
  type: article-journal
  url: "http://dx.doi.org/10.1080/10489223.2012.685026"
  volume: 19
---

[jgm]

Or, possibly, your input is properly UTF-8 encoded but contains a control character (e.g. ascii 07, BEL). Maybe GH strips this out when it displays HTML, so it was removed in pasting. In any case, you certainly shouldn't have such a thing in your bibliography, but that is not really a problem in pandoc.

[amunn]

I do understand that the problem is in the .bib file, initially, but the weird part is that pandoc deals with the .bib file without problem, but then the generated YAML is ill formed.

[jgm]

Here's a hex dump:

[jgm]

The garbled bit has these bytes: c3 a2 c2 80 c2 93.
This converts to code points: 226 (latin small a circumflex), 128 (PAD), 147 (STS = set transmit state).
I assume the 147 is the control character.

I don't think I'm going to worry about this. It's an edge case, obviously a corrupted input.

[amunn]

Thanks, I understand. I'll raise the issue with the BibDesk people to see if it's a problem on their end, or a problem with the data itself that they scrape from the DOI server, since the bad .bib file entry is autogenerated from pasting a DOI into BibDesk.