I've added training data to a ControlNet, but the colors are wrong?

[brook-seaton]

I've tried adding more training data to a Canny ControlNet (trained using Huggingface on JuggernautXL-v8, but I'm getting weird color images produced when I use the Fooocus styles:
https://drive.google.com/drive/u/0/folders/1PbKzj8-Rw2RJkeYzHxj09x-UgPz_8Ntx

When I use Fooocus style the color changes to weird hues, but when I take the words from https://github.com/lllyasviel/Fooocus/blob/main/sdxl_styles/sdxl_styles_fooocus.json and paste it into the prompt or negative prompt, the color does not change.

I'm thinking it might have something to do with this line of code, because without the styles, the boolean "use_expansion" is set to true and "use_style" is set to false:

Fooocus/modules/async_worker.py

Lines 691 to 741 in 59f183a

	if use_style:
	placeholder_replaced = False
	for j, s in enumerate(task_styles):
	if s == random_style_name:
	s = get_random_style(task_rng)
	task_styles[j] = s
	p, n, style_has_placeholder = apply_style(s, positive=task_prompt)
	if style_has_placeholder:
	placeholder_replaced = True
	positive_basic_workloads = positive_basic_workloads + p
	negative_basic_workloads = negative_basic_workloads + n
	if not placeholder_replaced:
	positive_basic_workloads = [task_prompt] + positive_basic_workloads
	else:
	positive_basic_workloads.append(task_prompt)
	negative_basic_workloads.append(task_negative_prompt) # Always use independent workload for negative.
	positive_basic_workloads = positive_basic_workloads + task_extra_positive_prompts
	negative_basic_workloads = negative_basic_workloads + task_extra_negative_prompts
	positive_basic_workloads = remove_empty_str(positive_basic_workloads, default=task_prompt)
	negative_basic_workloads = remove_empty_str(negative_basic_workloads, default=task_negative_prompt)
	tasks.append(dict(
	task_seed=task_seed,
	task_prompt=task_prompt,
	task_negative_prompt=task_negative_prompt,
	positive=positive_basic_workloads,
	negative=negative_basic_workloads,
	expansion='',
	c=None,
	uc=None,
	positive_top_k=len(positive_basic_workloads),
	negative_top_k=len(negative_basic_workloads),
	log_positive_prompt='\n'.join([task_prompt] + task_extra_positive_prompts),
	log_negative_prompt='\n'.join([task_negative_prompt] + task_extra_negative_prompts),
	styles=task_styles
	))
	if use_expansion:
	if advance_progress:
	current_progress += 1
	for i, t in enumerate(tasks):
	progressbar(async_task, current_progress, f'Preparing Fooocus text #{i + 1} ...')
	expansion = pipeline.final_expansion(t['task_prompt'], t['task_seed'])
	print(f'[Prompt Expansion] {expansion}')
	t['expansion'] = expansion
	t['positive'] = copy.deepcopy(t['positive']) + [expansion] # Deep copy.

I've trained the ControlNet with 50% regularizaiton images as stock LAION images, and tried again with a fixed seed and inferenced images.

Any ideas what is causing this issue, or what I can do to fix this issue?