airflow-webserver crashes if log files are larger than webserver available RAM

[getaaron]

Apache Airflow version

2.6.0

What happened

I tried to view a 400MB dag log file in the web server UI.

The available RAM on the Kubernetes pod is 50MB.

The Airflow webserver crashed.

What you think should happen instead

Either

1. The log file should be truncated and an error shown that there's not enough RAM, or
2. The log file should not be fully realized into RAM and should be streamed to frontend instead

How to reproduce

1. create airflow on k8s with 50Mi requested pod sizes
2. make a dag that generates 400 megabyte log files
3. attempt to view the log file in the airflow UI

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow==2.6.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-oracle==3.0.0
apache-airflow-providers-slack==7.2.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[getaaron]

Haven't worked in airflow before, I could potentially open a PR if someone can describe what the fix should be

[potiuk]

Haven't worked in airflow before, I could potentially open a PR if someone can describe what the fix should be

I think you pretty well described it :) .

What you think should happen instead
Either

The log file should be truncated and an error shown that there's not enough RAM, or
The log file should not be fully realized into RAM and should be streamed to frontend instead

One of those seems to be a good solution, just the place where airflow loads the file in memory has to be find and one of the solutions above appied (there are many different ways of loading logs - depending on the view you used and state of the task and log configuration (for example if there is remote logging or streaming it is different than when local logs are used and different when celery logs are used).

I marked it as good-first-issue for you whoever would like to fix it , it's likely a bit more than first issue though and requires quite a bit of digging in the code, but if someone knows a bit about Python/webserver/flask it should not be that difficult to find the right place.

[getaaron]

@SkandaPrasad-S I’m not picking it up so you should feel free to grab it

…

On Mon, May 8, 2023 at 10:25 AM SkandaPrasad-S ***@***.***> wrote: Is anyone picking this up? I can take this up as as I have experience in flask and webserver. Can I give it a go? — Reply to this email directly, view it on GitHub <#31105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGAYSOFUBC36HKFOBL5PUTXFEF5BANCNFSM6AAAAAAXXSKOG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[killua1zoldyck]

@potiuk if no one's working on this and if this has not been fixed yet I can work on this.

[potiuk]

Feel free!

[killua1zoldyck]

@potiuk The current log-streaming logic pulls the whole file into memory before moving to the specific log_pos based on the metadata after combining all the logs and sorting them based on timestamp.

airflow/airflow/utils/log/file_task_handler.py

Lines 334 to 350 in 24e3d6c

	logs = "\n".join(
	_interleave_logs(
	*local_logs,
	*remote_logs,
	*(executor_logs or []),
	*served_logs,
	)
	)
	log_pos = len(logs)
	messages = "".join([f"*** {x}\n" for x in messages_list])
	end_of_log = ti.try_number != try_number or ti.state not in [State.RUNNING, State.DEFERRED]
	if metadata and "log_pos" in metadata:
	previous_chars = metadata["log_pos"]
	logs = logs[previous_chars:] # Cut off previously passed log test as new tail
	out_message = logs if "log_pos" in (metadata or {}) else messages + logs
	return out_message, {"end_of_log": end_of_log, "log_pos": log_pos}

So, I don't think we can change the logic to make it streaming. Is my thinking correct?

[potiuk]

I have not looked at it befor so I have no idea. I was kind of hoping that whoever takes on the task will be able to propose the solution - possibly involving chnaging the way how it is done.