Python: Modernize File Not Always Closed query

[joefarebrother]

Rewrites py/file-not-closed query to not rely on pointsTo analysis.

Reviewing per-commit may be helpful.

[github-actions]

QHelp previews:

python/ql/src/Resources/FileNotAlwaysClosed.qhelp

File is not always closed

When a file is opened, it should always be closed.

A file opened for writing that is not closed when the application exits may result in data loss, where not all of the data written may be saved to the file. A file opened for reading or writing that is not closed may also use up file descriptors, which is a resource leak that in long running applications could lead to a failure to open additional files.

Recommendation

Ensure that opened files are always closed, including when an exception could be raised. The best practice is often to use a with statement to automatically clean up resources. Otherwise, ensure that .close() is called in a try...except or try...finally block to handle any possible exceptions.

Example

In the following examples, in the case marked BAD, the file may not be closed if an exception is raised. In the cases marked GOOD, the file is always closed.

def bad():
    f = open("filename", "w")
    f.write("could raise exception") # BAD: This call could raise an exception, leading to the file not being closed.
    f.close()


def good1():
    with open("filename", "w") as f:
        f.write("always closed") # GOOD: The `with` statement ensures the file is always closed.

def good2():
    f = open("filename", "w")
    try:
       f.write("always closed")
    finally:
        f.close() # GOOD: The `finally` block always ensures the file is closed.
   

References

• Python Documentation: Reading and writing files.
• Python Language Reference: The with statement, The try statement.
• Python PEP 343: The "with" Statement.
• Common Weakness Enumeration: CWE-772.

[tausbn]

Good stuff! I've added a couple of comments, though many of them are more me musing about how this fits in the grander scheme of things. In the interest of expedience, I would be happy to leave some of the more broad changes I suggest as potential future work. (Once we have more of these quality queries ported over, we'll probably have a better feel for what would make sense as a framework for this sort of thing.)

[tausbn]

Is a type tracker actually needed here? As far as I can tell FileOpenSource only contains API graph nodes at the moment, so I would hope that the type tracking done within the API graph calculation would be sufficient.

[tausbn]

By e do you mean raises?

[tausbn]

This extension of local flow makes me wonder if it would make more sense to rewrite this part of the query as a proper data-flow query (with an additional step for file wrapper calls). My main worry is that calculating the fileLocalFlow relation might result in bad performance.

[tausbn]

Unless I've misread the rest of the code, source will in fact always be a FileOpen instance. If this is true, it might make sense to specialise the argument to that class (a kind of manual "magic"), as this would certainly reduce the size of the fileLocalFlow predicate (assuming the compiler hasn't already figured that this kind of magic is available).

[tausbn]

I'm slightly puzzled by this very weak restriction. Do we really not require anything of the class that's wrapping it?

[tausbn]

This feels like it's a subset of a more generic concept of "returning some structure containing a thing of interest". For instance, what if the file object is put in a dict that's returned?
If we rewrite the query to use data-flow, I could see this potentially being more widely useful as a standard set of additional flow steps.

[tausbn]

Maybe move the raises argument into this exists (as it doesn't seem to be used anywhere outside of this predicate)?