IO Streams in Standard Library #1620

garlic0x1 · 2025-08-20T17:21:50Z

This is a stream library pulled from https://github.com/garlic0x1/coalton-streams. I think it ought to be in the standard library since most basic IO will need something like this and I think the current file streams are insufficient.

TODO (maybe in later PRs?):

Add better error types for the base methods
Dedupe coalton-library/file

One cost of this PR is adding flexi-streams as a dependency to the standard library.

shirok

I have some ideas to explore, but I think this is good to be merged and then we exlore later.

One major issue is whether we want parameterized types on elements; I don't think we'll ever have other element types (non-octet byte streams are mostly useless with customizable endianness support). In that case, we may expose TextualStream and BinaryStream externally, and keep the parameterized stream only for internal implementaiton. But that can come in future.

library/stream.lisp

stylewarning

I am blocking this on my review. I will try to finish my review in the next couple of days.

garlic0x1 · 2025-09-12T02:28:06Z

Performance is not great when reading millions of bytes compared to a plain CL stream. Here are some ways to improve:

Abandon peek and unread operations and get rid of flexi-streams
Offer *-unchecked operations that don't cons a Result

With these changes we can get as good or better performance compared to CL

* speeeed * fundeps are fun

garlic0x1 · 2025-09-12T15:49:46Z

Added some functions included in #1644, better to merge that first with its included tests I think.

I need to clean up and benchmark write operations.

garlic0x1 · 2025-09-13T08:55:14Z

Setup

(ql:quickload :coalton)
(ql:quickload :coalton-streams)
(defpackage #:stream-benchmark
  (:use #:coalton #:coalton-prelude)
  (:local-nicknames
   (#:file #:coalton-streams/file)
   (#:stream #:coalton-library/stream)))

Read Functions

(coalton-toplevel
  (define (read-rand-byte bytes) 
    (file:with-input-file "/dev/random"
      (fn (stream)
        (the (stream:InputStream U8) stream)
        (coalton-library/experimental:dotimes (_ bytes)
          (stream:read stream)))))

  (define (read-unchecked-rand-byte bytes) 
    (file:with-input-file "/dev/random"
      (fn (stream)
        (the (stream:InputStream U8) stream)
        (coalton-library/experimental:dotimes (_ bytes)
          (stream:read-unchecked stream)))))

  (define (read-unchecked-peekable-rand-byte bytes)
    (file:with-input-file "/dev/random"
      (fn (stream)
        (the (stream:InputStream U8) stream)
        (let stream = (stream:make-peekable stream))
        (coalton-library/experimental:dotimes (_ bytes)
          (stream:read-unchecked stream))))))

(cl:defun read-rand-byte-cl (bytes)
  (uiop:with-input-file (s "/dev/random" :element-type '(cl:unsigned-byte 8))
    (cl:dotimes (_ bytes)
      (cl:read-byte s))))

(cl:defun read-rand-byte-cl-flexi (bytes)
  (uiop:with-input-file (s "/dev/random" :element-type '(cl:unsigned-byte 8))
    (cl:let ((s (flexi-streams:make-flexi-stream s)))
      (cl:dotimes (_ bytes)
        (cl:read-byte s)))))

Write Functions

(coalton-toplevel
  (define (write-bytes bytes)
    (file:with-output-file* "/dev/null" file:Append file:DNEError
      (fn (stream)
        (the (stream:OutputStream U8) stream)
        (coalton-library/experimental:dotimes (_ bytes)
          (stream:write stream 97)))))
  (define (write-bytes-unchecked bytes)
    (file:with-output-file* "/dev/null" file:Append file:DNEError
      (fn (stream)
        (the (stream:OutputStream U8) stream)
        (coalton-library/experimental:dotimes (_ bytes)
          (stream:write-unchecked stream 97))))))

(cl:defun write-bytes-cl (bytes)
  (uiop:with-output-file (s "/dev/null" :element-type '(cl:unsigned-byte 8) :if-exists :append)
    (cl:dotimes (_ bytes)
      (cl:write-byte 97 s))))

Read Benchmarks

(cl:let ((cl:*trace-output* cl:*standard-output*))
  (cl:format cl:t "Reading 100 Million Bytes:~%")
  (cl:print :CL)
  (cl:time (read-rand-byte-cl        100000000))
  (cl:print :FLEX)
  (cl:time (read-rand-byte-cl-flexi  100000000))
  (cl:print :COAL-OPTIONAL)
  (cl:time (read-rand-byte           100000000))
  (cl:print :COAL-UNCHECKED)
  (cl:time (read-unchecked-rand-byte 100000000))
  (cl:print :COAL-UNCHECKED-PEEKABLE)
  (cl:time (read-unchecked-peekable-rand-byte 100000000)))

Reading 100 Million Bytes:

:CL 
Evaluation took:
  0.656 seconds of real time
  0.656323 seconds of total run time (0.468481 user, 0.187842 system)
  100.00% CPU
  154,112 bytes consed


:FLEX 
Evaluation took:
  3.738 seconds of real time
  3.738573 seconds of total run time (3.553288 user, 0.185285 system)
  100.03% CPU
  131,040 bytes consed


:COAL-OPTIONAL 
Evaluation took:
  1.344 seconds of real time
  1.344823 seconds of total run time (1.162907 user, 0.181916 system)
  100.07% CPU
  131,040 bytes consed


:COAL-UNCHECKED 
Evaluation took:
  0.437 seconds of real time
  0.437596 seconds of total run time (0.259091 user, 0.178505 system)
  100.23% CPU
  196,592 bytes consed


:COAL-UNCHECKED-PEEKABLE 
Evaluation took:
  1.805 seconds of real time
  1.805170 seconds of total run time (1.620884 user, 0.184286 system)
  100.00% CPU
  131,040 bytes consed

Write Benchmarks

(cl:let ((cl:*trace-output* cl:*standard-output*))
  (cl:format cl:t "Writing 100 Million Bytes:~%")
  (cl:print :CL)
  (cl:time (write-bytes-cl        100000000))
  (cl:print :COAL-BOOLEAN)
  (cl:time (write-bytes           100000000))
  (cl:print :COAL-UNCHECKED)
  (cl:time (write-bytes-unchecked 100000000)))

Reading 100 Million Bytes:

:CL 
Evaluation took:
  0.568 seconds of real time
  0.568569 seconds of total run time (0.566969 user, 0.001600 system)
  100.18% CPU
  0 bytes consed


:COAL-BOOLEAN 
Evaluation took:
  1.385 seconds of real time
  1.385121 seconds of total run time (1.381807 user, 0.003314 system)
  100.00% CPU
  0 bytes consed


:COAL-UNCHECKED 
Evaluation took:
  0.394 seconds of real time
  0.394355 seconds of total run time (0.393289 user, 0.001066 system)
  100.00% CPU
  0 bytes consed

stylewarning · 2025-09-13T13:32:11Z

Awesome benchmarks. I'm surprised Coalton is faster than Lisp.

stylewarning

I think there are some critical design questions to answer. Perhaps the biggest one is whether the stream types offered should be parameterized on type.

library/stream.lisp

stylewarning · 2025-09-13T15:33:16Z

library/stream.lisp

+(coalton-toplevel
+  (repr :native cl:stream)
+  (define-type (InputStream :elt)
+    "A stream that can be read from.")


Do we want to promise that these are Lisp streams so that they can be passed into lisp forms?

Question remains. If so, we should document it in the type here, and also in the interop doc.

Doesn't the (repr :native cl:stream) promise this?

library/stream.lisp

stylewarning · 2025-09-13T15:40:21Z

library/stream.lisp

+Signals a condition on error."
+     (:stream :elt -> :elt)))
+
+  (define-class (IntoPeekable :stream :elt :peekablestream (:stream -> :peekablestream))


I'm not sure we should make this a type class; we might provide some functions to take our offered stream data types and turn them into peekable streams. This class suggests we will have all sorts of (stream, peekable-stream) pairs to extend with in the future.

Any new integer-type stream might want to implement make-peekable

library/stream.lisp

stylewarning · 2025-09-13T15:53:07Z

library/stream.lisp

+  (define (read stream)
+    "Consume 1 element from a stream."
+    (catch (Some (read-unchecked stream))
+      (_ None)))


Ideally we would limit to the cl:end-of-file condition.

I don't think there is a way to catch specific CL conditions yet

library/stream.lisp

stylewarning · 2025-09-13T15:54:48Z

library/stream.lisp

+;; High Level Functions
+;;
+
+(coalton-toplevel


I don't think the below functions should be a part of this module.

What was the resolution on this?

I removed the tokenizing read functions with the understanding that boolean predicate read-to functions were still desired

stylewarning · 2025-09-13T16:01:48Z

library/stream.lisp

+
+(coalton-toplevel
+  (declare stdout (Unit -> OutputStream :elt))
+  (define (stdout)


This is not type safe.

We don't know what kind of stream *standard-output* will be bound to when this call is made.

Even if we did, the function is presenting itself as polymorphic on the stream element type.

The standard is kind of bad when it comes to specifying streams, but we can be guaranteed *terminal-io* won't be re-bound, and we can basically guarantee all such streams will be character streams.

I think it is fine now?

:elt below needs to be changed.

…ed function feels right in coalton...

stylewarning · 2025-10-06T20:19:44Z

library/stream.lisp

+(coalton-toplevel
+  (repr :native cl:stream)
+  (define-type (InputStream :elt)
+    "A stream that can be read from.")


Question remains. If so, we should document it in the type here, and also in the interop doc.

stylewarning · 2025-10-06T20:21:33Z

library/stream.lisp

+    (stream (IOStream   :elt))
+    (buffer (vec:Vector :elt)))
+
+  (define-class (Closable :stream :elt)


I see :elt as superfluous here. Just :stream should work.

stylewarning · 2025-10-06T20:22:14Z

library/stream.lisp

+     "Consume 1 element from a stream.
+
+Signals a condition on error."
+     (:stream :elt -> :elt))


I would have :stream -> :elt, and not have :stream be higher-kinded.

stylewarning · 2025-10-06T20:22:28Z

library/stream.lisp

+     "Consume `n` elements from a stream and destructively fill array."
+     (:stream :elt -> array:LispArray :elt -> UFix)))
+
+  (define-class (Writable :stream :elt)


Same comment as Readable.

stylewarning · 2025-10-06T20:23:26Z

library/stream.lisp

+     "Flush buffered output."
+     (:stream :elt -> Unit)))
+
+  (define-class (Readable :stream :elt => Peekable :stream :elt)


Same comment as Readable.

stylewarning · 2025-10-06T20:33:19Z

library/stream.lisp

+    (define (unread stream elt)
+      (vec:push! elt (.buffer stream))
+      Unit)
+    (define (peek-unchecked stream)


What is the resolution here?

stylewarning · 2025-10-06T20:33:44Z

library/stream.lisp

+            (vec:push! elt (.buffer stream))
+            elt))))
+
+  (define-instances (Readable (PeekableInputStream U8) (PeekableIOStream U8))


Does this need to be specialized on U8?

stylewarning · 2025-10-06T20:34:10Z

library/stream.lisp

+         (vec:clear! (.buffer stream))
+         result))))
+
+  (define-instance (Writable PeekableIOStream U8)


Same question for everything below. Does it need to be specialized?

Probably not, although right now I can't figure out how to define the instances to do it correctly

stylewarning · 2025-10-06T20:35:32Z

library/stream.lisp

+;; High Level Functions
+;;
+
+(coalton-toplevel


What was the resolution on this?

stylewarning · 2025-10-06T20:35:53Z

library/stream.lisp

+
+(coalton-toplevel
+  (declare stdout (Unit -> OutputStream :elt))
+  (define (stdout)


:elt below needs to be changed.

garlic0x1 · 2025-10-08T19:55:03Z

Some improvements made, I am struggling to figure out how to reduce the kind arity though

garlic0x1 added 4 commits August 20, 2025 13:16

stdlib streams

55da20b

stream tests

7cb802f

some docstrings

14ecdc4

clean up some silly lambdas

ccb746b

garlic0x1 marked this pull request as ready for review August 20, 2025 21:55

shirok approved these changes Aug 22, 2025

View reviewed changes

library/stream.lisp Outdated Show resolved Hide resolved

library/stream.lisp Outdated Show resolved Hide resolved

stylewarning requested changes Aug 22, 2025

View reviewed changes

garlic0x1 marked this pull request as draft September 12, 2025 13:42

Stdlib streams fast (#1646)

07ab31a

* speeeed * fundeps are fun

garlic0x1 added 3 commits September 12, 2025 19:17

clean up

0006af3

docstrings

13391a7

more documentation

94f1be0

nickname in coalton-user package

8479ea2

garlic0x1 marked this pull request as ready for review September 13, 2025 14:40

useless progn

efa0035

stylewarning requested changes Sep 13, 2025

View reviewed changes

garlic0x1 added 5 commits September 18, 2025 14:40

k&r streams

a59ec63

inline methods

6363ca0

remove token stuff

8a6cc86

read-to drop-to outer match

7443a93

inline unix stream fns, should these even be functions? idk, an inlin…

cd645da

…ed function feels right in coalton...

stylewarning requested changes Oct 6, 2025

View reviewed changes

garlic0x1 added 2 commits October 8, 2025 22:49

stylewarnings

e117393

implement writable for peekable char streams

6a13157

IO Streams in Standard Library #1620

Are you sure you want to change the base?

IO Streams in Standard Library #1620

Uh oh!

Conversation

garlic0x1 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shirok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stylewarning left a comment

Choose a reason for hiding this comment

Uh oh!

garlic0x1 commented Sep 12, 2025

Uh oh!

garlic0x1 commented Sep 12, 2025

Uh oh!

garlic0x1 commented Sep 13, 2025

Table of Contents

Setup

Read Functions

Write Functions

Read Benchmarks

Write Benchmarks

Uh oh!

stylewarning commented Sep 13, 2025

Uh oh!

stylewarning left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

garlic0x1 Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

garlic0x1 commented Aug 20, 2025 •

edited

Loading

garlic0x1 Oct 8, 2025 •

edited

Loading