Add content type negotiation for text types

[vincentjames501]

Most other clj based http clients (clj-http, http-kit, hato, etc) either default to returning a String for :body or attempt to negotiate the content type. When they make a request to something like GET https://apple.com or GET https://google.com, the :body is returned as text instead of an InputStream. It would be nice when a response's content-type is text/plain, text/html, text/css, etc for the body to be automatically converted to a string similar to other content negotiations. I'd argue any content type that is text/* should be auto coerced to a string w/ proper encoding. Thoughts? I was actually surprised muuntaja doesn't do this automatically though we could add additional formats like so:

(defn decoder [_]
  (reify
    mf/Decode
    (decode [_ data charset]
      (slurp data :encoding charset))))

(defn encoder [_]
  (reify
    mf/EncodeToBytes
    (encode-to-bytes [_ data charset]
      (.getBytes ^String data ^String charset))
    mf/EncodeToOutputStream
    (encode-to-output-stream [_ data charset]
      (fn [^OutputStream output-stream]
        (.write output-stream (.getBytes
                                ^String data
                                ^String charset))))))

(def text-format
  (mf/map->Format
    {:name    "text/plain"
     :decoder [decoder]
     :encoder [encoder]}))

(def muuntaja (m/create (-> m/default-options
                            (update :formats assoc "text/plain" text-format)
                            (assoc :return :output-stream))))

It would be nice to support :as a bit better as well. For example adding support for :byte-array, :text/:string, :input-stream etc can be pretty nice where the default if content-negotiation couldn't be done automatically be a string?

[vincentjames501]

Hmmm... So this code:

(some? as)
              (let [decoded (mun/format-stream muuntaja as body)]
                (assoc response :body decoded))

That's in wrap-decode-responses may also not do quite the right thing in all cases. For example if I set :as to :json, it won't use the charset in the response header and will always use "UTF-8" which may not be correct. It should try to honor the source content encoding/charset when provided.

[RutledgePaulV]

Yes, agreed that text/* should be read to a string by default and should use the charset in the response header if present. I think adding other muuntaja formats as you suggested is the right approach.

[vincentjames501]

@RutledgePaulV , the more I think about this, the more I think the way most other clj-based http clients work is best to avoid accidentally shooting yourself in the foot. It's going to be WAYYY too easy to be expecting a json response, then you magically get something else back like a 500/503 (because #internet and you may not actually get a reliable content-type in the response with poorly written servers) and bam, you've got an input stream now and a resource leak to boot because you're not closing your streams. I think relying on text/* like I mentioned above is actually a bad idea because SOOO many things that are really text have content types application/xml, application/xhtml+xml, etc and it's going to be impossible to maintain a complete list. I really think we should follow the other patterns established in the community and do the following:

:text coercion is the default. This is sure to consume the stream and close it to avoid resource leaks.
:input-stream coercion for when you know you are dealing with something large and YOU NEED to manage its consumption.
:auto is an option to use muuntaja content negotiation, if it fails, it returns text like other clients
:json/:clojure/:etc to force coercion to a specific type, text otherwise