CookieJar filter_cookies does not preserve expiration date

[bwright1558]

Long story short

Using CookieJar.filter_cookies() does not preserve the expiration date of the returned cookies.

Expected behaviour

Return the full cookie Morsels that have the expires, domain, path, etc. fields.

Actual behaviour

The expires field is not included in the cookie Morsels returned from CookieJar.filter_cookies().

Steps to reproduce

Here's a full working example to demonstrate. It's a contrived example, but it gets the point across. Iterating through cookie_jar gives me the full Morsel objects with expiration. But iterating through filtered_cookies does not contain the expiration.

from http.cookies import SimpleCookie

import yarl
from aiohttp import CookieJar


cookies = SimpleCookie()
cookies.load('browser_session=secret; Domain=.test.com; Path=/; Expires=Tue, 30-Jul-69 14:47:41 GMT; HttpOnly')

cookie_jar = CookieJar()
cookie_jar.update_cookies(cookies)

for cookie in cookie_jar:
    print(f'{cookie.value!r} expires {cookie["expires"]!r}')

url = yarl.URL('http://www.test.com')
filtered_cookies = cookie_jar.filter_cookies(url)
for _, cookie in filtered_cookies.items():
    print(f'{cookie.value!r} expires {cookie["expires"]!r}')

Your environment

I'm using the latest stable release of aiohttp (3.5.4).

[asvetlov]

expire makes sense for server-side Set-Cookie header.
filter_cookies() returns a data for generating client-side Cookie header. This header doesn't contain an expiration attribute: if the cookie is expired it is not sent to the server at all.
Why do you want this attribute in filter_cookies() result?

[bwright1558]

filter_cookies() may be the wrong way for me to go, but I'll describe what I am ultimately trying to do. I am automating signing into a website. Eventually, my session will expire and the program will have to sign in again. I figure I can use the expires value from the website's cookies to estimate when my program needs to sign back into the website.

[asvetlov]

Do you have an idea of how the desired API should look like?
How other libraries solve your problem?

[bwright1558]

Below is more or less how I expect filter_cookies to work. Why does filter_cookies not just return the original Morsels (or a full copy of the originals)?

def filter_cookies(self, request_url: URL=URL()) -> 'BaseCookie[str]':
    """Returns this jar's cookies filtered by their attributes."""
    self._do_expiration()
    if not isinstance(request_url, URL):
        warnings.warn("The method accepts yarl.URL instances only, got {}"
                        .format(type(request_url)),
                        DeprecationWarning)
        request_url = URL(request_url)
    filtered = SimpleCookie()
    hostname = request_url.raw_host or ""
    is_not_secure = request_url.scheme not in ("https", "wss")

    for cookie in self:
        name = cookie.key
        domain = cookie["domain"]

        # Send shared cookies
        if not domain:
            # Assign original Morsel (or a full copy) to `filtered[name]`.
            filtered[name] = cookie
            continue

        if not self._unsafe and is_ip_address(hostname):
            continue

        if (domain, name) in self._host_only_cookies:
            if domain != hostname:
                continue
        elif not self._is_domain_match(domain, hostname):
            continue

        if not self._is_path_match(request_url.path, cookie["path"]):
            continue

        if is_not_secure and cookie["secure"]:
            continue
        
        # Assign original Morsel (or a full copy) to `filtered[name]`.
        filtered[name] = cookie

    return filtered

[vladkolotvin]

I have same issue and case. Automating signing into a website. I expect to get full cookie object from filter. Current behaviour is a bug.

[spenweb]

I feel there should be some public method to get the cookies (aka morsels). At the moment filter_cookies seems to be the closest public method for this purpose, but seems to only be meant for creating cookie headers being sent by a client. I also need to somehow get the full cookie (morsel) with all of its attributes (at least expires and max-age) so that I can reuse these cookies in another cookie jar.

Here is my current work-around:

from aiohttp import ClientSession
from http.cookies import SimpleCookie

session = ClientSession()

# Some requests later...

morsels = {
    name: morsel
    for (domain, path), cookie in session._cookie_jar._cookies.items()
    for name, morsel in cookie.items()
    if name
}

[bdraco]

Something like this might be OK

diff --git a/aiohttp/cookiejar.py b/aiohttp/cookiejar.py
index 158d6d103..ba444da89 100644
--- a/aiohttp/cookiejar.py
+++ b/aiohttp/cookiejar.py
@@ -11,6 +11,7 @@ import time
 import warnings
 from collections import defaultdict
 from http.cookies import BaseCookie, Morsel, SimpleCookie
+from types import MappingProxyType
 from typing import (
     DefaultDict,
     Dict,
@@ -122,6 +123,16 @@ class CookieJar(AbstractCookieJar):
     def quote_cookie(self) -> bool:
         return self._quote_cookie
 
+    @property
+    def cookies(self) -> MappingProxyType[Tuple[str, str], SimpleCookie]:
+        """Return the cookies stored in this jar."""
+        return MappingProxyType(self._cookies)
+
+    @property
+    def host_only_cookies(self) -> FrozenSet[Tuple[str, str]]:
+        """Return the host-only cookies stored in this jar."""
+        return frozenset(self._host_only_cookies)
+
     def save(self, file_path: PathLike) -> None:
         file_path = pathlib.Path(file_path)
         with file_path.open(mode="wb") as f:

The downside is that it would expose the SimpleCookie objects to the caller, and they are mutable. Any changes to them would likely break internals.

Another option would be to subclass CookieJar with the above, but it would still carry the same risk that the protected names would change under the hood

[Br1an67]

I'd like to work on this. Adding read-only cookies and host_only_cookies properties to CookieJar as suggested by @bdraco — cookies returns a MappingProxyType view of the internal storage, host_only_cookies returns a frozenset.

PR: #12174