merge rexport, pushshift and gdpr reddit data

[purarue]

Was able to get pushshift as mentioned in the README working to export old comments. Thought I'd mention it here.

It's possible to use pushshift to get data further back, but I'm not sure if it should be part of this project, since some of the older comments don't have the same JSON structure, I can only assume pushshift is getting data from multiple sources the further it goes back. It requires some normalization, like here and here.

The only data I was missing due to the 1000 limit on queries from using rexport were comments. It exported the last 1000 but I have about 5000 on reddit in total.

Regarding HPI:

Wrote a simple package to request/save that data, with a dal (whose PComment NamedTuple has similar @property attributes to rexports DAL), and a merge function, and now:

In [15]: from my.reddit import comments, _dal, pushshift_comments

In [16]: len(list(_dal().comments())) # from dal.py in rexport
Out[16]: 999

In [17]: len(list(pushshift_comments())) # from pushshift
Out[17]: 4891

In [18]: len(list(comments())) # merged data, using utc_time to remove duplicates
Out[18]: 4893

In [19]: comments
Out[19]: <function my.reddit.comments() -> Iterator[Union[rexport.dal.Comment, pushshift_comment_export.dal.PComment]]>

Its possible that one could write enough @property wrappers to handle the differences in the JSON representations of old pushshift data, unsure if thats something you want to pursue here.

[karlicoss]

Nice job! Just checked it out too (although I don't have as many comments to need it), and it works!

Wondering, how do you approach Python entry points/executable scripts while developing?

E.g. one could either:

• make a symlink __main__.py -> __init__.py, then you can run via python3 -m pushshift_comment_export

• make __init__.py executable and run pushshift_comment_export/__init__.py. Looks kinda odd, but nothing wrong with it in theory, although I tend not to name the script __init__.py in such cases

These have a benefit of not messsing with installed packages at all, which I feel is nice, but maybe not that big of a deal?
There is a third option, which I believe you're using:

• install in 'editable' mode (pip3 install --user .). The only annoying thing, imo, is that it messes with ~/.local/bin.

  Ultimately though, I guess installing (whether editable or normal) is preferable. I kind of regret messing with dynamic imports like this, it only adds complexity, so in the future I'll probably get rid of them in favor of the users installing packages properly. Perhaps in a dedicated HPI virtualenv to avoid cluttering user packages.

[karlicoss]

Regarding merge function: looks good!
It would be cool to merge into upstream too, but my only concern with the current way (keeping everything in reddit.py) would be rexport and pushshift_comment_export imports. If someone only uses one of these, the whole data provider would fail.

I guess ideally I see it organized similarly to other multisource data providers, e.g. something like:

• my.reddit.all -- top level comments, submission, the stuff that would be actually exposed to the data consumers
  Maybe this actually can live in my/reddit/__init__.py? But need to be careful about circular dependencies.
• my.reddit.common -- merge functions.
  I guess for multisource data providers it's also inevitable to have some code duplication and add schemas here? It's not necessary in runtime, but Union[rexport.dal.Comment, pushshift_comment_export.dal.PComment] won't typecheck if someone only has rexport or pushift_comment_export installed. But can probably be worked around and not that big of a deal for most users.
• my.reddit.rexport -- basically this stuff
• my.reddit.pushshift -- what you added
• my.reddit.gdpr -- potentially could be added in the future if/when Reddit automates GDPR takeouts properly

The main goal in this structure IMO is to keep my.reddit.all as small as possible, so it can be easily overridden by other people.
If you have any interesting ideas about it, feel free to open a separate issue in HPI so we can discuss there!

[purarue]

Wondering, how do you approach Python entry points/executable scripts while developing?

Recently new thought process, but; If I'm developing something that I'd run once and it exports to JSON/some nice format, and I don't have to do any sort of normalization when loading in, (for example my recent forum_parser and steam_scraper projects), then I just use individual python scripts with no packaging, which require you to clone and run directly.

If its something that requires a DAL, or it would be nice to have library usage to be able to merge exports together (so, like ffexport), I prefer to just use setuptools to install the package globally. That way its relatively easy to use as a library. I do get the appeal of this (dal_helper), but its not something I wanted to wrap my head around.

I would agree with what you say for pushshift_comment_export, making it __main__.py makes more sense here. If I expected more wider/periodic usage, and there was no dal.py, you could just have the one script, I would just use the py_modules setuptools flag. Being able to do python3 -m pushshift_comment_export is nice as well. I will probably edit that to be part of __main__.py instead.

If something is installed with setuptools:

• I have two cookiecutter templates I use, one is relatively old and uses click/has a __main__.py wrapper, the other is a somewhat minimal setuptools module. The latter is what I used for pushshift_comments_export

• I will typically install it as an editable package with developing, even though this causes annoyances. It just gives me a faster feedback loop than uninstalling/reinstalling. That puts a line in ~/.local/lib/python/*/easy-install.pth that doesn't get removed when you uninstall the editable package, so after Im done with development I run:

  • pip uninstall -y package_name
  • rm -i ~/.local/lib/python*/site-packages/package_name*
  • rm -v $(which executable_name)
  • pip-editable-installs: aliased to e "$(find ~/.local/lib/python* -name easy-install.pth)"

It is quite annoying that this isn't a supported use case for python development. Once I'm done, I'll reinstall using pip3 install --user .

• I typically run an entr one-liner in another terminal, like one of these:

  99990  find harden_frontmatter | entr -c sh -c "pandoc ./post/server_setup/README.md -t json | ./harden_frontmatter"
  102146  fd -E .git | entr -c ./build
  77806  while true; do git ls-files | entr -cdp sh -c 'make uninstall && python3 setup.py install && find -type f -name *.py -exec pyflakes {} ..
> 120547  while true; do; find ./parse.py | entr python3 ./parse.py --from-file ./data.json --to-file ./parsed.json; sleep 1; done
  124024  find facebook.py | entr python3 -c 'from my.facebook import events; print(list(events()))'

That re-runs a script/file I currently have installed/runs the python3 setup.py and then a script, whenever I make any changes to the file I'm currently writing, so I have a faster feedback loop.

so in the future I'll probably get rid of them in favor of the users installing packages properly. Perhaps in a dedicated HPI virtualenv to avoid cluttering user packages

I don't really think of it as cluttering user packages. I agree the way pip handles installing into a bin directory isn't ideal. For newer users they might not want to mess with the $PATH, so making scripts python -m compliant is a good idea.

But, as long as the package name isn't parser, or something that could cause name conflicts, no ones creating something called pushshift_comment_export or ffexport. setuptools already exists, it allows users to install packages I dont push to pypi using the git+https pattern, and it allows you to import the package from any directory to use a library. Writing a setup.py (by which I mean using my cookiecutter-lib template) has too many benefits if I'm going to use something more than once.

If cluttering the namespace is that much of an issue, a virtualenv could potentially solve that, but I think virtualenvs themselves are another layer of complexity that people might be turned off from.

Deciding whether or not to push a setuptools package to pypi isn't really that much of a concern of mine, I do it if I feel the project might have breaking changes/I want it to be more stable.

Regarding dynamic imports, I think specifying a directory for the dal in HPI for individual projects isn't a great workflow, paths may change and it feels prone to errors. However, for something like configuration; my.config, which more conforms to some sort of xdg standard and it makes sense to split the configuration as not part of the project, dynamic imports seem pretty nice there. A couple other projects, like ipython and thefuck do things similarly.

If someone only uses one of these, the whole data provider would fail.

Right. It would probably be a breaking change anyways, to convert rexport to a setuptools package. One could potentially keep it like:

.
|-rexport
  |- __init__.py
  |-- dal.py
  |-- export.py
dal.py
setup.py

The upper dal.py references the rexport package in the same directory, so you're still able to import by doing from rexport.dal import DAL (I believe?), so this pattern could be kept backwards compatible.

Converting to a package has other benefits, like someone can also do something like:

from rexport.export import Exporter
json_values = Exporter.export(**some_params)

I think allowing people to import from rexport using setuptools in cases like this is pretty nice, because it means one can do pip install git+https... and its possibly easier to subclass/monkey-patch things in your own scripts. Otherwise you have to git clone and edit the files yourself.

If the namespace package names are unique enough, it could potentially let you handle multiple providers in a nicer way, like:

def events() -> Iterable[Type1, Type2]:
    yield from _merge_events(*provider_one(), *provider_two())

and then for each provider, you could have:

try:
	from rexport.dal import events as provider_one
except ImportError:
    # warn the user?
    provider_one = lambda: ()  # returns nothing, so this provider doesnt work
.... (another one)

_merge_events would have to know how to handle the different formats, but that complexity has to live somewhere.

If you have any interesting ideas about it, feel free to open a separate issue in HPI so we can discuss there!

Sounds good. I'm still currently trying to figure out my HPI process, been doing this for less than 2 weeks. Once I'm a bit more comfortable with this and I have things I'd like to merge back in, I'll patch changes into this fork and make some PRs. I think my own fork has diverged so much that its not possible to do so anymore.

[karlicoss]

then I just use individual python scripts with no packaging, which require you to clone and run directly.

Yep, agree, that's mostly what I've been doing so far.

I typically run an entr one-liner in another terminal, like one of these

Oh yeah, it's really nice! I should remember to use entr more often too.

But, as long as the package name isn't parser, or something that could cause name conflicts

Yeah agree. I had wtfs maybe a couple of times when something was used as an editable, and not cleaned up properly (I think?), but apart from that no real issues.

but I think virtualenvs themselves are another layer of complexity that people might be turned off from.

Yep! It would certainly be optional, just thinking that potentially might be useful if the number of data providers gets out of hand. I guess another similarity to Emacs packages, having them all in a single directory (even potentially under a massive git repository) helps with quick experiments and rollback.

Regarding dynamic imports, I think specifying a directory for the dal in HPI for individual projects isn't a great workflow, paths may change and it feels prone to errors
Converting to a package has other benefits, like someone can also do something like:

Yep, that's the plan. At first I had this urge to impose as minimal structure as possible, but it seems that conforming to setuptools is worth it. The only price to pay is adding a minimal setup.py and a subdirectory for the code (but perhaps this is a good habit anyway).

because it means one can do pip install git+https

Yep, this as well, certainly easier than carefully cloning at the correct location or copy pasting the path into the config.

and then for each provider, you could have:

That way could work too, yeah. I guess one downside is that it feels too defensive, e.g. if something is misconfigured, you might fallback onto empty data provider without realizing it. But on the other hand breaking things is more annoying.

Maybe it could benefit from different 'strategies' for handing missing data providers, e.g.

• by default, warn in logs, and also instead of fallback onto an empty list, return a suitable Result value, e.g. Exception("Couldn't use rexport data because of ImportError ...."). Ideally the downstream tools would handle and propagate error values anyway so the user would notice (while also having the rest of the data sources work as intended)

• allow 'disabling' data providers if they don't want to use it.
  If it's disabled, we can simply fallback onto empty data and only keep a debug/info log message.

  Actually, for disabling, it might as well reuse the same mechanism we discussed here!

• and there could be a global policy for the default behaviour -- e.g. during active development I would prefer for things to crash immediately instead of returning error values, so would be nice to have a toggle for that. (related to Support different error handling policies #17)

I think my own fork has diverged so much that its not possible to do so anymore.

Yeah no problem, having fun with the code is more important :) Always can just copy-paste the code onto a fresh fork later.

[karlicoss]

I've added the link to readme and moving the issue to HPI (most of the discussion is relevant to it), hope you don't mind!

[purarue]

total misclick there, sounds good.

[purarue]

Once I've merged changes from rexport and #83 into my fork...

Would be possible to merge some of this (pushishift). Would probably be optional, like:

@dataclass
class reddit(uconfig)
	...
    # path[s]/glob to one or more comment exports
    # https://github.com/purarue/pushshift_comment_export
    pushshift_export_path: Optional[Paths]

	
....
# if user has set pushshift comment export path...
	from pushshift_comment_export.dal import PComment, read_file

I can probably work on a PR at some point if thats something you're interested in.

[karlicoss]

Just merged #83 ! Let me know if you have any issues :)

Absolutely, would be great to merge!

1. Configs: IMO it's best to separate the configs, e.g. class reddit_pushshift, class reddit_rexport. Or even better?:

   class reddit:
     class pushshift:
         export_path: Paths
   
     class rexport:
         export_path: Paths
   

   That way:

   • the naming for export_path is consistent
   • if there are some extra config attributes, they wouldn't mix up, and less potential for clashes

2. Optional[Paths]: I thought that Optional[Paths] would look awkward (suggests it's sort of expected to be None). Considering that setting it to None would be fairly rare (you usually either use the module and set it to something reasonable, or don't have the config at all), I though that it would make sense to reserve empty string for that purpose:

   HPI/tests/get_files.py

   Line 105 in 1c20eb2

   assert get_files('') == ()

3. Code structure: I think ideally it's best to separate the different data providers into different files, extract the schemas/common stuff to something like common.py and have a minimal all.py file that merges them together, e.g.:

   • https://github.com/karlicoss/HPI/blob/master/my/github/all.py
   • https://github.com/karlicoss/HPI/blob/master/my/twitter/all.py

   The idea is that if the user adds their own data source, they can benefit from common.py, and supply their own all.py (that's why it should be as minimal as possible). I think it will also make backwards compatibility easier. What do you think?

P.S. Just recalled, I also had some experiments/notes about possible ways of overriding/extending modules, will dig them out and post somewhere for the discussion.

[purarue]

any issues

all good on my end, been merging commits as you've been merging them into master; Though, I'm only using ghexport and rexport currently.

nested classes for config

Would that work with my.core.cfg.make_config? If so looks good. Would need to add some checks for backward compatibility. The avoiding clashes point is good as well, for the possibility of sharing secret files for data sources.

I though that it would make sense to reserve empty string for that purpose

Feels sort of wrong from a typing perspective, but simple enough. Though, if we're doing nested classes, just don't include the pushshift class if its not used.

all.py/common.py

Yeah, this is better organized, I'll follow how its done in github. I suppose following that pattern, in common.py, Instead of doing a try/catch to stop an ImportError for pushshift_comment_export.dal in the merge function, would be easiest to catch that in the data-source specific (my.reddit.pushshift.py) file, and return an empty list to the merge function if the user didn't provide config/export path.

[karlicoss]

Would that work with my.core.cfg.make_config?

I think so? As far as I understand, nested classes in Python are merely a syntactic thing (e.g. even if you want to access an attribute of the enclosing class, you'd have to specify a fully qualified name)

Would need to add some checks for backward compatibility

Yeah. Regarding backward compatibility, also thought that my/reddit/__init__.py would be handy then, so old imports of my.reddit continue working. However then my.reddit wouldn't be a namespace package anymore, and that means that overrides won't work. I guess if the individual modules were properly versioned, that would be easier to handle, could be a major version change, or something similar. I'll think more about it...

would be easiest to catch that in the data-source specific (my.reddit.pushshift.py) file, and return an empty list to the merge function if the user didn't provide config/export path.

Ah yeah. Ideally, if the config has empty path, the list of inputs is empty, and the module just emits nothing and everything just works. The issue now is that the modules (e.g. import ghexport) are usually imported on the top level, both because it's more convenient, and for type annotation purposes. That means that if the user never wanted to use it in the first place, it still crashes on import, so they either have to install the dependencies, or override all.py and comment out the modules they aren't using.

Something I can think on spot: reorganize all.py to something like:

from my.core import X
from .common import merge_events, Results

def gdpr():
    from . import gdpr as source
    return source

def ghexport():
    from . import ghexport as source
    return source

def events() -> Results:
    yield from merge_events(
        X(gdpr()    .events(), []),
        X(ghexport().events(), []),
    )

X is a function which adds some defensive behaviour, something like:

def X(factory, default):
    try:
        res = factory()
        return res
    except ImportError: # presumably means that the user hasn't installed the module. other exceptions would be legitimate errors?
        warn(f"Module {factory.__name__} is not configured blah blah blah")
        return default

, it could perhaps be somehow configurable, for example, respect disabled_modules and in that case it won't emit the warning?

• a bit boilerplaty. Really wish python allowed lambdas with multiple statements... maybe alternatively could use importlib
• this is not mypy transparent (although all.py is meant to be super short so maybe it's not a biggie)

I'll think about it more.. maybe I'll come up with a way of lazily importing module dependencies so it's mypy friendly.