Merge pull request #22 from wesselhuising/xrn-pandas-plugin

Pandas plugin

Author: xaviernogueira

.pre-commit-config.yaml

@@ -53,19 +53,8 @@ repos:
         entry: isort
         require_serial: true
         language: system
-      - id: pylint
-        name: pylint
-        entry: pylint
+      - id: safety
+        name: check for vulnerable dependencies with Safety
+        entry: safety
         language: system
-        types: [python]
-        args:
-          [
-            "-rn", # Only display messages
-            "-sn", # Don't display the score
-            "src", # Only source code, skip tests folder
-          ]
-      # - id: safety
-      #   name: check for vulnerable dependencies with Safety
-      #   entry: safety
-      #   language: system
-      #   args: [./pyproject.toml]
+        args: [./pyproject.toml]

README.md

@@ -6,6 +6,10 @@ First, install `pandantic` by using pip (or any other package managing tool).
 
 ```pip install pandantic```
 
+## Docs
+
+Documentation can be found [here](https://pandantic-rtd.readthedocs.io/en/latest/)
+
 ## parse_df
 
 To validate `pd.DataFrame`s using Pydantic `BaseModel`s make sure to import the `BaseModel` class from the `pandantic` package.
@@ -14,11 +18,11 @@ To validate `pd.DataFrame`s using Pydantic `BaseModel`s make sure to import the
 
 The `pandantic.BaseModel` subclasses the original `pydantic.BaseModel` which means the `pandantic.BaseModel` includes all functionality from the original `pydantic.BaseModel` but it adds the `parse_df` class method which should be used to parse DataFrames.
 
-## A quick example
+### A quick example
 
 Enough of the talking, lets just make things easier by showing a very minor but quick example. Make sure to import the `BaseModel` class from `pandantic` and create a schema like we normally would when using `pydantic`.
 
-```
+```python
 from pydantic.types import StrictInt
 
 from pandantic import BaseModel
@@ -33,7 +37,7 @@ class DataFrameSchema(BaseModel):
 
 Let's try this schema on a simple `pandas.DataFrame`. Use the class method `parse_df` from the freshly defined `DataFrameSchema` and specify the `dataframe` that should be validated using the arguments of the method. In this example, we want to `filter` out the bad records (there are more options like the good old `raise` to raise a ValueError after validating the whole DataFrame). In this case, only the second record would be kept in the returned DataFrame.
 
-```
+```python
 df_invalid = pd.DataFrame(
     data={
         "example_str": ["foo", "bar", 1],
@@ -46,11 +50,38 @@ df_filtered = DataFrameSchema.parse_df(
     errors="filter",
 )
 ```
-### Custom validators
 
-One of the great features of Pydantic is the ability to create custom validators. Luckily, those custom validators will also work when parsing DataFrames using `pandantic`. Make sure to import the original decorator from the `pydantic` package and keep in mind that `pandantic` is using the V2 of Pydantic (so `field_validation` it is). In the example below the `BaseModel` will validate the `example_int` field and makes sure it is an even number.
+## Pandas plugin
+
+Another way to use `pandantic` is via our [`pandas.DataFrame` extension](https://pandas.pydata.org/docs/development/extending.html) plugin. This adds the following methods to `pandas` (once "registered" by `import pandantic.plugins.pandas`):
+* `DataFrame.pydantic.validate(schema:PandanticBaseModel)`, which returns a boolean for all valid inputs.
+* `DataFrame.pydantic.filter(schema:PandanticBaseModel)`, which wraps `PandanticBaseModel.parse_obj(errors="filter")` and returns as dataframe.
+
+**Example:**
+```python
+from pandantic import BaseModel
+import pandantic.plugins.pandas
+
+df1: pd.DataFrame = pd.DataFrame({"a": [1, 2, 3], "b": ["a", "b", "c"]})
+class MyModel(BaseModel):
+    a: int
+    b: str
 
+df1.pydantic.validate(MyModel)  # returns True
+df1.pydantic.filter(MyModel)  # returns the same dataframe
+
+# but if we have a mixed DataFrame
+df2: pd.DataFrame = pd.DataFrame({"a": [1, 2, "3"], "b": ["a", 3, "c"]})
+
+df2.pydantic.validate(MyModel)  # returns False
+df2.pydantic.filter(MyModel)  # returns the filtered DataFrame with only the first row
 ```
+
+## Custom validator example
+
+One of the great features of Pydantic is the ability to create custom validators. Luckily, those custom validators will also work when parsing DataFrames using `pandantic`. Make sure to import the original decorator from the `pydantic` package and keep in mind that `pandantic` is using the V2 of Pydantic (so `field_validation` it is). In the example below the `BaseModel` will validate the `example_int` field and makes sure it is an even number.
+
+```python
 from pydantic import ValidationError, field_validator
 
 
@@ -72,7 +103,7 @@ class DataFrameSchema(BaseModel):
 
 By setting the `errors` argument to `raise`, the code will raise an ValueError after validating every row as the first row contains an uneven number.
 
-```
+```python
 example_df_invalid = pd.DataFrame(
     data={
         "example_str": ["foo", "bar", "baz"],
@@ -90,7 +121,7 @@ df_raised_error = DataFrameSchema.parse_df(
 ### Optional
 As the DataFrame is being parsed into a dict, a `None` value is considered as a `nan` value in cases there are different values in the dict. Therefore, specifying `Optional` columns (where the value can be empty) can be speciyfied by using the custom `pandantic.Optional` type. This type is a replacement for `typing.Optional`.
 
-```
+```python
 from pandantic import BaseModel, Optional
 
 class Model(BaseModel):
@@ -101,6 +132,3 @@ df_example = pd.DataFrame({"a": [1, None, 2], "b": ["str", 2, 3]})
 
 df_filtered = Model.parse_df(df_example, errors="filter", verbose=True)
 ```
-
-## Docs
-Documentation can be found [here](https://pandantic-rtd.readthedocs.io/en/latest/)