Skip to content

Retain resource attributes in write_package() - #195

Merged
peterdesmet merged 6 commits into
mainfrom
retain_attributes
Mar 29, 2024
Merged

Retain resource attributes in write_package()#195
peterdesmet merged 6 commits into
mainfrom
retain_attributes

Conversation

@peterdesmet

@peterdesmet peterdesmet commented Mar 26, 2024

Copy link
Copy Markdown
Member

Fix #191, fix #160.

  • write_resource() no longer builds a resource from scratch for added resources, but uses the existing metadata (including custom properties).
  • add_resource() will set some of the reserved properties like mediatype (to NULL), so they are at a logical position. These are then populated by write_package()
  • The custom properties ... are now included BEFORE the schema.
  • I also changed ifelse(condition, x, y) for if(condition) x else y, since it allows to return NULL in y 64e4f9d

Reprex for fixed #191:

library(magrittr)
library(frictionless)
my_package <-
  create_package() %>%
  add_resource(
    resource_name = "iris",
    data = iris,
    title = "Iris dataset", # Your additional metadata
    description = "The built-in dataset in R."
  )

# the package indeed includes the properties title and description
str(my_package)
#> List of 2
#>  $ resources:List of 1
#>   ..$ :List of 10
#>   .. ..$ name       : chr "iris"
#>   .. ..$ data       :'data.frame':   150 obs. of  5 variables:
#>   .. .. ..$ Sepal.Length: num [1:150] 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
#>   .. .. ..$ Sepal.Width : num [1:150] 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
#>   .. .. ..$ Petal.Length: num [1:150] 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
#>   .. .. ..$ Petal.Width : num [1:150] 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
#>   .. .. ..$ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
#>   .. ..$ profile    : chr "tabular-data-resource"
#>   .. ..$ format     : NULL
#>   .. ..$ mediatype  : NULL
#>   .. ..$ encoding   : NULL
#>   .. ..$ dialect    : NULL
#>   .. ..$ title      : chr "Iris dataset"
#>   .. ..$ description: chr "The built-in dataset in R."
#>   .. ..$ schema     :List of 1
#>   .. .. ..$ fields:List of 5
#>   .. .. .. ..$ :List of 2
#>   .. .. .. .. ..$ name: chr "Sepal.Length"
#>   .. .. .. .. ..$ type: chr "number"
#>   .. .. .. ..$ :List of 2
#>   .. .. .. .. ..$ name: chr "Sepal.Width"
#>   .. .. .. .. ..$ type: chr "number"
#>   .. .. .. ..$ :List of 2
#>   .. .. .. .. ..$ name: chr "Petal.Length"
#>   .. .. .. .. ..$ type: chr "number"
#>   .. .. .. ..$ :List of 2
#>   .. .. .. .. ..$ name: chr "Petal.Width"
#>   .. .. .. .. ..$ type: chr "number"
#>   .. .. .. ..$ :List of 3
#>   .. .. .. .. ..$ name       : chr "Species"
#>   .. .. .. .. ..$ type       : chr "string"
#>   .. .. .. .. ..$ constraints:List of 1
#>   .. .. .. .. .. ..$ enum: chr [1:3] "setosa" "versicolor" "virginica"
#>  $ directory: chr "."
#>  - attr(*, "class")= chr [1:2] "datapackage" "list"

# writing the package
write_package(my_package, "irisdir")

# when reading the package again, title and description have disappeared FIXED
p <- read_package("irisdir/datapackage.json")
unclass(p)
#> $resources
#> $resources[[1]]
#> $resources[[1]]$name
#> [1] "iris"
#> 
#> $resources[[1]]$path
#> [1] "iris.csv"
#> 
#> $resources[[1]]$profile
#> [1] "tabular-data-resource"
#> 
#> $resources[[1]]$format
#> [1] "csv"
#> 
#> $resources[[1]]$mediatype
#> [1] "text/csv"
#> 
#> $resources[[1]]$encoding
#> [1] "utf-8"
#> 
#> $resources[[1]]$title
#> [1] "Iris dataset"
#> 
#> $resources[[1]]$description
#> [1] "The built-in dataset in R."
#> 
#> $resources[[1]]$schema
#> $resources[[1]]$schema$fields
#> $resources[[1]]$schema$fields[[1]]
#> $resources[[1]]$schema$fields[[1]]$name
#> [1] "Sepal.Length"
#> 
#> $resources[[1]]$schema$fields[[1]]$type
#> [1] "number"
#> 
#> 
#> $resources[[1]]$schema$fields[[2]]
#> $resources[[1]]$schema$fields[[2]]$name
#> [1] "Sepal.Width"
#> 
#> $resources[[1]]$schema$fields[[2]]$type
#> [1] "number"
#> 
#> 
#> $resources[[1]]$schema$fields[[3]]
#> $resources[[1]]$schema$fields[[3]]$name
#> [1] "Petal.Length"
#> 
#> $resources[[1]]$schema$fields[[3]]$type
#> [1] "number"
#> 
#> 
#> $resources[[1]]$schema$fields[[4]]
#> $resources[[1]]$schema$fields[[4]]$name
#> [1] "Petal.Width"
#> 
#> $resources[[1]]$schema$fields[[4]]$type
#> [1] "number"
#> 
#> 
#> $resources[[1]]$schema$fields[[5]]
#> $resources[[1]]$schema$fields[[5]]$name
#> [1] "Species"
#> 
#> $resources[[1]]$schema$fields[[5]]$type
#> [1] "string"
#> 
#> $resources[[1]]$schema$fields[[5]]$constraints
#> $resources[[1]]$schema$fields[[5]]$constraints$enum
#> [1] "setosa"     "versicolor" "virginica" 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> $directory
#> [1] "irisdir"

Created on 2024-03-26 with reprex v2.1.0

The latter doesn't allow to set NULL as the last argument
- In add_resource(): set reserved properties format, mediatype, encoding, dialect explicitly to NULL so they are at the correct position. For CSV resources dialect is set explicitely to null (correct position) and then updated or removed
- In write_resource(): no longer start a resource from scratch, but edit the existing one: 1) add path right behind name and 2) set the reserved properties
- Update tests
@peterdesmet
peterdesmet changed the base branch from main to replace_null March 26, 2024 13:26
@peterdesmet
peterdesmet requested a review from PietrH March 26, 2024 13:26
@peterdesmet peterdesmet added this to the 1.1.0 milestone Mar 26, 2024
Base automatically changed from replace_null to main March 27, 2024 12:27

@damianooldoni damianooldoni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice solutions to these two issues. And nothing against the use of standard if else on one line. It allows indeed to assign NULL to the output, and it's in my opinion slightly more readable than ifelse() or if_else() expressions. I will keep it in mind in my code writing.

Just two questions. Do you think worth to:

  1. Add a small paragraph in the vignette where you show what is retained in the metadata by running write_package()?
  2. Add a sentence with the limitations of write_package()? For example, it writes resources as CSVs only right now.

Just a reflection while reading the very clear Getting Started.

Tests, checks passed locally and documentation correctly compiled.

@peterdesmet

Copy link
Copy Markdown
Member Author

Thanks for the review and suggestions Damiano! Here's why I won't update the documentation further:

  • I show in the vignette how to add properties to a resource, but I don't find a good place to indicate that they are written, without breaking the flow of the text. I think it's a fair assumption of the user that they are written, without having to show it.
  • The "limitations" (or features 😉 ) of write_package() are explained in the function documentation: it lists what happens for each type of resource. The vignette shows what files were written for my_package.

I'll merge this and get started on releasing 1.1.0.

@peterdesmet
peterdesmet merged commit 6e252f2 into main Mar 29, 2024
@peterdesmet
peterdesmet deleted the retain_attributes branch March 29, 2024 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metadata properties are not written in datapackage.json Retain resource title and description

2 participants