Migrate from TealData to teal_data

[gogonzo]

⚠️ Comments will be opened for external users after release.

Migrate from TealData to teal_data

The CRAN release of teal (v0.15.0) brings major changes the way data is created and processed by teal, streamlining data flow across teal modules and making teal packages much easier to maintain.
The data argument in teal::init no longer accepts the TealData class, the basic data container is now the teal_data class.
We introduce a method for the base function within as a convenient way to run code within the data container.
We also introduce the teal_data_module concept for cases when the teal_data container must be built during the application run time.

The table below shows how to adjust your existing apps to the new class and the following text that explains the changes in more detail.

Changes summary

Execute this code to obtain old `teal` (<=0.14.0) datasets `rADSL` and `rADTTE`. (click)

rADSL <- scda::synthetic_cdisc_data("latest")$adsl
rADTTE <- scda::synthetic_cdisc_data("latest")$adtte

Section	teal 0.14.0 and lower	teal 0.15.0 and higher
Create non-reproducible data	teal_data( dataset("iris", iris), dataset("mtcars", iris) ) cdisc_data( dataset("ADSL", rADSL), dataset("ADTTE", rADTTE) )	teal_data( iris = iris, mtcars = mtcars ) cdisc_data( ADSL = rADSL, ADSL = rADSL )
Create reproducible data	# supply code for all data x <- teal_data( dataset("iris", iris), dataset("mtcars", mtcars), code = "iris <- iris mtcars <- mtcars" ) # supply code for individual datasets x <- teal_data( dataset("iris", iris, code = "iris <- iris"), dataset("mtcars", mtcars, code = "mtcars <- mtcars") ) # retrieve code x$get_code("iris")	It is no longer necessary to specify code for individual datasets. 'teal.data::get_code()' can extract the code necessary to reproduce a specific dataset, using the `datanames` argument. # create data by evaluating code x <- teal_data() |> within({ iris <- iris mtcars <- mtcars }) # retrieve code get_code(x, datanames = "iris")
Access data	x <- teal_data( dataset("iris", iris), dataset("mtcars", mtcars) ) get_raw_data(x, "iris") # returns data.frame	x <- teal_data() |> within({ iris <- iris mtcars <- mtcars }) x[["iris"]] # returns data.frame
Modify data	teal_data(dataset("mtcars", mtcars)) |> mutate_data("mtcars <- subset(mtcars, cyl == 4)") # pseudocode - don't run dataset_connector(...) |> mutate_data("mtcars <- subset(mtcars, cyl == 4)")	teal_data(mtcars = mtcars) |> within(mtcars <- subset(mtcars, cyl == 4)) # pseudocode - don't run teal_data_module(...) |> within(mtcars <- subset(mtcars, cyl == 4))
Data validation	x <- teal_data( dataset("iris", iris), dataset("mtcars", mtcars), code = "iris <- iris mtcars <- mtcars", check = TRUE ) x$check()	x <- teal_data( iris = iris, mtcars = mtcars, code = expression( iris <- iris, mtcars <- mtcars ) ) verify(x)
Join keys	x <- cdisc_data( cdisc_dataset("ADSL", rADSL), cdisc_dataset("ADTTE", rADTTE) ) x$get_join_keys()$get("ADSL", "ADTTE") # return joining keys mutate_join_keys(x, "ADSL", "ADTTE", c("STUDYID", "USUBJID")) # set joining keys	x <- cdisc_data( ADSL = rADSL, ADTTE = rADTTE ) join_keys(x)["ADSL", "ADTTE"] # return joining keys join_keys(x)["ADSL", "ADTTE"] <- c("USUBJID", "USUBJID") # set joining keys
Connectors	# pseudocode - do not run my_connection <- data_connection(ping, open, close) my_ds_connector <- dataset_connector(dataname, pull, keys, label) my_data <- TealDataConnector$new(my_connection, my_ds_connector) my_data$set_pull_args(...) my_data$set_ui( # ui code ) my_data$set_server( # server code )	# pseudocode - do not run my_data <- teal_data_module( ui = function(id) { ns <- NS(id) # ui code }, server = function(id) { moduleServer(id, function(input, output, session) { # server code returning teal_data in as reactive }) } )

Create simple data

Until now, supply data to a teal application was a multi-step process, where one would pass data objects to the dataset() function to create "datasets" and then include the datasets to teal_data() to create the final data container.
Now data objects are passed directly to teal_data().
Note that this way of specifying data is recommended only in case where no reproducible code is necessary.
Including data in teal applications is detailed in this vignette.

library(teal.data)
x <- teal_data(
  d1 = data.frame(a = 1:10, b = 1:10),
  d2 = data.frame(a = 1:10, b = 1:10)
)
print(x)

Create reproducible data

Tracking code for reproducibility requires that the data creation code be included in the teal_data object.
In old teal this was done by creating data objects by running code in the global environment and then passing both the objects and the code to teal_data().
While this is still possible, we recommend the superior approach of initializing teal_data as an empty container and creating data objects by running code within the container.
Keeping the data in an isolated environment from the moment of creation throughout the whole analysis is the best way ensure that the code represenst the data exactly.
A detailed explanation of reproducibility can be found in this vignette.

library(teal.data)
x <- teal_data() |> within({
  d1 <- data.frame(a = 1:10, b = 1:10)
  d2 <- data.frame(a = 1:10, b = 1:10)
})
get_code(x)
get_code(x, datanames = "d1")
get_code(x, datanames = "d2")

Access data

The new teal_data class stores data objects much like a list.
Objects can be accessed using public functions exported from hte underlying teal.code package.

library(teal.data)
x <- teal_data() |> within({
  d1 <- data.frame(id = 1:10, a = 1:10)
  d2 <- data.frame(id = 1:10, b = 1:10)
})

x[["d1"]]

Modify data

The functions mutate_data and mutate_dataset are superseeded by within.
The usage of the functions is similar but within accepts inline R expressions rather than character.
(If there is a valid reason to pass code as character, use the eval_code function instead of within.)
Note that it is no longer necessary to which data object the code refers to.
The expression is evaluated and changes are directly applied to the contents of the container.

library(teal.data)
x <- teal_data() |> within({
  d1 <- data.frame(id = 1:10, a = 1:10)
  d2 <- data.frame(id = 1:10, b = 1:10)
})  
x_modified <- x |> within({ 
  d1 <- subset(d1, id > 5)
  d2 <- subset(d2, id > 5)
})
get_code(x_modified, datanames = "d1")

within also works with teal_data_module (more on that later).
Please visit the "Introduction to teal.data" vignette for more details.

Data validation

The old TealData class had a data$check() method which was used to validate that the code corresponds to the data.
The check was done by evaluating the code and comparing the results to the data stored in the object.
This action is now performed by the verify function, defined in the teal.data package.
The verification process is described in detail in the teal_data reproducibility vignette.

library(teal.data)
d1 <- data.frame(id = 1:10, a = 1:10)
d2 <- data.frame(id = 1:10, b = 1:10)
x <- verify(
  teal_data(
    d1 = d1, 
    d2 = d2, 
    code = expression(
      d1 <- data.frame(id = 1:10, a = 1:10),
      d2 <- data.frame(id = 1:10, b = 1:10)
    )
  )
)

Join keys

Join keys are now accessed and set with the join_keys function.
Running join_keys on a teal_data object returns a list of joining keys.
Modifying part of a joining key set is done with the [ operator.
A detailed explanation of joining keys can be found in the "Join Keys" vignette.

library(teal.data)
x <- teal_data() |> within({
  d1 <- data.frame(id = 1:10, b = 1:10)
  d2 <- data.frame(id = 1:10, b = 1:10)
})

# set joining keys
join_keys(x) <- join_keys(
  join_key("d1", "d1", "id"),
  join_key("d2", "d2", "id"),
  join_key("d2", "d2", "id")
)

# get joining keys
join_keys(x)

# modify keys
join_keys(x)["d1", "d1"] <- "b"

Connectors

All hitherto existing connector classes and their related functions are now deprecated and we do not offer 1:1 replacements.
The old Teal* classes contained data, ui, server and various other attributes and methods.
That solution was not flexible enough to fullfil incoming user requests and also hampered maintenance.

Intead we introduce the teal_data_module concept.
A special shiny module is created with the teal_data_module function.
The module is passed to a teal application in place of a teal_data object and creates data once the application starts.
The server function runs all code necessary to connect to an external data source and obtain the desired data objects and returns a teal_data object.
teal_data_module is thoroughly explained in the "Data as shiny module" vignette.

library(teal)
data_mod <- teal_data_module(
  ui = function(id) {
    ns <- NS(id)
    actionButton(ns("btn"), "Click me")
  },
  server = function(id) {
    moduleServer(id, function(input, output, session) {
      eventReactive(input$btn, {
        teal_data() |> within({
          d1 <- data.frame(id = 1:10, b = 1:10)
          d2 <- data.frame(id = 1:10, b = 1:10)
        })
      })
    })
  }
)

app <- teal::init(
  data = data_mod,
  modules = example_module()
)

shiny::shinyApp(app$ui, app$server)

[Mia-data]

Hi,

I'm trying to translate my teal app based on teal version < v.0.14 to the latest release v.0.15. The problem is with setting primary and foreign keys. I've read the materials + join key vignitte https://insightsengineering.github.io/teal.data/latest-tag/articles/join-keys.html#anatomy-of-join_keys but still I cannot figure out the issue of the app - it still throws an error that "extracted data has not correctly set joining keys"

#I need to simply translate old code to teal migration:
app <- init(
  data = teal_data(
    dataset("x", x,  keys=c("id"))
), 
modules=modules(
tm_g_response("Response",
                  response = response2,
                  x = response1,
                  row_facet = NULL,
                  col_facet = data_extract_spec(
                    dataname = "x",
                    filter = filter_spec(
                      vars = vars,
                      choices = value_choices(x, vars$selected),
                      selected = value_choices(x, vars$selected),
                      multiple = T
                    )
                  ),
                  coord_flip = FALSE
    )
  ))
)


#I tried this but it didn't worked in v0.15.0:
data<- within(teal_data(), {
  x=x
  response1=response1
  response2=response2
})

datanames <- c("x", "response1", "resposne2")
datanames(data) <- datanames

join_keys(data) <- join_keys(
  join_key("x", keys = c("id")),
 join_key("response1", keys = c("id")),
 join_key("response2", keys = c("id")),
 join_key("response1", "response2" keys = c("id"="id"))
)
app <- init(data=data,
modules=modules(
tm_g_response("Response",
                  response = response2,
                  x = response1,
                  row_facet = NULL,
                  col_facet = data_extract_spec(
                    dataname = "x",
                    filter = filter_spec(
                      vars = vars,
                      choices = value_choices(x, vars$selected),
                      selected = value_choices(x, vars$selected),
                      multiple = T
                    )
                  ),
                  coord_flip = FALSE
    )
  )
)

I would appreciate your help

[gogonzo]

Hi @Mia-data

1. Do your app use only one dataset x?

2. Is message below precisely what you got in return? I can't find exact messages in our codebase

extracted data has not correctly set joining keys

---

Would be easiest if you just send us the code of the former (working) application so we can help you directly - if it contains sensitive data please just replace sensitive information with some more implicit names/paths/symbols. It is hard to deduce from the code you've provided

[Mia-data]

Hi @gogonzo,

I have opened also the issue here #1164 (comment) so I will continue provide more info there to avoid duplication of the topic.