data.qmd

---
title: "Data"
---

{{< include components/_dataset_jsonld.qmd >}}

```{r}
#| label: pilot-table-pages
#| echo: false
#| message: false
# Which tables have a landing page under /tables/ (ben-domingue/irw#1706).
# Read from the emitter's own list so the two cannot drift: when the pilot grows,
# this grows with it, and neither the prose count below nor the OJS link logic
# needs touching. The emitter runs post-render and cannot tell this chunk
# anything, so the shared file is the contract.
pilot_tables_chr <- local({
  f <- file.path("landing", "pilot_tables.txt")
  if (!file.exists(f)) return(character(0))
  ln <- trimws(sub("#.*$", "", readLines(f, warn = FALSE)))
  sort(unique(tolower(ln[nzchar(ln)])))
})
n_pilot <- length(pilot_tables_chr)
ojs_define(pilot_tables = pilot_tables_chr)
```

This page has two tools, and individual tables have their own pages:

- [**Explore all datasets**](#explore-all-datasets) — filter and browse across all IRW datasets by size, response type, and other properties. As you filter, this tool builds a ready-to-run R code snippet that reproduces your selection with the `irw` package.
- [**Look up a single dataset**](#look-up-a-single-dataset) — search for one dataset by name and generate its citation (reference and BibTeX) for use in your own work.
- [**Browse table pages**](/tables/) — every table has its own page, giving a description of what it measures and where it came from, its size and shape (responses, respondents, items and response categories), the tags that classify it, item text statistics where they exist, the columns it contains, and ready-to-run R and Python code to fetch it. Each page names the IRW version it describes, so it can be cited. **This is a pilot: only `r n_pilot` tables have a page so far**, chosen to test the page generator against awkward cases rather than because they are the most useful tables. Use *Explore all datasets* above to search the whole warehouse.

---

## Explore all datasets

{{< include _load-data-explore.qmd >}}

{{< include components/_interval.qmd >}}
{{< include components/_hist.qmd >}}
{{< include components/_tol.qmd >}}

```{ojs prelims}
// import newer version of observable plot than the one embedded in quarto
Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm")
import {yamultiselect} from '@saneef/yet-another-multi-select'

md_tagged = transpose(datasets_tagged)
md_notags = transpose(datasets_all)
md = use_qual_filters ? md_tagged : md_notags
```

{{< include components/_style.qmd >}}

```{ojs info}
// info for numeric variables
_vi = ({
  "n_responses":               { label: "Number responses (n_responses)",                        base: 10 },
  "n_categories":              { label: "Number categories (n_categories)",                      base:  2 },
  "n_participants":            { label: "Number participants (n_participants)",                  base: 10 },
  "n_items":                   { label: "Number items (n_items)",                                base: 10 },
  "responses_per_participant": { label: "Responses per participant (responses_per_participant)", base: 10 },
  "responses_per_item":        { label: "Responses per item (responses_per_item)",               base: 10 },
  "density":                   { label: "Density (n_responses/(n_participants*n_items))",        base: 10 }
})

// compute and add in ranges for numeric variables
vi = Object.fromEntries(Object.entries(_vi).map(([k, v]) => [k, {...v, range: log_range(md.map(d => d[k]), v.base)}]))

// info for tag variables
ti = ({
  "longitudinal":                          { label: "Longitudinal"                          } ,
  "has_item_text":                         { label: "Has item text?"                        } ,
  "age_range":                             { label: "Age range"                             } ,
  "child_age__for_child_focused_studies_": { label: "Child age (for child-focused studies)" } ,
  "construct_type":                        { label: "Construct type"                        } ,
  "sample":                                { label: "Sample"                                } ,
  "measurement_tool":                      { label: "Measurement tool"                      } ,
  "item_format":                           { label: "Item format"                           } ,
  "primary_language_s_":                   { label: "Primary language(s)"                   } ,
  "license":                               { label: "License"                               } ,
})
```

```{ojs filter-funs}
// formatting function for how two ended slider displays selected range
// (use SI prefix if base 10 otherwise  default format)
pow_format = (v, b) => b === 10 ? d3.format("~s")(Math.pow(b, v)) : Plot.formatNumber("en-US")(Math.pow(b, v))
interval_format = (b) => ([start, end]) => `[${pow_format(start, b)}; ${pow_format(end, b)}]`

// any base log
log = (x, b) => Math.log(x) / Math.log(b)
// log-transformed extent of array of values
log_range = (x, b) => [Math.floor(log(d3.min(x), b)), Math.ceil(log(d3.max(x), b))]

// interval input for a given numeric variable
var_interval = (v) => {
  //return interval(log_range(md.map(d => d[v]), vi[v].base), {
  return interval(vi[v].range, {
    step: 1, label: vi[v].label, format: interval_format(vi[v].base), width: "95%"
  })
}

// checkbox input for a given tag variable
tag_checkbox = (t) => Inputs.checkbox(tags[t], {value: tags[t]})
```

:::::: {.column-screen-inset}
::::: {layout="[ [19,55,26] ]"}

:::: {.panel-input}

{{< bi filter >}} _Filter by quantitative properties._

::: {.side-inputs}

::: {.filters-container}

```{ojs filter-responses}
viewof n_responses_range = var_interval("n_responses")
```

```{ojs filter-categories}
viewof n_categories_range = var_interval("n_categories")
```

```{ojs filter-participants}
viewof n_participants_range = var_interval("n_participants")
```

```{ojs filter-items}
viewof n_items_range = var_interval("n_items")
```

```{ojs filter-responses-per-participant}
viewof responses_per_participant_range = var_interval("responses_per_participant")
```

```{ojs filter-responses-per-item}
viewof responses_per_item_range = var_interval("responses_per_item")
```

```{ojs filter-density}
viewof density_range = var_interval("density")
```


:::
:::

-----

{{< bi filter >}} _Filter by variables._

::: {.side-inputs}

::: {.filters-container}

```{ojs filter-variable}
viewof variable = yamultiselect(vars.variable, { label: "Variable" })
```

```{ojs filter-prefix}
viewof prefix = yamultiselect(vars.prefix, { label: "Variable prefix" })
```

```{ojs filter-collection}
// Collections (issue #1633). yamultiselect rather than the checkbox idiom used
// for the tag facets: 22 values is too many to scan, and the checkbox default
// is all-selected, which would read as a constraint the user never set. Empty
// here correctly means "no constraint" -- hence overlap_null_filter below.
// Lives in this panel, not the qualitative one, because the design collections
// come from metadata and so apply even when qualitative filtering is off.
viewof collection_sel = yamultiselect(collection_opts, { label: "Collection" })
```

:::
:::

::::

:::: {.panel-fill}

```{ojs data-filtered}
// generate range filter function for a given numeric variable and input range
range_filter = (v, v_range) => {
  return (d) => d[v] >= Math.pow(vi[v].base, v_range[0]) &
                d[v] <= Math.pow(vi[v].base, v_range[1])
}

// generate overlap filter function for a given categorical variable and input values
overlap_filter = (t, t_vals) => {
  return (d) => t_vals.some(v => d[t].includes(v))
}

// generate overlap filter function as above but that defaults to including everything if input values empty
overlap_null_filter = (t, t_vals) => {
  return (d) => t_vals.length ? t_vals.some(v => d[t].includes(v)) : true
}

// specify range-based inputs (as variable, values)
range_inputs = ([
  { v: "n_responses",               vals: n_responses_range               },
  { v: "n_categories",              vals: n_categories_range              },
  { v: "n_participants",            vals: n_participants_range            },
  { v: "n_items",                   vals: n_items_range                   },
  { v: "responses_per_participant", vals: responses_per_participant_range },
  { v: "responses_per_item",        vals: responses_per_item_range        },
  { v: "density",                   vals: density_range                   },
])

// specify overlap-based inputs (as variable, values)
overlap_inputs = ([
  { v: "longitudinal",                          vals: longitudinal_vals     },
  { v: "has_item_text",                         vals: has_item_text_vals    },
  { v: "age_range",                             vals: age_range_vals        },
  { v: "child_age__for_child_focused_studies_", vals: child_age_vals        },
  { v: "construct_type",                        vals: construct_type_vals   },
  { v: "sample",                                vals: sample_vals           },
  { v: "measurement_tool",                      vals: measurement_tool_vals },
  { v: "item_format",                           vals: item_format_vals      },
  { v: "primary_language_s_",                   vals: primary_language_vals },
  { v: "license",                               vals: license_vals          },
])

// specify variable and prefix filters
var_inputs = ([
  { v: "variable", vals: variable },
  { v: "prefix",   vals: prefix   },
])

// The multiselect shows "rct  (178)" so the size and the coverage caveat are
// visible at the point of choosing; map the chosen labels back to bare slugs.
collection_vals = collection_sel.map(o => collection_key[collection_opts.indexOf(o)])

// Null-filter semantics: nothing selected means no constraint.
//
// This does NOT reuse overlap_null_filter. That helper does `d[t].includes(v)`,
// which is exact membership on an array but SUBSTRING matching on a string --
// and ojs_define collapses a length-1 list to a bare string, so a table in
// exactly one collection arrives as e.g. "intensive_longitudinal". Substring
// matching would then make it match a "longitudinal" filter, because
// "intensive_longitudinal".includes("longitudinal") is true. Normalise to an
// array first and compare exactly.
as_array = (x) => Array.isArray(x) ? x : (x === null || x === undefined || x === "" ? [] : [x])
collection_filter = (vals) => (d) =>
  vals.length ? as_array(d.collection).some(c => vals.includes(c)) : true

// generate filter function for each input and combine them
range_filters = range_inputs.map(({ v, vals }) => range_filter(v, vals))
overlap_filters = use_qual_filters ? overlap_inputs.map(({ v, vals }) => overlap_filter(v, vals)) : []
overlap_null_filters = var_inputs.map(({ v, vals }) => overlap_null_filter(v, vals))
collection_filters = [collection_filter(collection_vals)]
all_filters = [...range_filters, ...overlap_filters, ...overlap_null_filters, ...collection_filters]

// apply filters to metadata
ds = all_filters.reduce((mdf, fun) => mdf.filter(fun), md)
// replace "NA" with null in filtered data
dsf = ds.map(d => Object.fromEntries(Object.entries(d).map(([key, value]) => [key, value === "NA" ? null : value])))
```

:::: {.output-container}

::: {.plot-container}
::: {.plot-inputs}

```{ojs x-var}
num_vars = new Map(Object.entries(vi).map(([key, value]) => [value.label, key]))
viewof x_var = Inputs.select(num_vars, {value: "n_items", label: "X axis"})
```

```{ojs y-var}
viewof y_var = Inputs.select(num_vars, {value: "n_participants", label: "Y axis"})
```

```{ojs color-var}
cat_array = use_qual_filters ? color_vars.map((t) => [ti[t].label, t]) : []
cat_vars = new Map([["None", null], ...cat_array])
viewof color_var = Inputs.select(cat_vars, {label: "Color"})
```

:::


```{ojs scatter}
// color scheme
scheme = tol.QualMuted

// default color when there's no color variable
default_color = "darkgrey"

// color for missing values
unk_color = "lightgrey"

// indicator whether to use color
use_color = color_var !== null

// turn value into array if it isn't one
arrayify = (x) => Array.isArray(x) ? x : [x]

// given array of strings or single string, collapses to one comma separated string
stringify = (x) => x === null ? null : arrayify(x).join(", ")

// domain of values to use for color
color_vals = color_var === null ? default_color : tags[color_var].filter((v) => v !== "NA")

default_symbol = "circle"
select_symbol = "times"

default_size = 1
select_size = 4

// scatter plot
Plot.plot({
  x: {type: "log", base: vi[x_var].base},
  y: {type: "log", base: vi[y_var].base},
  width: 700,
  grid: true,
  color: {
    domain: color_vals,
    range: scheme,
    unknown: unk_color,
    legend: use_color
  },
  marks: [
    // points
    Plot.dot(dsf, {
      x: x_var,
      y: y_var,
      stroke: use_color ? (d) => stringify(d[color_var]) : default_color,
      symbol: (d) => row === null ? default_symbol : d.table !== row.table ? default_symbol : select_symbol,
      r: (d) => row === null ? default_size : d.table !== row.table ? default_size : select_size
    }),
    // regression line
    Plot.linearRegressionY(dsf, {
      x: x_var,
      y: y_var,
      stroke: use_color ? (d) => stringify(d[color_var]) : default_color
    }),
    // tooltips
    Plot.tip(dsf, Plot.pointer({
      x: x_var,
      y: y_var,
      stroke: use_color ? (d) => stringify(d[color_var]) : default_color,
      title: "table",
    }))
  ]
})
```
:::


::: {.table-container}
```{ojs result-message}
html`<i>Filtered to ${ds.length} datasets out of ${md.length} total. ${use_qual_filters
  ? "Datasets with incomplete metadata (tags, bibliography, or quantitative descriptors) are excluded from this table."
  : "Qualitative filtering is disabled, so datasets missing qualitative tags are included; datasets with incomplete bibliography or quantitative descriptors are still excluded."}</i>`
```

```{ojs table-funs}
// sparkbar element generator given max value and log base
sparkbar = (max, b) => {
  return x => htl.html`<div class="sparkbar" style="width: ${100 * log(x, b) / log(max, b)}%;">${x.toLocaleString("en")}`
}

// sparkbar element generator for a given numeric variable
var_sparkbar = (v) => sparkbar(d3.max(ds, d => d[v]), vi[v].base)

// sparkbars for each numeric variable
sparks = Object.fromEntries(Object.entries(vi).map(([key, value]) => [key, var_sparkbar(key)]))

// inline histogram element given array of values and header string
inlinehist = (vals, head) => {
  return htl.html`<span class="hist">${hist(vals)}${head}</span>`
}

// inline histogram for a given numeric variable
var_hist = (v) => inlinehist(ds.map(d => log(d[v], vi[v].base)), v)

// histograms for each numeric variable
hists = Object.fromEntries(Object.entries(vi).map(([key, value]) => [key, var_hist(key)]))

// combine two arrays into pairs in one array
zip = (a, b) => a.map((k, i) => [k, b[i]])

// set colors for array of values, using default if there are too values
pal = (v, colors, default_color) => v.length <= colors.length ? colors.slice(0, v.length) : Array(v.length).fill(default_color)

// create object with colors for each tag value
tag_colors = Object.fromEntries(Object.entries(tags).map(([tag, tag_values]) => {
  //const vals = tag_values.sort().filter((v) => v !== "NA");
  const vals = tag_values.filter((v) => v !== "NA");
  return [tag, Object.fromEntries(zip(vals, pal(vals, scheme, default_color)))]
}))

// badge element for a given tag and values
badgify = (vals, tag) => htl.html`<div>
  ${arrayify(vals).map(val => htl.html.fragment`
    <span class="badge" style="background-color: ${tag_colors[tag][val]};">${val}</span>`)}
</div>`

// badge elements for each tag
badges = Object.fromEntries(Object.keys(ti).map(tag => [tag, d => badgify(d, tag)]))
```


```{ojs table}
// data table for selected tables
viewof row = Inputs.table(dsf, {
  rows: 20,
  sort: "n_responses", reverse: true,
  required: false, multiple: false,
  //value: dsf[1],
  // show columns "table", all numeric variables, and (if enabled) all tag variables
  columns: ["table", ...Array.from(Object.keys(vi)), ...(use_qual_filters ? Array.from(Object.keys(ti)) : [])],
  // use inline histograms in header
  header: hists,
  // use sparkbars and badges in cells
  format: {...sparks, ...badges}
})
```
:::

::: {.callout-note collapse=true}

## {{< bi info-circle class=header-icon >}}Information on selected dataset

```{ojs}
// Tables that have a landing page. ojs_define collapses a length-1 list to a
// bare string and gives an empty vector as undefined, so both are normalised.
has_page = new Set(
  (typeof pilot_tables === "undefined" || pilot_tables === null
     ? []
     : (Array.isArray(pilot_tables) ? pilot_tables : [pilot_tables])
  ).map(s => String(s).toLowerCase())
)

table_page_href = (name) => `/tables/${String(name).toLowerCase()}/`
```

```{ojs}
// specify which variables to show in info box
info = ({
  "table":       { label: "Dataset"        },
  "description": { label: "Description"    },
  "variables":   { label: "Variables"      },
  "url":         { label: "Link to table"  },
  "reference":   { label: "Reference"      },
  "data_url":    { label: "Link to source" },
  "license":     { label: "License"        },
})

// element for each info entry ($head: $text) with $text wrapped in <a> if head matches "Link"
pullout = (head, text) => html.fragment`<p><b>${head}</b>: ${/Link/.test(head) ? html.fragment`<a href="${text}" target="_blank">${text}</a>` : text}</p>`

// elements for all info entries, placeholder if no table selected
placeholder = "No table selected"
row ? html`<div class="pullout">
  ${Object.entries(info).map(([field, properties]) => pullout(properties.label, row[field]))}
  ${has_page.has(String(row.table).toLowerCase())
      ? html.fragment`<p><b>Citable page</b>: <a href="${table_page_href(row.table)}">${table_page_href(row.table)}</a> &mdash; a permanent page for this table, naming the IRW version it describes.</p>`
      : html.fragment``}
</div>` : html`<div class="pullout"><em>${placeholder}</em></div>`
```
:::

::: {.callout-note}

## {{< bi code class=header-icon >}}Code snippet

The R code snippet below uses the <a href="https://itemresponsewarehouse.github.io/Rpkg" target="_blank">irw package</a> to fetch all of the tables that match the filters specified here. _Note: a difference between the functionality of the filters here and the behavior of_ `irw_filter()` _is that specifying values for a given argument of_ `irw_filter()` _will result in_ `NA` _values being dropped, while they can be included here. The "Has item text?" filter is also not reflected in the snippet, since_ `irw_filter()` _does not support filtering by item-text availability; use_ `irw_list_itemtext_tables()` _separately for that. When qualitative filtering is disabled (see the toggle above the "Filter by qualitative features" panel), none of the qualitative-tag arguments (age range, sample, license, etc.) are included in the snippet, matching the fact that no qualitative filtering is applied to the table above._

```{ojs}
// simple templating function: replace strings of the form "{{variable}}" in the
// template with the corresponding value from values
fillTemplate = (template, values) => {
  return template.replace(/\{\{(\w+)\}\}/g, (match, key) => {
    return values[key] !== undefined ? values[key] : match;
  });
};

// given array, create string representation R character vector of values
c = (vals) => `c(${vals.map(s => `"${s}"`).join(", ")})`

// construct arguments for range inputs if they're not their default values
is_default_range = (v, vals) => vals[0] === vi[v].range[0] && vals[1] === vi[v].range[1]
range_arg = (v, vals) => is_default_range(v, vals) ? null : `${v} = c(${Math.pow(vi[v].base, vals[0])}, ${Math.pow(vi[v].base, vals[1])})`
range_args = range_inputs.map(({ v, vals }) => range_arg(v, vals))

// construct arguments for overlap inputs if they're not their default values
// (only meaningful when qualitative filtering is enabled)
is_default_overlap = (v, vals) => tags[v].every(t => vals.includes(t))
overlap_arg = (v, vals) => is_default_overlap(v, vals) ? null : `${v} = ${c(vals.filter(e => e !== "NA"))}`
overlap_args = use_qual_filters ? overlap_inputs.filter(d => d.v !== l && d.v !== "has_item_text").map(({ v, vals }) => overlap_arg(v, vals)) : []

// special handling to construct var argument by combining variables and prefixes
var_arg_vals = var_inputs.map(({v, vals}) => v === "prefix" ? vals.map(s => `${s}_`) : vals).flat()
var_arg = var_arg_vals.length ? `var = ${c(var_arg_vals)}` : null

// special handling to construct longitudinal argument as boolean
l = "longitudinal"
long_vals = overlap_inputs.filter(d => d.v === l)[0].vals
long_arg = use_qual_filters && !is_default_overlap(l, long_vals) ? `${l} = ${long_vals[0] === "longitudinal" ? "TRUE" : "FALSE"}` : null

// collection argument; the OJS name must match irw_filter()'s formal exactly
collection_arg = collection_vals.length ? `collection = ${c(collection_vals)}` : null

// combine all arguments
filter_args = [...range_args, var_arg, collection_arg, ...overlap_args, long_arg].filter(e => e !== null)
filter_str = filter_args.length ? `\n  ${filter_args.join(",\n  ")}  \n` : ""

// load template and fill arguments into it
r_filter_template = FileAttachment("resources/templates/query_filter_snippet.R").text();
r_filter_snippet = () => fillTemplate(r_filter_template, { filter_str: filter_str })
```

```{ojs}
// code snippet box with copy button
ds.length === 0 ? html`<em>No datasets match filters.</em>` : html`
<div class="pullout">
<div class="code-copy-outer-scaffold">
<div class="sourceCode">
<pre class="sourceCode r code-with-copy">
<code class="sourceCode r">
<span>${r_filter_snippet()}</span></code></pre>
</div>
<button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button>
</div>
</div>
`
```

:::

::::
::::


:::: {.panel-input}

{{< bi filter >}} _Filter by qualitative features._

```{ojs filter-qual-toggle}
viewof use_qual_filters = Inputs.toggle({label: "Enable qualitative filtering", value: true})
```

```{ojs filter-qual-style}
html`<style>
#qual-filter-panel { opacity: ${use_qual_filters ? 1 : 0.45}; pointer-events: ${use_qual_filters ? "auto" : "none"}; transition: opacity 0.15s ease; }
</style>`
```

```{ojs filter-qual-note}
html`${use_qual_filters ? "" : html`<p><em>Qualitative filtering is disabled. Datasets missing qualitative tags (age range, sample, license, etc.) are now included in the results, but can't be filtered on those tags. Enable the toggle above to filter by them (this will also exclude untagged datasets again).</em></p>`}`
```

::: {.side-inputs #qual-filter-panel}

::: {.filters-container}

::: {.callout-note collapse="true"}
## {{< bi graph-up class=header-icon >}}Longitudinality

```{ojs filter-longitudinal}
viewof longitudinal_vals = tag_checkbox("longitudinal")
```
:::

::: {.callout-note collapse="true"}
## {{< bi file-text class=header-icon >}}Has item text?

```{ojs filter-has-item-text}
viewof has_item_text_vals = tag_checkbox("has_item_text")
```
:::

::: {.callout-note collapse="true"}
## {{< bi person-lines-fill class=header-icon >}}Age range

```{ojs filter-age-range}
viewof age_range_vals = tag_checkbox("age_range")
```
:::

::: {.callout-note collapse="true"}
## {{< bi person-arms-up class=header-icon >}}Child age (for child-focused studies)

```{ojs filter-child-age}
viewof child_age_vals = tag_checkbox("child_age__for_child_focused_studies_")
```
:::

::: {.callout-note collapse="true"}
## {{< bi grid class=header-icon >}}Construct type

```{ojs filter-construct-type}
viewof construct_type_vals = tag_checkbox("construct_type")
```
:::

::: {.callout-note collapse="true"}
## {{< bi people class=header-icon >}}Sample

```{ojs filter-sample}
viewof sample_vals = tag_checkbox("sample")
```
:::

::: {.callout-note collapse="true"}
## {{< bi clipboard-data class=header-icon >}}Measurement tool

```{ojs filter-measurement-tool}
viewof measurement_tool_vals = tag_checkbox("measurement_tool")
```
:::

::: {.callout-note collapse="true"}
## {{< bi sliders class=header-icon >}}Item format

```{ojs filter-item-format}
viewof item_format_vals = tag_checkbox("item_format")
```
:::

::: {.callout-note collapse="true"}
## {{< bi translate class=header-icon >}}Primary language(s)

```{ojs filter-primary-language}
viewof primary_language_vals = tag_checkbox("primary_language_s_")
```
:::

::: {.callout-note collapse="true"}
## {{< bi cc-circle class=header-icon >}}License

```{ojs filter-license}
viewof license_vals = tag_checkbox("license")
```
:::

:::
:::
::::
:::::

::::::

## Look up a single dataset

{{< include _load-data.qmd >}}

Below we show metadata for the entire IRW, an example dataset, and illustrations of how to access the data programmatically. You can also explore the data [here](https://redivis.com/datasets/as2e-cv7jb41fd/tables).

Individual tables also have their own citable pages, listed at [IRW table pages](/tables/) and linked from the panel above where one exists.

### Metadata

<iframe width="800" height="500" allowfullscreen src="https://redivis.com/embed/tables/datapages.irw_meta:bdxt:current.metadata#cells" style="border:0;"></iframe>

```{ojs}
// Load the dictionary and bib data
dictionary = data_index

function createDatasetMap(df) {
  let dataArray = Array.from({ length: df[Object.keys(df)[0]].length }, (_, i) => {
    return Object.fromEntries(Object.entries(df).map(([key, values]) => [key, values[i]]));
  });

  // Sort the array alphabetically by dataset
  dataArray.sort((a, b) => a.dataset.localeCompare(b.dataset));

  // Create the map from sorted array
  let datasetMap = new Map();
  for (let row of dataArray) {
    datasetMap.set(row.dataset, row);
  }
  return datasetMap;
}

// Create dataset map from data_index instead of metadata
dataset_map = createDatasetMap(data_index);

viewof dataset_name = {
  const allNames = Array.from(dataset_map.keys());
  let selectedName = allNames[0];

  // Attach list to body so it's never clipped by layout overflow
  const list = document.createElement("div");
  list.style.cssText = "position:fixed;z-index:99999;max-height:260px;overflow-y:auto;border:1px solid #ccc;border-radius:0 0 4px 4px;background:white;display:none;box-shadow:0 4px 12px rgba(0,0,0,0.15);";
  document.body.appendChild(list);

  const container = html`<div style="display:inline-block;width:400px;margin-bottom:20px;">
    <label style="font-weight:600;display:block;margin-bottom:4px;">Search Dataset</label>
    <div style="position:relative;">
      <svg style="position:absolute;left:8px;top:50%;transform:translateY(-50%);pointer-events:none;color:#888;" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
        <circle cx="11" cy="11" r="8"/><line x1="21" y1="21" x2="16.65" y2="16.65"/>
      </svg>
      <input type="text" placeholder="Type to search datasets…"
        style="width:100%;padding:6px 10px 6px 28px;font-size:smaller;border:1px solid #ccc;border-radius:4px;box-sizing:border-box;" />
    </div>
  </div>`;

  const input = container.querySelector("input");

  function positionList() {
    const r = input.getBoundingClientRect();
    list.style.top = r.bottom + "px";
    list.style.left = r.left + "px";
    list.style.width = r.width + "px";
  }

  function renderList(filter) {
    const matches = filter
      ? allNames.filter(n => n.toLowerCase().includes(filter.toLowerCase()))
      : allNames;
    list.innerHTML = "";
    for (const name of matches.slice(0, 300)) {
      const item = document.createElement("div");
      item.textContent = name;
      item.style.cssText = "padding:5px 10px;cursor:pointer;font-size:13px;border-bottom:1px solid #f0f0f0;";
      item.onmousedown = (e) => {
        e.preventDefault();
        selectedName = name;
        input.value = name;
        list.style.display = "none";
        container.value = selectedName;
        container.dispatchEvent(new Event("input", {bubbles: true}));
      };
      item.onmouseenter = () => item.style.backgroundColor = "#f0f4ff";
      item.onmouseleave = () => item.style.backgroundColor = "";
      list.appendChild(item);
    }
  }

  input.addEventListener("focus", () => {
    input.value = "";
    renderList("");
    positionList();
    list.style.display = "block";
  });
  input.addEventListener("input", (e) => {
    e.stopPropagation();
    renderList(input.value);
    positionList();
    list.style.display = "block";
  });
  input.addEventListener("blur", () => {
    setTimeout(() => { list.style.display = "none"; input.value = selectedName; }, 150);
  });

  function onScroll(e) { if (list.contains(e.target)) return; list.style.display = "none"; input.value = selectedName; }
  window.addEventListener("scroll", onScroll, true);

  invalidation.then(() => { list.remove(); window.removeEventListener("scroll", onScroll, true); });

  input.value = selectedName;
  container.value = selectedName;
  return container;
}

// Function to find dataset info from dictionary (fixed field name)
function findDatasetInfo(dictionary, datasetName) {
  // Convert dictionary column data into array of objects
  const dictionaryArray = Array.from({ length: dictionary[Object.keys(dictionary)[0]].length }, (_, i) => {
    return Object.fromEntries(Object.entries(dictionary).map(([key, values]) => [key, values[i]]));
  });

  // Now we can use find on the array
  const dictRow = dictionaryArray.find(row => row.dataset === datasetName);
  return {
    reference: dictRow?.reference || "Reference not available",
    BibTex: dictRow?.BibTex || "BibTeX not available",
    doi: dictRow?.doi
  };
}

dataset_info = findDatasetInfo(dictionary, dataset_name);

// Create the display elements with reference and BibTeX
display = {
  const sanitizedDatasetName = dataset_name.replace(/\./g, "_"); // Replace all dots with underscores
  const dsRefIdx = table_ds_ref.table.findIndex(t => t === dataset_name.toLowerCase());
  const dsRef = dsRefIdx >= 0 ? table_ds_ref.ds_ref[dsRefIdx] : "item_response_warehouse:as2e";
  const url = `https://redivis.com/embed/tables/datapages.${dsRef}:current.${sanitizedDatasetName}`;

  const container = html`<div>
    <div style="margin-bottom: 20px;">
      <button onclick="toggleInfo(this)" style="
        padding: 8px 16px;
        background-color: #4287f5;
        color: white;
        border: none;
        border-radius: 6px;
        cursor: pointer;
        font-weight: 500;
        display: flex;
        align-items: center;
        gap: 5px;
        transition: background-color 0.2s;
        box-shadow: 0 2px 4px rgba(0,0,0,0.1);
      ">
        <span>Show Citation Info</span>
        <svg width="12" height="12" viewBox="0 0 12 12" fill="none" style="transform: rotate(0deg); transition: transform 0.3s;">
          <path d="M2 4L6 8L10 4" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
        </svg>
      </button>

      <div style="
        display: none;
        margin-top: 10px;
        padding: 20px;
        background-color: #ffffff;
        border-radius: 8px;
        border: 1px solid #e1e4e8;
        transition: all 0.3s ease;
        box-shadow: 0 2px 8px rgba(0,0,0,0.05);
      ">
        <div style="display: flex; gap: 20px;">
          <div style="flex: 1;">
            <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 5px;">
              <div style="font-weight: 600; color: #24292e;">Reference</div>
              <button onclick="copyText(this, 'reference')" style="
                padding: 4px 12px;
                background-color: #f6f8fa;
                border: 1px solid #e1e4e8;
                border-radius: 6px;
                cursor: pointer;
                font-size: 0.8em;
                color: #24292e;
                display: flex;
                align-items: center;
                gap: 4px;
                transition: all 0.2s;
              ">
                <svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor">
                  <path d="M8 4v12a2 2 0 0 0 2 2h8a2 2 0 0 0 2-2V7.242a2 2 0 0 0-.602-1.43L16.083 2.57A2 2 0 0 0 14.685 2H10a2 2 0 0 0-2 2z" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
                  <path d="M16 18v2a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V9a2 2 0 0 1 2-2h2" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
                </svg>
                Copy
              </button>
            </div>
            <div style="
              font-size: 0.95em;
              line-height: 1.6;
              color: #24292e;
              font-style: italic;
              background-color: #f8f9fa;
              padding: 12px;
              border-radius: 6px;
              border: 1px solid #e1e4e8;
            " class="reference-text">${dataset_info.reference}</div>
          </div>
          <div style="flex: 1;">
            <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 5px;">
              <div style="font-weight: 600; color: #24292e;">BibTeX</div>
              <button onclick="copyText(this, 'bibtex')" style="
                padding: 4px 12px;
                background-color: #f6f8fa;
                border: 1px solid #e1e4e8;
                border-radius: 6px;
                cursor: pointer;
                font-size: 0.8em;
                color: #24292e;
                display: flex;
                align-items: center;
                gap: 4px;
                transition: all 0.2s;
              ">
                <svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor">
                  <path d="M8 4v12a2 2 0 0 0 2 2h8a2 2 0 0 0 2-2V7.242a2 2 0 0 0-.602-1.43L16.083 2.57A2 2 0 0 0 14.685 2H10a2 2 0 0 0-2 2z" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
                  <path d="M16 18v2a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V9a2 2 0 0 1 2-2h2" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
                </svg>
                Copy
              </button>
            </div>
            <pre style="
              margin: 0;
              padding: 12px;
              background-color: #f8f9fa;
              border-radius: 6px;
              border: 1px solid #e1e4e8;
              font-size: 0.9em;
              line-height: 1.5;
              overflow-x: auto;
              white-space: pre-wrap;
              word-wrap: break-word;
              color: #24292e;
              font-family: 'Monaco', 'Menlo', 'Ubuntu Mono', 'Consolas', 'source-code-pro', monospace;
            " class="bibtex-text"></pre>
          </div>
        </div>

        <!-- R Package Citation -->
        <div style="margin-top: 20px;">
          <button onclick="toggleRCitation(this)" style="
            padding: 6px 12px;
            background-color: #4287f5;
            color: white;
            border: none;
            border-radius: 6px;
            cursor: pointer;
            font-size: 0.9em;
            transition: background-color 0.2s;
            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
          ">To cite using IRW's R package</button>

          <div class="r-citation" style="
            display: none;
            margin-top: 10px;
            padding: 12px;
            background-color: #f8f9fa;
            border-radius: 6px;
            border: 1px solid #e1e4e8;
            font-family: monospace;
          ">
            <div style="
              padding: 8px;
              background-color: #f8f9fa;
              border-left: 3px solid #4287f5;
              margin-bottom: 10px;
            ">
              > <code>irw_citation('${dataset_name}')</code>
            </div>
          </div>
        </div>
      </div>
    </div>
    <iframe id="myIframe" width="800" height="500" allowfullscreen style="border: 1px solid #e1e4e8; border-radius: 8px;" src="${url}"></iframe>
  </div>`;

  // Add toggle functions to window object
  window.toggleInfo = function(button) {
    const infoDiv = button.nextElementSibling;
    const arrow = button.querySelector('svg');
    const isVisible = infoDiv.style.display === 'block';

    infoDiv.style.display = isVisible ? 'none' : 'block';
    arrow.style.transform = isVisible ? 'rotate(0deg)' : 'rotate(180deg)';
    button.querySelector('span').textContent = isVisible ? 'Show Citation Info' : 'Hide Citation Info';
  };

  window.toggleRCitation = function(button) {
    const citationDiv = button.nextElementSibling;
    citationDiv.style.display = citationDiv.style.display === 'none' ? 'block' : 'none';
  };

  // Add copy function to window object
  window.copyText = function(button, type) {
    const container = button.closest('div').parentElement;
    const text = type === 'reference' ?
      container.querySelector('.reference-text').textContent :
      container.querySelector('.bibtex-text').textContent;

    navigator.clipboard.writeText(text).then(() => {
      const originalText = button.textContent;
      button.textContent = 'Copied!';
      button.style.backgroundColor = '#28a745';
      button.style.borderColor = '#28a745';
      button.style.color = 'white';

      setTimeout(() => {
        button.innerHTML = `
          <svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor">
            <path d="M8 4v12a2 2 0 0 0 2 2h8a2 2 0 0 0 2-2V7.242a2 2 0 0 0-.602-1.43L16.083 2.57A2 2 0 0 0 14.685 2H10a2 2 0 0 0-2 2z" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
            <path d="M16 18v2a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V9a2 2 0 0 1 2-2h2" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
          </svg>
          Copy
        `;
        button.style.backgroundColor = '#f6f8fa';
        button.style.borderColor = '#e1e4e8';
        button.style.color = '#24292e';
      }, 2000);
    });
  };

// Update BibTeX text
const bibtexElement = container.querySelector('.bibtex-text');
if (bibtexElement) {
  bibtexElement.textContent = dataset_info.BibTex || "BibTeX not available";
}

  return container;
}
```

_If you have any questions or feedback, please feel free to [contact us](contact.qmd)._