On Demand API

[knapply]

As discussed in #52 (review)

How best to leverage simdjson's On Demand API?

[knapply]

@jkeiser, @lemire Following up on your suggestion from here after I played with it a bit.

Are there any examples of iterating through arrays of numbers, strings, bools, etc.?

I can only seem to find examples accessing the scalar values in objects using their keys.

As an example (I was trying this out on some GeoJSON), how would we access the individual points in the following (after accessing the coordinates with something like auto coords = doc["coordinates"])?

{
  "type": "LineString", 
  "coordinates": [
    [1, 2], [3, 4], [5, 6]
  ]
}

For some R context, we would need to turn "coordinates" into a 3x2 matrix (no offense intended if you're R aficionados - I don't want to assume)...

geojson <- '{
  "type": "LineString", 
  "coordinates": [
    [1, 2], [3, 4], [5, 6]
  ]
}'

parsed <- RcppSimdJson::fparse(geojson)
parsed
#> $type
#> [1] "LineString"
#> 
#> $coordinates
#>      [,1] [,2]
#> [1,]    1    2
#> [2,]    3    4
#> [3,]    5    6

... and since R is column-major, we're really building the following (with attached dimension attributes):

as.vector(parsed$coordinates)
#> [1] 1 3 5 2 4 6

[jkeiser]

I'd expect it to look like:

int i=0;
for (auto point : doc["coordinates"]) {
  int j=0;
  for (double val : point) {
    matrix[i][j] = val;
    j++;
  }
  i++;
}

[knapply]

Thank you. Is there a way to get an array's size (so we can tell how big matrix needs to be)?

[lemire]

@knapply rbind?

[eddelbuettel]

No, for two reasons. Those things tend to be performance killers as the underlying R data structures don't grow easily. So a general Rcpp pattern is to collect everything in C++ (hello old friend STL) and then convert at end. Plus, rbind is at the R level and we are not going 'up and down' all the time. (Unless I misread your suggestion ...)

[lemire]

I was half kidding.

[eddelbuettel]

(And @lemire for example on of my all-time fave little tools in R is data.table::rbindlist(someList) which scans your list, allocates properly and then inserts. Runs circles around the standard do.call(rbind, someList).)

[eddelbuettel]

Hah! Point taken with a grin!

[lemire]

:-)

[knapply]

Edit... welp, I typed too slow 😬

If only. I think I over complicated the question with a matrix.

Let's say we have an array...

[1, 2, 3, 4, 5,  6]

... and we want to insert the data into a vector (R/Rcpp or STL) equivalent to this...

std::vector<double>{1, 2, 3, 4, 5,  6};

We would need to know the size of the array...

std::vector<double> out(std::size(array));     // how do we do this?

int i = 0;
for (double element : array) {
  out[i++] = element;
 }

[lemire]

std::vector<double> out;
for (double element : array) { out.push_back( element); }

[knapply]

... still half kidding? :D

(I can't tell if knowing the size would even be possible with the On Demand API, but it sorta make this a non-starter).

[lemire]

The above code is standard C++, although there is room for optimizations....
https://lemire.me/blog/2012/06/20/do-not-waste-time-with-stl-vectors/