Anyway simpleschema, collection2 and peerdb can work together

[emmanuelbuah]

First of all, I want to say kudos for putting in the work to get a reactive layer on top of mongo (server-side) to handle declarative updates. Both collection2 and simpleschema solve specific issues, same as peerdb. I personally see peerdb more of a better package to replace collection-hooks but I wonder if peerdb api can be made to work with collection2 and simpleschema. I really see 3 concerns here:

Data schema integrity and validation -> SimpleSchema
Collection data schema validation on insert, upsert and update -> Collection2
Provide Collection data hooks on insert, upsert, update and delete -> Collection-hooks || Peerdb

I will personally like to use Peerdb over collection-hooks because of its reactive nature and the ability to handle hooks even if the changes are made by another client. For me peerdb is a clear winner but I cant seem to feel like it will even be much better if it can work in tandem with SimpleSchema and Collection2.

My question is - how do I get the benefit of simpleschema + collection2 + peerdb. Is it be possible to integrate all 3 to work together. Will it be better to integrate peerdb definitions into simpleschema since both of them are in a way defining metadata information. Example, If would be great if we could do something like

//data validation provided by simpleschema 
PostSchema = new SimpleSchema({
    content: {
        type: String,
        optional: false,
    },
   author: {
     type:Object,
     optional: false
   },
   "author.username": {
      type: String
  },
   "author.displayName": {
      type: String
  },
   hits: {
        type: Number
    }
   //etc
});

//peerdb hook through SimpleSchema
PostSchema.References({
    fields: function() {
      return {
        author: Post.ReferenceField(Person, ['username', 'displayName']),
      };
    }
  });

PostSchema.Triggers({
    //trigger definition here ....
  });

Post = new Meteor.Collection('posts');
Post.attachSchema(PostSchema); //provided by collection2

In the approve sample code, simpleschema handles validation (i.e. make sure author's username and displayname is a string), collection2 handle running validation and peerdb extension to simpleschema + collection2 maintains data references and hooks. This is just a thought but I think something along these lines will provide the community with 80% of what its needed for data manipulation. I want to use peerdb now but I also don't want to loose my schema validation checks on my collection (as well as on the client through autoform)

[mitar]

So how I imagine this to work is that I extend definitions of fields also to include non-reference fields and then validate documents in objectify. Once we have that we can also add a way to export schema definition as simple-schema, so that you can use it in meteor-autoforum. The issue I have by using simple-schema internally as well is that PeerDB support references, so schema returned will have to be normalized for simple-schema output (so subdocuments will not be seen as references), while objectify might be able to recursively go and check things because it will know where are subdocuments in fact instances of another document.

[mitar]

So I think that PeerDB replaces collection2, but can use simple-schema for compatibility with other packages.

[emmanuelbuah]

@mitar Thanks for your response. You know best but I think integration with simpleschema will increase peerdb adaptation since its widely used for Meteor projects. Another place of concern (subjective) is the name and constructor api.

Name

When I saw peerdb, I thought it was a new reactive/ddp database like redis-livedata. Just like you said, its a better collection2 so might be better to give it a name relating to mongo/collections. Examples peer:meteor.collection or peer:collection. This will make it easy for people to identify its use at first glance.

Definition api

In relation to collection definition api, I love coffeescript but I think I'm right to say there are a lot more people pure javascript devs than coffeescrip. With that said, the current peerdb document definition api seems a bit ....verbose if written in javascript by hand. Below is a person document definition in pure javascript without coffeescript

var Person,
  __hasProp = {}.hasOwnProperty,
  __extends = function(child, parent) { for (var key in parent) { if (__hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor(); child.__super__ = parent.prototype; return child; };

Person = (function(_super) {
  __extends(Person, _super);

  function Person() {
    return Person.__super__.constructor.apply(this, arguments);
  }

  Person.Meta({
    name: 'Person'
  });

  return Person;

})(Document);

Compared to Person collection definition is Mongo.Collection or Collection2

  var Person = new Meteor.Collection('Person')

Is it possible to make the metadata definition api as simple (hide the inheritance complexity under a simple api). Example. Peerdb collection/document can be defined using the same kind of definition style as Meteor.Collection/Collections2 and simpleschema such as ...

//definition 

 var Person =  new Peer.Collection('Person',{
    idGeneration: 'STRING'      //options passed to Mongo.Collection
   } ,
  fields :{
      username: {
        type: String,
        optional: false,
      },
      ...
  });
  var Posts =  new Peer.Collection('Posts',{
    idGeneration: 'MONGO'      //options passed to Mongo.Collection
   } ,
  fields :{
      content: {
        type: String,
        optional: false,
      },
      author: {
       type:  Object
       reference: ReferenceField(Person, ['username', 'displayName']),  
       optional: false
     },
     subscribers: {
       type:  [Object]
       reference: [ReferenceField(Person)],
       optional: true
     }
  }
}

//fetch
Posts.find({}).fetch()
//schema 
Posts.schema() 

This is all subjective. I just wanted to offer my thoughts as a dev and supporter. Keep up the great work.

[Sivli-Embir]

I personally dislike SimpleSchema but I think @emmanuelbuah makes a lot of good points. I would agree the name could be better. Also while I love Coffeescript and I feel @emmanuelbuah is right people will want a choice between pure JS and Coffee.

[mitar]

Just for the record, name stays. :-) I have many projects starting with "peer" in this is one of them. :-) It is a family of related projects and while PeerDB currently resembles collection2, it has quite some other features in the future I am planning to implement. Like versioning. My goal with the project is to create an easy way to build collaborative tools on top of Meteor. And PeerDB is a layer for that. If others find some other uses to it, even better. But in my context PeerDB is a nice name for something which helps collaboration.

About JS integration: I would welcome a pull request with somebody making an alternative API which you can call with something like Document.create(...) or something, which is more suitable for JS programmers. And which would then internally extend CoffeeScript class and do the rest. Maybe it could be simply something similar to what you pass to Meta. So same function or something. (You need function to be able to define lazy references between documents. So a way how SimpleSchema works cannot really work once you have references.) So, some boiler-plate code for making it easy to use in JavaScript.

About SimpleSchema integration: I welcome also a pull request which would add to Meta a method which converts it into SimpleSchema compatible schema. So once you create a document, you can access fields for example with Post.Meta.fields, so there could be Post.Meta.simpleSchema(). This can then be used to pass to autoform and similar packages.

For that to work we first have to create also other fields, not just references. So the idea is to create similar classes for other types of fields, where then community can create even more specialized fields.

It would be interesting also to create fields which allow serialization/deserialization to internal MongoDB types from higher-level types (think IP field), but on the other hand Meteor does this through EJSON, so an idea would be that defining a PeerDB document would also based on fields define an EJSON type for direct serialization from JavaScript document. Just some ideas.

I am designing PeerDB to be pretty similar to how Django (or Mongoose) is doing things (class-based, having abstract class, using Meta and document managers). So if anyone like to implement parts of that (for example, field classes for various basic types found in MongoDB) in similar spirit, that would be a pull request I would be happy about. (So if you ever wonder how to design something that I would gladly merge, think from Django/Mongoose approach.)

[mitar]

I see above also another implicit suggestion, to allow other forms of ID generation. I think that could be made into another ticket and could become part of Meta definition if somebody wants to tackle. For now, you can always define class like:

class Post extends Document
  @Meta =>
     name: 'Post'
     collection: new Meteor.Collection 'Posts', idGeneration: 'MONGO'

I think it should work, but I have not tried it. Feel free to make a pull request with a test. (We already have 0.5 MB of tests, so why not more.)

[emmanuelbuah]

@mitar Yes. I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition. I noticed your PeerLibrary project a few days back. I think its a great project. I actually browsed some of the other relating libs. I will spend sometime to digest your feedback in relation to where help is needed. I might be able to take some of the tasks.

[mitar]

I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition.

I would not pass things directly, but provide configuration options for Meta.

[mitar]

I just published another package, a way to define publish middleware.

[testbird]

I can confirm that the name "peerdb" had me overlook this project, because it sounded like an alternative db backend, not like bringing a missing completion for meteor collections.

BTW: The corresponding collection2 issue is here: Meteor-Community-Packages/meteor-collection2#31 (simple-schema definitions will get references/relations support)

[testbird]

Thank you for your name explanations above mitar. I think it created the understanding in me that peerdb is not at the database level, but above meteor collections, it provides "peer collections" or "peer documents" :-)

[testbird]

Concerning the new "meteor-middleware" package you mentioned above, please allow me to comment that its naming may be non-self-descriptive for the community as well. (Making it unfortunately and unnecessarily hard to find, notice and grasp.)

If i understant it at correctly, the package improves what meteor users know as publish functions, and provides stackable publish hooks? May this be described as "peer publishing stack" or simmilar?

[mitar]

:-)

Ah, lovely naming discussions. :-)