SDR: Implement ways to measure quality of produced SDR

[breznak]

Relevant classes:

• SpatialPooler
• SDR
• Topology

Why?

• when making changes to SP, we don't have ways to meansure the quality of its outputs: SDRs.

Functionality:
SDR = sparse distributed representation

• sparse:
  • min, max active bits in SDR, compared to % size
  • avg distance between active bits in SDR (should be similar for all SDRs), uses Topology
• distributed:
  • uses activeDutyCycles of the active bits (=cols) to see that all cols are used equally
  • information/entropy of the bit/SDR

Implementation:

• helper method to SP (SDR?)

Hypothesis:

• what is a "quality" SDR?
• does higher quality SDR translate to better (how?) results? (in what?)

EDIT: latest update 14/01/2019

Update: Implemented in PR #184

• SDR Sparsity Metrics
• SDR Activation Frequency Metrics
• SDR Average Overlap Metrics
• SDR All Metrics Convenience Class

Summary: Ideas which are discussed here but not yet implemented:

• Cell death (via method SDR.killCells)
• SDR topology
• SP noise resistance (via method SDR.addNoise, also the example sp_tutorial will demonstraight this)
• SP long term stability
• TM estimate false positive & negative rates
• Test Hypothesises
• Write about how to measure HTM's using these metrics

[dkeeney]

when making changes to SP, we don't have ways to meansure the quality of its outputs: SDRs.

Yes, I agree. Having some sort of measure would be very useful. 👍

[ctrl-z-9000-times]

Great idea!

I would add stats: min/mean/std-dev/max for ac tiveDutyCycles, and then binary entropy which is a single fraction (in range 0-1) which describes utilization.

Ths SDR class has a hook which is called everytime its value is updated, could be useful for this task?

[breznak]

I would add stats: min/mean/std-dev/max for ac tiveDutyCycles

So maybe in 2 ways of implementation; as for the first type, I'd like only metrics that is computed instantly, just from the SDR. To make it simpler (no logic needs to be added to SP), and faster.

entropy .. which describes utilization.

For a single bit, whole SDR, or the SP?

[ctrl-z-9000-times]

We could split these metrics into different methods, and then have a print method which calls all four. Then the min/max can be computed fast and separately, but the print method (which is typically only called once at end of program) can display all of the stats.

class SDR_ActivationFrequency {
    SDR_ActivationFrequency( SDR &dataSource );
    Real min();
    Real max();
    Real mean();
    Real std();
    Real entropy();
    String pretty_print(); // Uses all the metrics.
}

entropy .. which describes utilization.

For a single bit, whole SDR, or the SP?

Entropy is for the activation frequency of the SDR as a whole. Here is my python function for it:

def _binary_entropy(p): // p is an array of floats in range [0, 1]
    p_ = (1 - p)
    s  = -p*np.log2(p) -p_*np.log2(p_)
    return np.mean(np.nan_to_num(s))

Then to scale entropy into range [0, 1] simply divide by the theoretical maximum entropy which is: entropy(mean(activationFrequency)).

[ctrl-z-9000-times]

min, max active bits in SDR, compared to % size

Another good idea, to which I would add mean & std. Min & max tell you about the extremes & outliers, which can be helpful for spotting bugs. Mean & std tell you about its normal operating behaviour.

Yet another interesting metric to track is: Average overlap between consecutive assignments to an SDR. This measures how quickly an SDR changes, sort of like a derivative. I have in past experiments used this to measure the quality of encoders, w/ regards to semantic similarity property. I've also used this metric in experiments with Layer 2/3 cell stability / view-point invariance.

[breznak]

split these metrics into different methods,

Yes, I'd like the metric to provide a mapping to [0, 1], but (also) return the separate stats.

Mean & std tell you about its normal operating behaviour.

First I've thought of "quality" as a one-shot measure of an SDR, you're suggesting to add statistics over the run of the program on the dataset (which is a good thing!) Only if these should be separate? Quality of SDR, and stats of SP. Or keep it together in one.

def _binary_entropy(p): // p is an array of floats in range [0, 1]

And the p here is? activation freq for each column(bit) after N runs?

Yet another interesting metric to track is: Average overlap between consecutive assignments to an SDR

What do you mean by this? If it's overlap between 2 consecutive (any) SDR values produced by SP, that imho has no meaning, as these do not have to be anyhow correlated...?

[ctrl-z-9000-times]

I think a good way to organize all of these would be to give each metric its own class. Then create a class named SDR_Metrics which would gather up all of the metrics into a single easy to use package.

Each metric could follow a common design pattern, such as:

SDR_MetricName {
    void SDR_MetricName( SDR &dataSource, ... );
    Real statistics(); // Min Mean Std Max
    String print();
    void save( ... );
    void load( ... );
}

[breznak]

• I'm looking into measuring the following property of an SDR:
  "Distributed = each bit can be REused in several different contexts(SDRs), and a collection of multiple bits is unique (a SDR)"

We can evaluate sparsity quite well, but this distributed-ness? Would information/entropy over column activations over the run over dataset (too much over-s :D) be enough? In SP we can use (active)DutyCycles as well...

[breznak]

I would like to tyrn this into a paper. The main ideas are:

• we can&should measure quality of the encoding (SDRs) - how? What features?
• (How) does the quality correlate with good algorithm results? (prediction & anomaly)
• Compare quality of (output) encodings of other ML algorithms. (Which? Only sparse representations?)
  • sparse auto-encoders
  • cortical.io retina (SDRs for NLP)
  • biological (BCI data from which regions, retina, ...)

[ctrl-z-9000-times]

And the p here is? activation freq for each column(bit) after N runs?

Yes.

What do you mean by [average overlap]? If it's overlap between 2 consecutive (any) SDR values produced by SP, that imho has no meaning, as these do not have to be anyhow correlated...?

This is only relevant for time-series datasets. The encoder output should have an overlap when the input value is slowly and smoothly moving, which indicates semantic similarity between encoded values. The SP should have very little overlap because it should map similar inputs to distinctive outputs. The column-pooler should have a significant average overlap because it is supposed to do view-point invariance.