Inconsistent division tracking

[holgerbrandl]

We should perform a consistency check - every cell appearing by division should be a daughter of some cell and daughters of each dividing cell should be cells in the database! Should we flag these cells also as segmentation errors?

movieDir="/Users/brandl/Desktop/WT_25deg_111102_ForTissueMiner"
db_name=basename(movieDir)

Sys.setenv(TM_HOME="/Volumes/projects/project-raphael/scripts/tissue_miner")
scriptsDir=Sys.getenv("TM_HOME")

source(file.path(scriptsDir, "commons/TMCommons.R"))

## [1] "using config file /Volumes/projects/project-raphael/scripts/tissue_miner/config/default_config.R"

movieDb <- openMovieDb(movieDir)

cellinfo <- dbGetQuery(movieDb, "select * from cell_histories")

Every cell appearing by division should be a daughter of some cell

daughterInfo <- cdByDaughters(movieDb)


ciWithMother <- left_join(cellinfo, daughterInfo %>% select(-first_occ))

ciWithMother %>% count(appears_by, !is.na(mother_cell_id))

## Source: local data frame [6 x 3]
## Groups: appears_by [?]
## 
##         appears_by !is.na(mother_cell_id)     n
##              (chr)                  (lgl) (int)
## 1         Division                  FALSE   212
## 2         Division                   TRUE 34702
## 3    MovedIntoMask                  FALSE 12339
## 4    MovedIntoMask                   TRUE    14
## 5 SegErrAppearance                  FALSE  2669
## 6     Unclassified                  FALSE  9380

divWithoutMother <- filter(ciWithMother, is.na(mother_cell_id) & appears_by=="Division")

divWithoutMother %>% count(first_occ)

## Source: local data frame [125 x 2]
## 
##    first_occ     n
##        (int) (int)
## 1          2     1
## 2          3     1
## 3          4     3
## 4          5     1
## 5          8     1
## 6          9     2
## 7         10     1
## 8         12     3
## 9         13     1
## 10        14     1
## ..       ...   ...

divWithoutMother %>% nrow

## [1] 212

divWithoutMother %>% count(generation)

## Source: local data frame [1 x 2]
## 
##   generation     n
##        (int) (int)
## 1          0   212

This means that the problematic cases are tied to the first generation which seem to be incorrectly tagged as being the result of a division.
Those flags seem to be directly extracted from cell_in_frame.dat, so it could be parser problem?

Daughters of each dividing cell should be cells in the database

allDaughters <- cellinfo %$%
    c(right_daughter_cell_id, left_daughter_cell_id) %>%
    na.omit() %>% data_frame(cell_id=.)
allDaughters %>% anti_join(cellinfo)  %>% nrow

## [1] 0

Seems fine at least for WT_25deg_111102_ForTissueMiner

[etournay]

These ideas are excellent indeed. I plan on incorporating this consistency check very soon.

So far the orphan cells are all localized at the marging of the segmentation mask.

• The parser systematically defines a margin object corresponding to "cell contours" overlapping the image margin. This allows to discard cell contours that overlap the image margin, which is good as they don't represent a biological cell.
• In contrast, TissueAnalyzer tracks every contour including its lineage upon "division". In TissueAnalyzer, a "cell contour overlapping the image margin" may divide. If one of the daughter cells doesn't overlap the image margin, the parser would still detect a "daughter cell" by reading the TIssueAnalyzer division mask, but won't be able to assing a mother cell (which was discarded).
• It is a margin effect which can easily be checked and corrected by reassigning a "MovedIntoMask" label in the cell_histories table (before building the DB).
• We can also check if an orphan occurs within the tissue (> 2 rows away from the margin). The good news is that I haven't found such case. But if if occurs, one could reassign a "TrackingErrAppearance" label in the cell_histories table (before building the DB).

@mpopovichr
Just to let you know that it seems to be easy to fix

[etournay]

Are orphan cells all in contact to the margin of the segmentation mask ?

db <- movieDb 

# 1/ Define neighbor relationship in each frame
dbonds <- dbGetQuery(db, "select cell_id, frame, dbond_id, conj_dbond_id, left_dbond_id from directed_bonds")

cellNeighbors <- with(dbonds, data.frame(frame, cell_id, dbond_id, left_dbond_id)) %>%
  dt.merge(with(dbonds, data.frame(dbond_id=conj_dbond_id, cell_id)), by=c("dbond_id")) %>% 
  select(frame, cell_id=cell_id.x, neighbor_cell_id=cell_id.y)

# 2/ Check symmetry
if (nrow(cellNeighbors %>% filter(neighbor_cell_id=="10000"))==nrow(cellNeighbors %>% filter(cell_id=="10000"))){
  print("OK: symmetric neighbor relationship at the margin")
} else warning("The neighbor relationship isn't symmetric at the margin")

# 3/ Identify border cells: 2 rows of cells away from the margin cell 10000 
borderCells <- cellNeighbors %>%
  filter(cell_id!="10000") %>% # simplify due to assumed symmetry of neighbor relationship at the margin
  mutate(isMarginNeighbor = ifelse(neighbor_cell_id=="10000", TRUE, FALSE)) %>% 
  group_by(frame, cell_id) %>%
  summarise(isCellTouchingMargin=any(isMarginNeighbor)) %>% 
  filter(isCellTouchingMargin) %>% print_head()

# borderCells %>% dt.merge(dbGetQuery(db, "select cell_id, frame, center_x, center_y from cells")) %>%
#   render_frame(15) + 
#   geom_point(aes(center_x, center_y), color="red")

borderCellsTwoRows <- cellNeighbors %>%
  filter(cell_id!="10000") %>%
  dt.merge(with(borderCells, data.frame(frame, neighbor_cell_id=cell_id)), by = c("frame", "neighbor_cell_id")) %>% 
  select(-neighbor_cell_id)%>% print_head()

# borderCellsTwoRows %>% dt.merge(dbGetQuery(db, "select cell_id, frame, center_x, center_y from cells")) %>%
#   render_frame(15) +
#   geom_point(aes(center_x, center_y), color="green")


# 4/ Check if orphan cells are in contact to the margin or to border cells (max 2 rows from the margin)
orphanMarginCheck <- divWithoutMother %>%
  rename(frame=first_occ) %>% 
  select(frame, cell_id, appears_by) %>%
  dt.merge(borderCellsTwoRows, by = c("frame", "cell_id"), all.x=T) %>% print_head()

if (all(complete.cases(orphanMarginCheck))){

  print("All orphan cells are present at the margin: change label to MovedIntoMask")

} else {

  warning("Some cell lineages are broken: some divided cells do not have a mother cell within the tissue:")
  brokenLineages <- orphanMarginCheck[!complete.cases(orphanMarginCheck),] %>% print_head()
  print("Change label to TrackingErrAppearance")

  marginOrphan <- orphanMarginCheck[complete.cases(orphanMarginCheck),] %>% print_head()
  print("Change label to MovedIntoMask")

}

# orphanMarginCheck %>% dt.merge(dbGetQuery(db, "select cell_id, frame, center_x, center_y from cells")) %>%
#   render_frame(15) +
#   geom_point(aes(center_x, center_y), color="green")

So far, YES, they are at the margin only !

[etournay]

@holgerbrandl
What is the most elegant way to fix this ? Before or after the DB creation ?
Cheers

[holgerbrandl]

Let's wait for Matthias' reply since a parser fix would be the best solution here. TM is doing the right thing here imho by keeping the appears_by categorization provided by the parser.