Skip to content

Updating master and background-rm (includes updated README) #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
a71f10e
initial slot machine algo and watershed bg removal
shreystechtips Nov 9, 2020
bd7e4e3
Merge branch 'master' of github.com:berkeleyauv/perception into shrey…
shreystechtips Nov 9, 2020
4561021
add haze and background removal initia files
shreystechtips Nov 9, 2020
2e15eb3
initial slot machine algo and watershed bg removal
shreystechtips Nov 9, 2020
b1460ae
add haze and background removal initia files
shreystechtips Nov 9, 2020
9b01bc7
Merge branch 'background-rm' of https://github.com/berkeleyauv/percep…
KDharmarajanDev Nov 15, 2020
1d32050
Added depth_map as a TaskPerceiver
KDharmarajanDev Nov 15, 2020
23eb535
adapt Background Removal for the visualizer
shreystechtips Nov 22, 2020
ddc3d82
Added DepthMap With Histogram For Vis
KDharmarajanDev Nov 22, 2020
efa33f4
PCA experimentation with Depth
KDharmarajanDev Nov 29, 2020
960ec72
add temporal blur function to remove background artifacts
shreystechtips Nov 29, 2020
8c5bfb9
Merge branch 'background-rm' of github.com:berkeleyauv/perception int…
shreystechtips Nov 29, 2020
2743f1d
Cleaned Up Various Algorithms
KDharmarajanDev Jan 18, 2021
12438dc
Deleted algo_stats
KDharmarajanDev Jan 18, 2021
d0c8718
Merge pull request #38 from berkeleyauv/master
KDharmarajanDev Jan 18, 2021
edec2da
Cleaned Up Imports
KDharmarajanDev Jan 18, 2021
2f735a5
switch to knn based background algo
shreystechtips Jan 25, 2021
3b6e4bb
Added MBD
KDharmarajanDev Jan 30, 2021
96c2d14
Added MBD to Code
KDharmarajanDev Jan 30, 2021
27c705a
Cythonized the saliency_mbd
KDharmarajanDev Jan 31, 2021
ad9dbf6
Removed Internal Resizing
KDharmarajanDev Feb 8, 2021
e95956f
add bitwise and for saliency and knn
shreystechtips Feb 8, 2021
b95e585
Merge branch 'background-rm' of github.com:berkeleyauv/perception int…
shreystechtips Feb 8, 2021
e06c29a
Fixed prev returned frame
KDharmarajanDev Feb 13, 2021
99e3b1e
Updated combined file
KDharmarajanDev Feb 15, 2021
db0149e
Removed experiment
KDharmarajanDev Feb 15, 2021
3698801
added build firle
ayush0624 Feb 15, 2021
96f7662
use numpy oriented C functions for better cython performance
shreystechtips Feb 15, 2021
42cad85
Added iou function
ayush0624 Feb 20, 2021
83a2186
Cleaned Up Combined Algorithm
KDharmarajanDev Feb 22, 2021
7d9a8a8
add gitignore and readme for cython
shreystechtips Feb 22, 2021
6e43dc7
Added Contours
KDharmarajanDev Feb 28, 2021
aed3d6b
Added Contours and Centroid Plotting
KDharmarajanDev Mar 1, 2021
a2efd7c
Implemented basics of switching
KDharmarajanDev Mar 7, 2021
9b54b5e
Fixed to being in opencv version 4
KDharmarajanDev Mar 7, 2021
a54ea21
Added slider to change if both algorithms should run
KDharmarajanDev Mar 7, 2021
ff47724
Added 0 area protections
KDharmarajanDev Mar 8, 2021
acb528e
Added sliders for area_percentage_weight and centroid_distance_weight
KDharmarajanDev Mar 26, 2021
7d0861e
Added multi-contour output
KDharmarajanDev Mar 28, 2021
5b57906
Updated to reflect debug mode
KDharmarajanDev Apr 4, 2021
300de4e
Integrated saliency detection with roulette algo
KDharmarajanDev Apr 11, 2021
c4ba9ba
saliency with the predefined array and loops
shreystechtips Apr 11, 2021
43fa064
Added dice detector
KDharmarajanDev Apr 18, 2021
894eec1
Merge branch 'background-rm' of https://github.com/berkeleyauv/percep…
KDharmarajanDev Apr 18, 2021
b7a21cc
finalize cleaner version of saliency with "chunks"
shreystechtips Apr 26, 2021
b04623c
Added Rectangle interpolator
KDharmarajanDev Jun 6, 2021
46f4f88
add prange changes
shreystechtips Jun 26, 2021
1cd369a
Updated README
KDharmarajanDev Nov 14, 2021
404e119
Fixed contours in python version
KDharmarajanDev Nov 15, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

# Byte-compiled / optimized / DLL files
__pycache__/
build/
.ipynb_checkpoints/
*.py[cod]
*$py.class
Expand All @@ -14,9 +15,11 @@ __pycache__/

# C extensions
*.so
*.o

# IDE files
.idea
.vs_code/
.vscode/

data/
15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,25 @@ activate it with

conda activate urobotics

and install all dependencies with

pip3 install -r requirements.txt

Then clone the repo in a directory of your choice

git clone https://github.com/berkeleyauv/perception.git

and install it
and change directory to the cloned directory and install all dependencies with

pip3 install -r requirements.txt

Then install the perception module with

pip3 install -e perception/

Also, our training data is stored here https://www.dropbox.com/sh/rrbfqfutrmifrxs/AAAfXxlcCtWZmUELp4wXyTIxa?dl=0 so download it and unzip it in the same folder as `perception`.

### Cython
To compile cythonized code, run the following commands after `cd`ing into the folder with Cython `setup.py` (`perception/tasks/segmentation/saliency_detection`)

python setup.py build_ext --inplace


## misc:
Misc code, camera calibration etc.
Expand Down
Binary file added algo_stats
Binary file not shown.
84 changes: 84 additions & 0 deletions misc/general_bg_rm.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
import cv2 as cv
import numpy as np
import os
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("file", help="file")
args = parser.parse_args()
from dark_channel.handler import process_frame as dark_channel



def DarkChannel(im):
b,g,r = cv.split(im)
dc = cv.min(cv.min(r,g),b);
kernel = cv.getStructuringElement(cv.MORPH_RECT,(im.shape[0],im.shape[1]))
dark = cv.erode(dc,kernel)
return dark

def resize_frame(frame,ratio = 0.4):
return cv.resize(frame,(int(frame.shape[1]*ratio),int(frame.shape[0]*ratio)))

def save_frames(frames,folder):
os.mkdir(folder)
[cv.imwrite(f'{folder}/{frame}.png', frames[frame]) for frame in frames]

def show_frames(frames):
[cv.imshow(frame,frames[frame]) for frame in frames]

def analyze(src):

# src = cv.imread(fn);
# src = resize_frame(src)
img = src
gray = cv.cvtColor(img,cv.COLOR_BGR2GRAY)
ret, thresh = cv.threshold(gray,0,255,cv.THRESH_BINARY_INV+cv.THRESH_OTSU)

# noise removal
kernel = np.ones((3,3),np.uint8)
opening = cv.morphologyEx(thresh,cv.MORPH_OPEN,kernel, iterations = 2)
# # sure background area
# sure_bg = cv.dilate(opening,kernel,iterations=3)
# # Finding sure foreground area
# dist_transform = cv.distanceTransform(opening,cv.DIST_L2,5)
# ret, sure_fg = cv.threshold(dist_transform,0.7*dist_transform.max(),255,0)
# # Finding unknown region
# sure_fg = np.uint8(sure_fg)
# unknown = cv.subtract(sure_bg,sure_fg)

# # Marker labelling
# ret, markers = cv.connectedComponents(sure_fg)
# # Add one to all labels so that sure background is not 0, but 1
# markers = markers+1
# # Now, mark the region of unknown with zero
# markers[unknown==255] = 0
# markers = cv.watershed(img,markers)
# orig = resize_frame(cv.imread(fn))
# orig[markers == -1] = [255,0,0]
return thresh
# frames = {'test':thresh,'dist':dist_transform,'sure_fg':sure_fg,'sure_bg':sure_bg,'unknown':unknown,'marked':orig}
# show_frames(frames)
# save_frames(frames,'binary_inv_dc/orig')
# cv.waitKey()


if __name__ == '__main__':
print(args.file)
cap = cv.VideoCapture(args.file)
while not cap.isOpened():
cap = cv.VideoCapture(args.file)
cv.waitKey(1000)
print ("Wait for the header")
while(True):
ret, frame = cap.read()
if ret:
frame = resize_frame(frame, 0.25)
dark = dark_channel(frame)[0]
# cv.imshow('OTSU+Dark',analyze(dark))
show_frames({'OTSU':analyze(frame),'dark':dark,'OTSU+DARK':analyze(dark),'orig':frame})
if cv.waitKey(1) & 0xFF == ord('q'):
break
if cv.waitKey(32) == ord(' '):
while(not cv.waitKey(32) == ord(' ')):
continue
cv.waitKey(-1)
13 changes: 11 additions & 2 deletions perception/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,21 @@
import perception.tasks.gate.GateSegmentationAlgoA as GateSegA
import perception.tasks.gate.GateSegmentationAlgoB as GateSegB
import perception.tasks.gate.GateSegmentationAlgoC as GateSegC
# import perception.tasks as tasks
import perception.tasks.segmentation.saliency_detection.MBD as MBD
from perception.tasks.segmentation.COMB_SAL_BG import COMB_SAL_BG
import perception.vis.TestTasks.BackgroundRemoval as BackgroundRemoval
import perception.tasks.roulette.color_detection as RouletteColorDetector
from perception.tasks.dice.DiceDetector import DiceDetector

ALGOS = {
'test': TestAlgo.TestAlgo,
'gateseg': GateSeg.GateCenterAlgo,
'gatesegA': GateSegA.GateSegmentationAlgoA,
'gatesegB': GateSegB.GateSegmentationAlgoB,
'gatesegC': GateSegC.GateSegmentationAlgoC
'gatesegC': GateSegC.GateSegmentationAlgoC,
'MBD': MBD.MBD,
'bg-rm': BackgroundRemoval.BackgroundRemoval,
'combined': COMB_SAL_BG,
'roulette': RouletteColorDetector.RouletteColorDetector,
'dice': DiceDetector
}
15 changes: 15 additions & 0 deletions perception/misc/dark_channel/dehaze.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import cv2 as cv
import numpy as np
from .haze_removal import HazeRemoval
from .utils import threshold_color_array
from ...vis.TaskPerceiver import TaskPerceiver

class Dehaze(TaskPerceiver):

def analyze(self, frame: np.ndarray, debug: bool, slider_vals: Dict[str, int]):
haze_removal_object = HazeRemoval(frame)
dark_channel = haze_removal_object.get_dark_channel(haze_removal_object.I)
A = haze_removal_object.get_atmosphere(dark_channel)
t = haze_removal_object.get_transmission(dark_channel, A)
recovered_image = haze_removal_object.get_recover_image(A, t)
return threshold_color_array(recovered_image)
17 changes: 17 additions & 0 deletions perception/misc/dark_channel/depth_map.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import cv2 as cv
import numpy as np
from .haze_removal import HazeRemoval
from .utils import threshold_color_array
from ...vis.TaskPerceiver import TaskPerceiver

class DepthMap(TaskPerceiver):

def __init__(self, beta=1):
self.beta = beta

def analyze(self, frame: np.ndarray, debug: bool, slider_vals: Dict[str, int]):
haze_removal_object = HazeRemoval(frame)
dark_channel = haze_removal_object.get_dark_channel(haze_removal_object.I)
A = haze_removal_object.get_atmosphere(dark_channel)
t = haze_removal_object.get_transmission(dark_channel, A)
return np.log(t) / -self.beta
42 changes: 42 additions & 0 deletions perception/misc/dark_channel/handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import cv2 as cv
import numpy as np
from .haze_removal import HazeRemovel
from .utils import threshold_color_array
# from combinedFilter import init_combined_filter

# cap = cv.VideoCapture('/Users/karthikdharmarajan/Documents/URobotics/Course Footage/GOPR1146.MP4')
# combined_filter = init_combined_filter()

def rescale_frame(frame, percent=75):
width = int(frame.shape[1] * percent/ 100)
height = int(frame.shape[0] * percent/ 100)
dim = (width, height)
return cv.resize(frame, dim, interpolation =cv.INTER_AREA)

def process_frame(frame):
haze_removal_object = HazeRemovel(frame)
dark_channel = haze_removal_object.get_dark_channel(haze_removal_object.I)
A = haze_removal_object.get_atmosphere(dark_channel)
t = haze_removal_object.get_transmission(dark_channel, A)
recover_image = haze_removal_object.get_recover_image(A, t)
return threshold_color_array(recover_image), t

# while cap.isOpened():
# ret, img_in = cap.read()

# if ret:
# img_in = rescale_frame(img_in,30)
# recovered_img, depth_map = process_frame(img_in)
# thresholded_img_without_haze = cv.threshold(combined_filter(recovered_img), 0, 255, cv.THRESH_BINARY | cv.THRESH_OTSU)[1]
# threshold_img_haze = cv.threshold(combined_filter(img_in), 0, 255, cv.THRESH_BINARY | cv.THRESH_OTSU)[1]
# cv.imshow('img_in', img_in)
# cv.imshow('recovered_img', recovered_img)
# cv.imshow('thresholded_img_haze', threshold_img_haze)
# cv.imshow('threshold_img_without_haze', thresholded_img_without_haze)
# cv.imshow('depth_map', depth_map)

# if cv.waitKey(1) & 0xFF == ord('q'):
# break

# cap.release()
# cv.destroyAllWindows()
60 changes: 60 additions & 0 deletions perception/misc/dark_channel/haze_removal.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# -*- coding: utf-8 -*-

import os

from PIL import Image
import cv2
import numpy as np

from .utils import number_to_integral


class HazeRemovel:

def __init__(self, image, refine=True, local_patch_size=15,
omega=0.95, percentage=0.001, tmin=0.1):
self.refine = refine
self.local_patch_size = local_patch_size
self.omega = omega
self.percentage = percentage
self.tmin = tmin
self.image = image
self.I = self.image.astype(np.float64)
self.height, self.width, _ = self.I.shape

def get_dark_channel(self, image):
min_image = image.min(axis=2)
kernel = cv2.getStructuringElement(
cv2.MORPH_RECT,
(self.local_patch_size, self.local_patch_size)
)
dark_channel = cv2.erode(min_image, kernel).astype(np.uint8)
return dark_channel

def get_atmosphere(self, dark_channel):
img_size = self.height * self.width
flat_image = self.I.reshape(img_size, 3)
flat_dark = dark_channel.ravel()
pixel_count = number_to_integral(img_size * self.percentage)
search_idx = flat_dark.argsort()[-pixel_count:]
a = np.mean(flat_image.take(search_idx, axis=0), axis=0)
return a.astype(np.uint8)

def get_transmission(self, dark_channel, A):
transmission = 1 - self.omega * \
self.get_dark_channel(self.I / A * 255.0) / 255.0
if self.refine:
transmission = self.get_refined_transmission(transmission)
return transmission

def get_refined_transmission(self, transmission):
gray = self.image.min(axis=2)
t = (transmission * 255).astype(np.uint8)
refined_transmission = cv2.ximgproc.guidedFilter(gray, t, 40, 1e-2)
return refined_transmission / 255

def get_recover_image(self, A, transmission):
t = np.maximum(transmission, self.tmin)
tiled_t = np.zeros_like(self.I)
tiled_t[:, :, 0] = tiled_t[:, :, 1] = tiled_t[:, :, 2] = t
return (self.I - A) / tiled_t + A
11 changes: 11 additions & 0 deletions perception/misc/dark_channel/utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# -*- coding: utf-8 -*-

import numpy as np


def number_to_integral(number):
return int(np.ceil(number))


def threshold_color_array(src):
return np.maximum(np.minimum(src, 255), 0).astype(np.uint8)
47 changes: 47 additions & 0 deletions perception/tasks/dice/DiceDetector.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
import cv2 as cv
import numpy as np
from perception.tasks.TaskPerceiver import TaskPerceiver
from typing import Dict
from perception.tasks.segmentation.COMB_SAL_BG import COMB_SAL_BG
import queue

class DiceDetector(TaskPerceiver):
def __init__(self, **kwargs):
super().__init__(heuristic_threshold=((5,255), 35), run_both=((0,1),0),
centroid_distance_weight=((0,200),1), area_percentage_weight=((0,200),60), num_contours=((1,5), 1))
self.sal = COMB_SAL_BG()
self.sal.set_num_contours(1)
self.sal.use_saliency = False
self.interpolator = RectangleInterpolator(20)

def analyze(self, frame: np.ndarray, debug: bool, slider_vals: Dict[str, int]):
contours = self.sal.analyze(frame, False, slider_vals)
if len(contours) > 0:
for contour in contours:
x,y,w,h = cv.boundingRect(contour)
cv.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
self.interpolator.insert_new_rectangle(x, y, w, h)
avg_coord1, avg_coord2 = self.interpolator.get_avg_coordinates()
cv.rectangle(frame,avg_coord1,avg_coord2,(255,0,255),2)
return contours, [frame]

class RectangleInterpolator():

def __init__(self, num_rectangles):
self.contour_queue = queue.Queue(maxsize=num_rectangles)
self.coord1Sum = (0, 0)
self.coord2Sum = (0, 0)

def insert_new_rectangle(self, x, y, w, h):
if self.contour_queue.full():
value = self.contour_queue.get()
self.coord1Sum = (self.coord1Sum[0] - value[0][0], self.coord1Sum[1] - value[0][1])
self.coord2Sum = (self.coord2Sum[0] - value[1][0], self.coord2Sum[1] - value[1][1])
self.contour_queue.put([(x, y), (x+w, y+h)])
self.coord1Sum = (self.coord1Sum[0] + x, self.coord1Sum[1] + y)
self.coord2Sum = (self.coord2Sum[0] + x + w, self.coord2Sum[1] + y + h)

def get_avg_coordinates(self):
avg_coord1 = (int(self.coord1Sum[0] / self.contour_queue.qsize()), int(self.coord1Sum[1] / self.contour_queue.qsize()))
avg_coord2 = (int(self.coord2Sum[0] / self.contour_queue.qsize()), int(self.coord2Sum[1] / self.contour_queue.qsize()))
return avg_coord1, avg_coord2
Loading