Skip to content

HyperLogLog example using threads #23

Description

@SaiCharanMarrivada

Hi, Thank you for such wonderful library. I have one simple example using HyperLogLog with multiple threads.

using Probably
using Base.Threads

function example(n::Int) 
    lock_ = ReentrantLock()
    hll = HyperLogLog()
    nthreads = Threads.nthreads()
    Threads.@threads for id in 1:nthreads
        temp_hll = HyperLogLog()
        for i in id:nthreads:n
            push!(temp_hll, i)
        end
        lock(lock_) do
            union!(hll, temp_hll)
        end
    end
    return length(hll)
end

@time @show example(Int(1e10))

This code takes 14.34s with 1 thread, 3.73s with 16 threads on my intel ™ 15-1240P machine

JULIA_NUM_THREADS=16 julia hll_parallel.jl 
# output
example(Int(1.0e10)) = 10022247613
  3.731444 seconds (35.67 k allocations: 2.804 MiB, 24.87% compilation time)
JULIA_NUM_THREADS=1 julia hll_parallel.jl 
# output
example(Int(1.0e10)) = 10022247613
 14.344354 seconds (35.21 k allocations: 2.542 MiB, 0.30% compilation time)

Is it possible to add this example in documentation?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions