Skip to content

Error after updating to convolve the entire 3D array #9

@shriram-jagan

Description

@shriram-jagan

3D convolutions at large resolutions use cuFFT internally and gives the following error. Reverting this PR fixes it since in that PR we loop over each variable and convolve instead of using convolutions for the entire 3D array. Creating this issue so that I'll remember to look into this in a couple of weeks.

Start: total_mass, total_energy: 152577474.45441937, 28337000790599.57
Signal 8 received by node 0, process 3158851 (thread 7fd8e8a20000) - obtaining backtrace
Signal 8 received by process 3158851 (thread 7fd8e8a20000) at: stack trace: 16 frames
  [0] = void cunumeric::cufft_convolution<double, 3>(Legion::FieldAccessor<(legion_privilege_mode_t)268435463, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Legion::FieldAccessor<(legion_privilege_mode_t)1, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Legion::FieldAccessor<(legion_privilege_mode_t)1, double, 3, long long, Realm::AffineAccessor<double, 3, long long>, false>, Realm::Rect<3, long long> const&, Realm::Rect<3, long long> const&, Realm::Rect<3, long long> const&) at unknown file:0 [00007fd8945c5324]
  [1] = void cunumeric::ConvolveImpl<(cunumeric::VariantKind)2>::operator()<(legate::Type::Code)11, 3, (void*)0>(cunumeric::ConvolveArgs&) const at unknown file:0 [00007fd8945c5b66]
  [2] = void cunumeric::convolve_template<(cunumeric::VariantKind)2>(legate::TaskContext&) at unknown file:0 [00007fd894565dba]
  [3] = cunumeric::ConvolveTask::gpu_variant(legate::TaskContext) at unknown file:0 [00007fd894565fd6]
  [4] = legate::detail::task_wrapper(void (*)(legate::TaskContext), legate_core_variant_t, std::optional<std::basic_string_view<char, std::char_traits<char> > >, void const*, unsigned long, void const*, unsigned long, Realm::Processor) at unknown file:0 [00007ffdcc9d77fb]
  [5] = void legate::LegateTask<cunumeric::ConvolveTask>::task_wrapper_<&cunumeric::ConvolveTask::gpu_variant, (legate_core_variant_t)2>(void const*, unsigned long, void const*, unsigned long, Realm::Processor) at unknown file:0 [00007fd8927b6d1b]
  [6] = Realm::Cuda::GPUProcessor::execute_task(unsigned int, Realm::ByteArrayRef const&) at unknown file:0 [00007ffff53d8a0d]
  [7] = Realm::Task::execute_on_processor(Realm::Processor) at unknown file:0 [00007ffff53c5872]
  [8] = Realm::KernelThreadTaskScheduler::execute_task(Realm::Task*) at unknown file:0 [00007ffff53c58f9]
  [9] = Realm::Cuda::GPUTaskScheduler<Realm::KernelThreadTaskScheduler>::execute_task(Realm::Task*) at unknown file:0 [00007ffff541e2d7]
  [10] = Realm::ThreadedTaskScheduler::scheduler_loop() at unknown file:0 [00007ffff53c3ec3]
  [11] = Realm::ThreadedTaskScheduler::scheduler_loop_wlock() at unknown file:0 [00007ffff53c4430]
  [12] = Realm::KernelThread::pthread_entry(void*) at unknown file:0 [00007ffff53c8325]
  [13] = start_thread at unknown file:0 [00007ffff2999608]
  [14] = __clone at unknown file:0 [00007ffff4b5a132]
  [15] = unknown symbol at unknown file:0 [ffffffffffffffff]

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions