RFC: repo is confusing for end-user

[victoredwardocallaghan]

Hi,

This is more feedback than anything else.

This repo is rather confusing way about working. Why can't AMD just work just directly and exclusively upstream into mainline in a incremental way rather than do these source drops here and mainline components also? It leaves the user confused, 'should I use mainline amdkfd or should I use this repo?'

If AMD could adopt the same practice as amdgpu by working directly into mainline that would be more optimal and less confusing. Any out of tree pending parts tend to be in the form of easy to rebase pull requests should someone want to be on the cutting edge..

I hope this feedback is useful.
Kind Regards.

[johnbridgman]

Thanks for raising this. Ideally we would be working only in the upstream tree, but we currently have some code (pinning of GPUVM-mapped system and local memory from userspace) which would not be acceptable upstream. We are working on upstream-acceptable versions of that code, based on an RFC we sent out a few months ago; once that work is done we should be able to resume alignment with upstream.

[victoredwardocallaghan]

Hi John,

I've been meaning to email you! Are you referring to getting Tonga support in mainline, however there is a issue with the use of pageable memory that requires iommuv2 modification so that it is available with dGPUs like it is for APUs?

My suggestion would be for AMD to be somewhat more fine grain in it's approach. Rather than one random day there is this huge source drop. I realize AMD has changed a lot internally in regards to amdgpu/amdkfd and you are trying to get the groundwork in place properly projecting forwards but it is hard to give early feedback for folks who would like to really get involved.

btw, thanks for getting powerplay out that's 1/2 of my blocking big ticket items for my needs resolved. I really hope the iommu issue is resolved in time for the 4.5 merge window, that would be killer for me as I could release my stack at the same time.

Kind Regards,
Edward.

[johnbridgman]

Hi Edward,

The biggest issue is actually device-local memory, which is not pageable and not directly helped by IOMMUv2. First case was Kaveri, where we don't have actual device memory but have a fast non-coherent path to carved-out system memory via GPUVM (the "Garlic" bus) which is ~2x as fast as coherent access (the "Onion" bus) and useful in certain cases.

On Carrizo the coherent path is much faster, close enough to non-coherent that we have not bothered adding "device" memory support at all. Once we get to dGPUs, however, the bandwidth difference between system memory and device memory comes back, particularly on Fiji with HBM.

With dGPUs we can access system memory via IOMMUv2 when running on platforms that have it, but we still have a lot of target platforms which do not provide ATS/PRI support to the PCIE bus so need to use GPUVM (and, currently, pinning from userspace) there as well.

Anyways, in the short term we plan to make more frequent releases; not as good as working directly upstream but should at least be a noticeable improvement.

Thanks,
John

[johnbridgman]

BTW I assumed you were talking about dGPU support (eg Tonga, Fiji) rather than APU (eg Kaveri, Carrizo) because of your reference to Tonga. We do have some APU fixes accumulated that I need to get upstream as soon as time permits. You mentioned "releasing your stack", feel free to email me (first.last@amd.com) if you want to coordinate in more detail.

[victoredwardocallaghan]

Time to kill this repo John?

[johnbridgman]

Don't think we can necessarily kill it (if it's the latest HSA-compliant stack we need to keep it for sure) but at minimum we should be updating the README to make sure everyone knows there is newer code over in the ROC repos. Will discuss on Monday, thanks for flagging this.

---

From: victoredwardocallaghan notifications@github.com
Sent: March 27, 2016 1:43 AM
To: HSAFoundation/HSA-Drivers-Linux-AMD
Cc: Bridgman, John
Subject: Re: [HSA-Drivers-Linux-AMD] RFC: repo is confusing for end-user (#16)

Time to kill this repo John?

You are receiving this because you commented.
Reply to this email directly or view it on GitHubhttps://github.com//issues/16#issuecomment-201994770

[victoredwardocallaghan]

John,

From the customers prospective this stuff is a giant unnavigable mess, just if this isn`t clear already.

You/AMD/Whoever needs to lay it out; Want compute on Radeon this is the driver you install for the moment.. and here is (in dated bullet points preferably) how things are going to look going forward with our upstream effort.

Right now, how things stand today and for the last however long, compute on Radeon is a giant mangled mess. If I were to go over to the competitor I can a nice landing page saying here is our one driver and toolchain CUDA and your setup in 15min flat. This works perfect out the box for years now (please dont say Radeons has because it has not, its frankly a joke). My strong advice is delete all the random bits of half finished stuff floating around everywhere and provide a clear message to the customer because right now this is all a joke.

I know this sounds harsh, and you know full well I am not a troll as I contrib directly to your driver effort. But this stuff needs to be fixed like yesterday because right now I have some compute projects I am about to do and I am forced to go buy 6 top end Nvidia cards as I have zero choice, its just ridiculous!

Kind Regards,

[gstoner]

Victor,
First I want let you know I hear your frustration. I recently stepped in as the Head of Radeon Open Compute for AMD looking to clean up lot of the issue you raise. Note I am also the Chairmen and Managing Director of the HSA Foundation. I need first clear something up since the current exchange, which show where some of the confusion is starting, since this dialog is happing on HSA Foundation managed Github site which is not run by AMD but separate company the HSA Foundation which is indecent corporation. HSA Foundation made it convince to our member companies to host open source software repo related to HSA software technologies. Currently the software on this site is mostly pre-production software. The vendors would develop production supported driver runtime packages and release them on there driver and develop portals site

On your question can you have Single Portal for AMD Official Packaged and supported Driver for our GPU’s and APU and a single portal to get Developer Information. You see it matches up how your get software for NVIDIA Driver + SDK. For these driver there is single programing model today just as NVIIDA has today it is OpenCL.

Company Developer Site Driver Site
AMD http://developer.amd.com http://support.amd.com/en-us/download
NVIDIA http://developer.nvidia.comhttp://developer.nvidia.com/ http://www.nvidia.com/Download/index.aspx?lang=en-us

Just like NVIDIA these driver are proprietary binary driver for Windows and Linux. which now open second discussion. If your looking for pure open source driver this were we are now working on new solution that build on the past work we did with HSA, honestly historical Radeon Open Source driver and Nouveau has very limited compute support. This is some thing we are going to address so the open source driver have world class compute capability and support.

HSA technology has been in pre-production engineering release for last two years so we could closer with number of partners. One thing that come out of these interaction with the partners and community strong desire for AMD to support HSA beyond the APU and also bring in dGPU support. So on November 16th 2015 we announced at Supercomputing 2015 the Boltzmann Initiative initially focused on Headless Linux computing. This is currently PRE-RELEASE software the stack is branded ROCm.

Moving forward for the our open source based projects we have single portal, GPUopen, http://gpuopen.com/professional-compute/ . Here you find the driver, compiler, libraries, and applications supporting our GPU Compute solutions. Also here you find instruction on how to use the the software. This portal will take you to the proper GitHub Repo if you want access to source. This portal will be where we talk about all thing related to ROCm Platform and application that leverage it. We have AMD and Guest BLOG here.

The current Pre-Release ROCm project today only support the Carizzo family of APU and our Fiji Family of discreet GPU today. This build off the HSA runtime and driver technology and will be the Commercial Foundation AMD bring forward supporting HSA Technology. Note HSA has very specialized HW capability needs in GPU hardware, CPU and BUS IO, which is why we are restricting which GPU we support today. On thing people forget is HSA 1.0 Specification we not release until March 16th 2015 and currently The Carrizo Family of products is the first device to be HSA 1.0 Compliant.

To help you understand all that we are working in pre-production here is overview that we are getting ready for ROCm 1.0. We made the first public release on Jan 26th as Developer Preview, 10 day ago we just release the Alpha for version 1.0. We will be moving to cleaner distribution model for those who do not want to start from source we will have ROCm Platform package.

• Compiler/Language Runtime
• HCC Compiler and Language Runtime https://gpuopen.com/compute-product/hcc-heterogeneous-compute-compilerhttps://gpuopen.com/compute-product/hcc-heterogeneous-compute-compiler/%3Ewith
  - Single Source Compiler supports
  - C++ 11/14/17 Support with Parallel STL
• OpenMP 3.1 and 4.0 C and C++ support for CPU, We are developing GPU offload support with OpenMP 4.5
• HCC is built on CLANG and LLVM compiler foundation with Native GCN ISA code generation
• Compiler supports GCN ISA Assembler and Disassembler
• Static Offline compiler which is separate from the driver so we can be more aggressive with optimizations
• HIP Runtime https://gpuopen.com/compute-product/hip-convert-cuda%E2%80%A6-portable-c-code/ and HIPIFY which are tool we can use to port CUDA application over our hardware more easily.
• Continuum IO Anaconda with NUMBA support for Accelerated Python
• Math Libraries
  - HcBLAS https://bitbucket.org/multicoreware/hcblas
  - HcFFT https://bitbucket.org/multicoreware/hcfft
  - HcSPARSE https://bitbucket.org/multicoreware/hcsparse
  - HcRNG https://bitbucket.org/multicoreware/hcrng
• ROCR Runtime ( which based on the HSA runtime API http://www.hsafoundation.com/html/HSA_Library.htm enhanced for the need for dGPU’s) for developing tools and getting the lowest level access to the hardware you need it.
• User Mode Queues - This bring HyperQ like functionality
• User Mode DMA
• Advanced Memory Management API supporting need of Multi-GPU systems
• Coarse-grain Shared Virtual Memory
• Peer to Peer Multi-GPU supports with Large Bar support
• GPU enabled Peer to Peer with RDMA support for Mellanox and Chelsio Network cards
• Process Concurrency & Preemption
• Debugger Runtime Foundation
• Profiler Trace and Event Collection API
• HSA Signals and Atomics (Using PCIe Gen3 Platform Atomics) for System level signaling and synchronization
• Standardized loader and Code Object Format for Binary GCN ISA ( ELF & DWARF support)
• The stack is Open Source based leveraging MIT and UIUC license.
• We are also working with Kokkos and Charm++ with the HCC compiler.

To run this today we need:

• Haswell CPU or newer CPU, like Xeon E5 v3 processor which has PCIe Platform Atomics support.
• Fiji GPU R9 Fury Nano, R9 Fury X, R9 Fury
• Today operates with Ubuntu 14.04 or Fedora 23, we are working with other distributions i.e. Redhat/Centos

https://github.com/RadeonOpenCompute/ROCR-Runtime/tree/devhttps://github.com/RadeonOpenCompute/ROCR-Runtime
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/tree/dev

We are in transition from our past solution of proprietary drivers and single GPU programing model to new foundation which I ask you please have patience as we work to make the foundation broadly available.

greg

On Mar 27, 2016, at 11:52 PM, victoredwardocallaghan <notifications@github.commailto:notifications@github.com> wrote:

John,

From the customers prospective this stuff is a giant unnavigable mess, just if this isn`t clear already.

You/AMD/Whoever needs to lay it out; Want compute on Radeon this is the driver you install for the moment.. and here is (in dated bullet points preferably) how things are going to look going forward with our upstream effort.

Right now, how things stand today and for the last however long, compute on Radeon is a giant mangled mess. If I were to go over to the competitor I can a nice landing page saying here is our one driver and toolchain CUDA and your setup in 15min flat. This works perfect out the box for years now (please dont say Radeons has because it has not, its frankly a joke). My strong advice is delete all the random bits of half finished stuff floating around everywhere and provide a clear message to the customer because right now this is all a joke.

I know this sounds harsh, and you know full well I am not a troll as I contrib directly to your driver effort. But this stuff needs to be fixed like yesterday because right now I have some compute projects I am about to do and I am forced to go buy 6 top end Nvidia cards as I have zero choice, its just ridiculous!

Kind Regards,

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/16#issuecomment-202230288