Skip to content

gvagenai.md

Latest commit

 

History

History
155 lines (134 loc) · 7.8 KB

gvagenai.md

File metadata and controls

155 lines (134 loc) · 7.8 KB

gvagenai

Performs inference with Vision Language Models using OpenVINO™ GenAI, accepts video and text prompt as an input, and outputs text description. It can be used to generate text summarization from video.

Configuration

Generation Config

The generation-config property accepts config parameters in KEY=VALUE,KEY=VALUE format. For detailed information about these parameters, refer to the OpenVINO™ GenAI GenerationConfig documentation.

Available generation-config keys:

Generation Config Parameters

Key Format
max_new_tokens Integer
max_length Integer
ignore_eos Boolean
min_new_tokens Integer
eos_token_id Integer
stop_strings String (semicolon-separated), example: STOP;END;DONE
include_stop_str_in_output Boolean
stop_token_ids Integer (semicolon-separated), example: 1;2;3
repetition_penalty Float
presence_penalty Float
frequency_penalty Float
num_beams Integer
num_beam_groups Integer
diversity_penalty Float
length_penalty Float
num_return_sequences Integer
no_repeat_ngram_size Integer
stop_criteria String, StopCriteria: EARLY, HEURISTIC, or NEVER
do_sample Boolean
temperature Float
top_p Float
top_k Integer
rng_seed Integer
assistant_confidence_threshold Float
num_assistant_tokens Integer
max_ngram_size Integer
apply_chat_template Boolean

Example:

generation-config="max_new_tokens=100,temperature=0.7,do_sample=true"

Scheduler Config

The scheduler-config property accepts config parameters in KEY=VALUE,KEY=VALUE format. For detailed information about these parameters, refer to the OpenVINO™ GenAI SchedulerConfig documentation.

Available scheduler-config keys:

Scheduler Config Parameters

Key Format
max_num_batched_tokens Integer
num_kv_blocks Integer
cache_size Integer
dynamic_split_fuse Boolean
use_cache_eviction Boolean
max_num_seqs Integer
enable_prefix_caching Boolean
cache_eviction_start_size Integer
cache_eviction_recent_size Integer
cache_eviction_max_cache_size Integer
cache_eviction_aggregation_mode String, AggregationMode: SUM or NORM_SUM
cache_eviction_apply_rotation Boolean
cache_eviction_snapkv_window_size Integer

Example:

scheduler-config="max_num_batched_tokens=256,cache_size=10,use_cache_eviction=true"

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)RGB, (string)RGBA, (string)RGBx, (string)BGR, (string)BGRA, (string)BGRx, (string)NV12, (string)I420 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)RGB, (string)RGBA, (string)RGBx, (string)BGR, (string)BGRA, (string)BGRx, (string)NV12, (string)I420 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  chunk-size          : Number of frames in one inference
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4294967295 Default: 1
  device              : Device to use (CPU, GPU, NPU, etc.)
                        flags: readable, writable
                        String. Default: "CPU"
  frame-rate          : Number of frames sampled per second for inference (0 = process all frames)
                        flags: readable, writable
                        Double. Range:               0 -   1.797693e+308 Default:               0
  generation-config   : Generation configuration as KEY=VALUE,KEY=VALUE format
                        flags: readable, writable
                        String. Default: null
  metrics             : Include performance metrics in JSON output
                        flags: readable, writable
                        Boolean. Default: false
  model-cache-path    : Path for caching compiled models (GPU only)
                        flags: readable, writable
                        String. Default: "ov_cache"
  model-path          : Path to the GenAI model
                        flags: readable, writable
                        String. Default: null
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "gvagenai0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  prompt              : Text prompt for the GenAI model
                        flags: readable, writable
                        String. Default: null
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  scheduler-config    : Scheduler configuration as KEY=VALUE,KEY=VALUE format
                        flags: readable, writable
                        String. Default: null