Skip to content

Commit c09f58c

Browse files
author
V3NU55
committed
minor edits
1 parent 6e4fc62 commit c09f58c

1 file changed

Lines changed: 11 additions & 11 deletions

File tree

index.html

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -358,7 +358,7 @@ <h2>Abstract</h2>
358358
<p>Vision-Language Models (VLMs) are increasingly proposed for autonomous driving tasks, yet their performance on sequential driving scenes remains poorly characterized, particularly regarding how input configurations affect their capabilities.</p>
359359
<p>We introduce <strong>VENUSS</strong>, a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes, establishing baselines for future research. Building upon existing datasets, VENUSS extracts temporal sequences from driving videos and generates structured evaluations across custom categories.</p>
360360
<p>By comparing 25+ existing VLMs across 2,600+ scenarios, we reveal how even top models achieve only <strong>57% accuracy</strong>, not matching human performance in similar constraints (65%) and exposing significant capability gaps. Our analysis shows that VLMs excel with static object detection but struggle with understanding the vehicle dynamics and temporal relations.</p>
361-
<p>VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations &mdash; resolution, frame count, temporal intervals, spatial layouts, and presentation modes &mdash; affect performance on sequential driving scenes.</p>
361+
<p>VENUSS offers the first systematic sensitivity analysis of VLMs focused on how input image configurations &ndash; resolution, frame count, temporal intervals, spatial layouts, and presentation modes &ndash; affect performance on sequential driving scenes.</p>
362362
</section>
363363

364364
<!-- ============================================================ -->
@@ -394,10 +394,10 @@ <h3>Four Modules</h3>
394394
<h3>Dataset-Agnostic Design</h3>
395395
<p>VENUSS is designed to be dataset-agnostic. Categories are automatically extracted from each dataset's textual descriptions. We release VENUSS with configurations for four datasets:</p>
396396
<ul>
397-
<li><strong>CoVLA</strong>: 7 <em>behavioral</em> categories (motion, direction, speed, following, acceleration, traffic lights, curvature) &mdash; extracted from natural language captions.</li>
398-
<li><strong>Honda Scenes</strong>: 16 <em>environmental</em> categories (road type, weather, surface, ambient lighting, infrastructure) &mdash; derived from categorical CSV annotations.</li>
399-
<li><strong>Waymo Open Dataset</strong>: Multi-sensor driving data with LiDAR and camera annotations &mdash; <em>coming soon</em>.</li>
400-
<li><strong>NuScenes</strong>: Multi-sensor urban driving data with 3D object annotations &mdash; <em>coming soon</em>.</li>
397+
<li><strong>CoVLA</strong>: 7 <em>behavioral</em> categories (motion, direction, speed, following, acceleration, traffic lights, curvature) &ndash; extracted from natural language captions.</li>
398+
<li><strong>Honda Scenes</strong>: 16 <em>environmental</em> categories (road type, weather, surface, ambient lighting, infrastructure) &ndash; derived from categorical CSV annotations.</li>
399+
<li><strong>Waymo Open Dataset</strong>: Multi-sensor driving data with LiDAR and camera annotations &ndash; <em>coming soon</em>.</li>
400+
<li><strong>NuScenes</strong>: Multi-sensor urban driving data with 3D object annotations &ndash; <em>coming soon</em>.</li>
401401
</ul>
402402
<p>Adding a new dataset requires modifying only 3 files: dataset configuration, annotation parser, and evaluation questions.</p>
403403
</section>
@@ -851,9 +851,9 @@ <h2>Human Evaluation</h2>
851851
<h3>Evaluation Protocol</h3>
852852
<p>The web application supports three presentation modes for human evaluators:</p>
853853
<ol>
854-
<li><strong>Image collages</strong> &mdash; matching the VLM input format for direct performance comparison</li>
855-
<li><strong>Animated GIFs</strong> &mdash; providing temporal continuity for more intuitive scenario understanding</li>
856-
<li><strong>Video playback</strong> &mdash; for annotation verification and data curation</li>
854+
<li><strong>Image collages</strong> &ndash; matching the VLM input format for direct performance comparison</li>
855+
<li><strong>Animated GIFs</strong> &ndash; providing temporal continuity for more intuitive scenario understanding</li>
856+
<li><strong>Video playback</strong> &ndash; for annotation verification and data curation</li>
857857
</ol>
858858
<p>8 evaluators participated: 3 using collages, 5 using GIFs (108 evaluations each). GIF-based evaluations consistently outperformed collage-based (65% vs 56% peak accuracy, +16% relative improvement).</p>
859859

@@ -900,7 +900,7 @@ <h2><span class="supp-badge">SUPPLEMENTARY</span> Prompt Templates</h2>
900900
</details>
901901

902902
<details>
903-
<summary>User Prompt &mdash; CoVLA Dataset</summary>
903+
<summary>User Prompt &ndash; CoVLA Dataset</summary>
904904
<div class="details-content">
905905
<p>The user prompt provides context about the image content and task requirements. For each evaluation, models receive:</p>
906906
<p class="code-label">User Message</p>
@@ -939,7 +939,7 @@ <h2><span class="supp-badge">SUPPLEMENTARY</span> Prompt Templates</h2>
939939
</details>
940940

941941
<details>
942-
<summary>User Prompt &mdash; Honda Scenes Dataset</summary>
942+
<summary>User Prompt &ndash; Honda Scenes Dataset</summary>
943943
<div class="details-content">
944944
<p>Honda Scenes uses 16 environmental categories instead of CoVLA's 7 behavioral categories, covering road type, weather, surface conditions, ambient lighting, and various infrastructure elements.</p>
945945
<p class="code-label">User Message</p>
@@ -1094,7 +1094,7 @@ <h2>Citation</h2>
10941094
</div><!-- /container -->
10951095

10961096
<footer>
1097-
VENUSS &mdash; VLM Evaluation oN Understanding Sequential Scenes
1097+
VENUSS &ndash; VLM Evaluation oN Understanding Sequential Scenes
10981098
</footer>
10991099

11001100
</body>

0 commit comments

Comments
 (0)