Create 2022-07-26-phone-number-capture-dataset.md#108
Create 2022-07-26-phone-number-capture-dataset.md#108Anirudh257 wants to merge 1 commit intoskit-ai:masterfrom
Conversation
|
|
||
| # Introduction | ||
|
|
||
| A bottleneck of a dialogue system is its ability to extract information from the utterances. The information is extracted in the form of **frames**, that represents all the different types of intentions that the system can extract from user utterances and **slots**, that are the different type of possible values. |
There was a problem hiding this comment.
is frames the same as intents ? lets use that to be consistent
|
|
||
| ## Sentence Error Rate | ||
|
|
||
| Sentence Error Rate is a robust extension to WER(Word Error Metric), used to evaluate the working of an ASR system. |
There was a problem hiding this comment.
Need more on how this is defined
| 1. **System performance** - this is captured through entity and slot metrics. in the case of alphanumeric here, we focus on SER (Sentence Error Rate) | ||
| 2. **User experience** - this is captured through a subjective UX score, assigned through analysis (by the CUX function) | ||
|
|
||
| We expect, system performance (≈SER) to be better for entities captured across two turns - since we are parsing smaller sub-entities independently and (naively) expect SER to be a function of CER (Character Error Rate) and length. |
There was a problem hiding this comment.
If we keep this note, then it needs more explanation. What is SER and why is it a function of these two ? seems like some assumptions are being skipped
|
|
||
| Sentence Error Rate is a robust extension to WER(Word Error Metric), used to evaluate the working of an ASR system. | ||
|
|
||
| We averaged the calculations across different callers, for each variation across different ASR systems. |
There was a problem hiding this comment.
better sentence construction
| We averaged the calculations across different callers, for each variation across different ASR systems. | ||
|  | ||
|
|
||
| ## UX Score |
There was a problem hiding this comment.
This section needs a lot more! Can start with a summary of the UX report
| > Single Turn is a more natural way to collect a phone number than Two turns. | ||
|
|
||
| # Future Work | ||
| ## Validate on a larger dataset |
There was a problem hiding this comment.
lets skip this, and add confidence intervals instead
| For example, for the above conversation to book a flight, the frame will be of the type: | ||
|
|
||
|  | ||
| # Motivation |
There was a problem hiding this comment.
Can this be merged with the previous section ? Introduction isnt saying much really
| categories: [Machine Learning] | ||
| image: assets/images/demo1.jpg | ||
| layout: post | ||
| authors: [anirudhthatipelli] |
There was a problem hiding this comment.
adding adithyanarayan as a co-author since he worked on all the evaluations here
2 participants
No description provided.