Provider extractive · golden dataset of 14 cases
(12 answerable, 2 expected to be refused).
No regression: 0 metric(s) improved, 9 held steady. Tolerance: 0.03 absolute.
| Metric | Score | Threshold | Verdict |
|---|---|---|---|
answer_relevancy |
0.6426 | ≥ 0.4 | pass |
citation_validity |
1.0000 | ≥ 1.0 | pass |
context_precision |
0.7639 | ≥ 0.45 | pass |
context_recall |
0.7073 | ≥ 0.45 | pass |
faithfulness |
1.0000 | ≥ 0.85 | pass |
hallucination_rate |
0.0000 | ≤ 0.1 | pass |
latency_compliance |
1.0000 | ≥ 1.0 | pass |
refusal_correctness |
1.0000 | ≥ 1.0 | pass |
toxicity |
0.0000 | ≤ 0.02 | pass |
| Question & answer | Scores | Latency | Verdict |
|---|---|---|---|
|
How long does the Aurora R2 battery last on a full charge?
The Aurora R2 battery lasts 14 hours on a full charge.
expected: The battery runs for 14 hours once it is fully charged.
cited: battery-care
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.43context_recall 0.50context_precision 0.50hallucination_rate 0.00 | 0.41 ms | pass |
|
Which dock should I use to charge the Aurora R2?
Charge the Aurora R2 with the 65 watt dock supplied in the box.
expected: Use the 65 watt dock that ships in the box.
cited: battery-care
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.58context_recall 0.67context_precision 0.50hallucination_rate 0.00 | 0.35 ms | pass |
|
How long is the warranty on the R2?
Aurora Robotics covers the R2 with a 24 month warranty.
expected: The R2 comes with a warranty of 24 months.
cited: warranty
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.68context_recall 0.80context_precision 1.00hallucination_rate 0.00 | 0.29 ms | pass |
|
Does the warranty cover water damage?
The warranty does not cover water damage.
expected: Water damage is excluded from the warranty.
cited: warranty
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.67context_recall 0.75context_precision 1.00hallucination_rate 0.00 | 0.18 ms | pass |
|
What does error E14 mean?
Error E14 means the lidar sensor is blocked.
expected: E14 indicates a blocked lidar sensor.
cited: error-codes
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.73context_recall 0.80context_precision 0.50hallucination_rate 0.00 | 0.26 ms | pass |
|
How do I clear error E14?
Clear error E14 by wiping the sensor window with a dry cloth.
expected: Wipe the sensor window with a dry cloth to clear E14.
cited: error-codes
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.76context_recall 0.86context_precision 1.00hallucination_rate 0.00 | 0.18 ms | pass |
|
How often are firmware updates delivered over wifi?
Firmware updates are delivered over wifi every quarter.
expected: Updates arrive over wifi once per quarter.
cited: firmware-updates
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.55context_recall 0.60context_precision 1.00hallucination_rate 0.00 | 0.19 ms | pass |
|
How long does a firmware update take to install?
A firmware update takes about 12 minutes to install.
expected: Installing an update takes roughly 12 minutes.
cited: firmware-updates
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.67context_recall 0.67context_precision 0.50hallucination_rate 0.00 | 0.32 ms | pass |
|
When does support answer during the week?
Support answers monday to friday from 9am to 6pm central european time.
expected: Support is open monday to friday, 9am to 6pm central european time.
cited: support-hours
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.89context_recall 0.89context_precision 0.50hallucination_rate 0.00 | 0.25 ms | pass |
|
How many days do I have to return the robot after delivery?
Returns are accepted within 30 days of delivery.
expected: You may return the robot within 30 days of delivery.
cited: returns
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.73context_recall 0.83context_precision 1.00hallucination_rate 0.00 | 0.31 ms | pass |
|
Is map data uploaded to the cloud?
Map data is never uploaded to the cloud without explicit consent.
expected: No, map data stays on the device unless you give explicit consent.
cited: privacy
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.53context_recall 0.62context_precision 1.00hallucination_rate 0.00 | 0.18 ms | pass |
|
How many countries in Europe does Aurora Robotics ship to?
Aurora Robotics ships to 14 countries in Europe.
expected: The company delivers to 14 countries across Europe.
cited: shipping
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00faithfulness 1.00answer_relevancy 0.50context_recall 0.50context_precision 0.67hallucination_rate 0.00 | 0.42 ms | pass |
|
What is the retail price of the Aurora R5 in Brazil?
I don't have information about that in the knowledge base.
expected: I don't have information about that in the knowledge base.
abstained
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00 | 0.16 ms | pass |
|
Can the Aurora R2 operate on the surface of Mars?
I don't have information about that in the knowledge base.
expected: I don't have information about that in the knowledge base.
abstained
|
citation_validity 1.00refusal_correctness 1.00toxicity 0.00latency_compliance 1.00 | 0.14 ms | pass |