NexBench Evaluation Process

1. Process overview

The remote-evaluation lifecycle consists of six standard stages, executed in order:

  1. 01SDK deployment and environment setupInstall the evaluation SDK and client scripts
  2. 02Inference service startupDeploy the model inference service and keep it reachable
  3. 03Evaluation requestRegister and sign in, create an evaluation token, submit the request
  4. 04Evaluation approvalContact the approver for review and agree on a time window
  5. 05Remote client runConfigure credentials and execute the remote evaluation script
  6. 06Result retrievalReview the evaluation report produced by the system
StageNameKey actionDeliverable
01SDK deployment and environment setupInstall the evaluation SDK and client scriptsA runnable evaluation client environment
02Inference service startupDeploy the model inference service and keep it reachableOnline inference service endpoint
03Evaluation requestRegister and sign in, create an evaluation token, submit the requestEvaluation task ID (task-id)
04Evaluation approvalContact the approver for review and agree on a time windowAn approved evaluation task
05Remote client runConfigure credentials and execute the remote evaluation scriptEvaluation logs and raw data
06Result retrievalReview the evaluation report produced by the systemEvaluation report (manual verdict and automated metrics)

2. Stage details

01

SDK deployment and environment setup

GoalSet up the local environment for running the evaluation client.

Steps

  1. Obtain and unzip the evaluation SDK package nexbenchremotesdk.zip.
  2. Deploy two client scripts to the working directory: client_touch_nex_luna.sh (remote evaluation entry script, Shell) and transport_client_touch_nex.py (transport-layer client, Python).
  3. Confirm the script execution permissions and the Python runtime dependencies.

Delivery criteriaThe scripts run normally with no missing-dependency errors.

02

Inference service startup

GoalDeploy the model inference service to be evaluated so the remote evaluation platform can reach it.

Steps

  1. Deploy the target model and its inference service (server).
  2. Confirm the listening port and network reachability.
  3. Verify that the inference endpoint responds correctly.

Delivery criteriaThe inference service is online and its endpoint responds normally.

03

Evaluation request

GoalCreate and submit an evaluation task in the evaluation system.

Steps

  1. Register and sign in: create an account in the evaluation system and sign in.
  2. Create an evaluation token: generate the token used for authentication.
  3. Submit the request: fill in the evaluation configuration and submit it to obtain the task ID (task-id).

Key credentials

CredentialDescriptionExample
usernameEvaluation system account[evaluation account]
passwordEvaluation system password[evaluation password]
task-idUnique evaluation task identifier[task-id]

All example values are placeholders. The real task-id is issued by the technical team after the evaluation task is created in the system. Never write account credentials in plain text into documents or scripts — manage them with environment variables or secure configuration.

04

Evaluation approval

GoalObtain authorization to execute the evaluation task.

Steps

  1. Contact the approval owner and submit the evaluation approval request.
  2. Confirm and agree on the execution time window with the approver.
  3. Wait for the approval.

Approval contact张子杰

An evaluation must be approved with the approver's help; a task that has not been approved cannot enter the execution stage.

05

Remote client run

GoalRun the remote evaluation client to trigger the evaluation task.

bash client_touch_nex_luna.sh \
  --username [evaluation account] \
  --password [evaluation password] \
  --task-id [task-id]

Parameters

ParameterRequiredDescription
--usernameYesEvaluation system account
--passwordYesEvaluation system password
--task-idYesEvaluation task ID obtained in stage 3

Execution logic

  1. The script authenticates against the evaluation platform with the provided credentials.
  2. Once authenticated, it triggers the remote evaluation task bound to the given task-id.
  3. The evaluation runs server-side; the local client handles communication and data transfer.
  4. transport_client_touch_nex.py acts as the underlying transport client and exchanges data with the inference service and the evaluation platform.

Input / Output

Input
Evaluation credentials (username/password), task ID (task-id), inference service endpoint.
Output
Evaluation status, runtime logs, and raw evaluation data.
06

Result retrieval

GoalRetrieve and interpret the evaluation report.

Where results appearOn the evaluation system platform.

Report structure

  • Evaluation report
  • Manual verdict
  • Success / Failure (binary)
  • Automated metrics
  • Task completion
  • Open instruction following
  • Planning
  • Precision
  • Fluency
  • Safety
  • Overall score

3. Evaluation dimensions

3.1 Manual evaluation

DimensionJudgementValue
Task success verdictHuman reviewSuccess / Failure

The manual verdict is the final authority and is used to review and calibrate the automated results.

3.2 Automated metrics

Six core metrics plus one composite score (Overall):

No.MetricAutomation sourceRole in the result
1Task completionSemantic judgement + target / container final-state verificationDescribes how fully the target final state is reached
2Open instruction followingVideo semantic modelChecks whether targets, attributes and actions match the instruction
3PlanningBehaviour sequence analysisEvaluates whether task decomposition and execution order are reasonable
4PrecisionSpatial pose analysisMeasures how precise the manipulation is
5FluencyMotion trajectory analysisEvaluates the continuity and smoothness of motion
6SafetyAnomalous behaviour detectionDetects safety risks during the operation
—OverallWeighted fusion of metricsComposite score; see the related USTC documentation for the formula

For detailed metric definitions and the Overall formula, see the VLA Real-Robot Video Automated Evaluation Review and Explainability Evidence Report and the related USTC technical documentation.

4. Contacts

NameResponsibility
张子杰Evaluation request and approval support
彭华彬Documentation maintenance and technical support

5. Attachments and resources

FileTypePurpose
nexbenchremotesdk.zipSDK archiveMain remote-evaluation SDK
client_touch_nex_luna.shShell scriptRemote evaluation entry point
transport_client_touch_nex.pyPython scriptTransport-layer client for data exchange

6. Quick-start checklist

Confirm every item below before running a remote evaluation:

  • The evaluation SDK is deployed and the client scripts are executable
  • The inference service is running and reachable on the network
  • The evaluation system account is registered and can sign in
  • The evaluation token has been created
  • The evaluation task has been submitted and the task-id obtained
  • The evaluation task has been approved (contact 张子杰)
  • The evaluation time window has been confirmed
  • The remote client command parameters are fully configured