# Experiment States

## Experiment tracking

The status of experiments will be one of the following:

* **`enqueued`**: The experiment is in the queue to run. Experiments that are `enqueued` will run in priority over experiments that are `paused`. Experiments that are `enqueued` with `compute_mode="cycle"` will run in priority over experiments that  are `enqueued` with `compute_mode="interruptible"`.
* **`interrupting`**: An experiment with `compute_mode="cycle"` has been enqueued and your experiment with `compute_mode="interruptible"` is being interrupted. Your experiment will be returned to the queue and resume when there are no more experiment with `compute_mode="cycle"` to be run.
* **`paused`**: The experiment has completed its cycle or been interrupted and is now waiting in the queue.
* **`running`**: The experiment is currently running on the ISC. From this state, you can check the logs and can see them updating in real time.
* **`cancelling`**/**`cancelled`**: The experiment has been cancelled using the web UI or by running `isc cancel <experiment_id>`. The experiment will stop if it was `running`, and will be removed from the queue.
* **`failed`**: The experiment has encountered an exception during runtime and has been stopped and removed from the queue.
* **`completed`**: The experiment has completed and exited without an error.
* **`strong_failed`**: In rare cases, the experiment may fail due to errors occurring in Strong Compute systems or due to hardware failure. If your experiment status is `strong_failed` this is our fault, not yours. Be assured we will be investigating what has gone wrong. You can try resubmitting the experiment and resume from your last checkpoint.

#### Burst  statuses

For experiments with `compute_mode="burst"` the following additional experiment statuses will be displayed before and during experiment initialisation.

* **`burst_staged`**: The experiment has been staged with `isc train <experiment-launch-file>` but still requires confirmation to launch. In this state, the experiment has incurred no cost to the authorising Organisation. Users must visit the Experiments page on Control Plane and click **Launch Burst** to start the experiment.
* **`burst_allocating`**: The experiment has been launched and the ISC is in the process of finding and provisioning a suitable cluster on a commercial cloud provider to run the experiment.
* **`burst_starting`**: A suitable cluster has been found and provisioned for the experiment and the ISC is in the process of downloading all necessary data for the experiment to the cluster including the User's container data and any datasets flagged for use in training.

Experiments are considered "active" if they are "**enqueued**", "**interrupting**", "**running**", or "**paused**".

You can view the most recent logs generated on node Rank 0 of your cluster by clicking the **Last Logs** button for your experiment on the Experiments page. In the modal that appears you will see the last 100 lines from the `rank_0.txt` file in your experiment outputs (refer to Section 3.8 of the [Hello World example](https://docs.strongcompute.com/getting-started/2.-hello-world-training-example#id-3.8-inspect-experiment-outputs-in-control-plane)).

<br>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.strongcompute.com/basic-concepts/experiment-states.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.