The HPC cluster

Harvard FAS ATG provides a high-performance computing (HPC) cluster for this course. It has 72 CPU nodes and 50 GPU nodes. The cluster is managed using SLURM. The cluster is accessible via the Open OnDemand portal. You can find documentation for the cluster at here.

We will figure out how to add you to the cluster. The login host is academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com — see the SSH config below, which gives it a short name. Still to be announced: the OnDemand portal URL, and how you get an account and get added to the CS2680 group. Those will be posted here and on the home page. Registering an SSH key is not announced, because you do it yourself: connect through the OnDemand portal once, and append your public key to ~/.ssh/authorized_keys from the browser terminal, as step 3 of first-time setup describes.

Quick start

Once, ever — from an OnDemand terminal on the login node:

Then every time you sit down to work:

And when you need a GPU instead (Assignment 4 onward):

exit ends the job and releases the node. That is the whole workflow; the rest of this page is what to do when it does not go like that.

How SLURM HPC works

There are three machines in this story. Your laptop, where you type. The login node, which is what you reach when you open an OnDemand terminal or SSH in — one small shared machine that everyone in the course lands on at once. And the compute nodes, which are the actual hardware: many CPUs, lots of memory, and in one partition, a GPU.

You are not allowed to just run things on a compute node. Slurm — the scheduler — owns them, and you ask it for a lease: give me 4 CPUs and 16 GB for 4 hours. When hardware is free it hands you a node and starts your shell on it. That lease is a job, and everything you run inside it is charged against it. When the time runs out, or you type exit, the lease ends.

The one thing that makes this pleasant is that /home is shared. The login node and every compute node see the same home directory, so anything you installed once is already there when a job starts — claude in ~/.local/bin. Nothing to re-install per job, nothing to copy onto the node, and the login you did in week one still applies, because the credentials Claude Code writes live in your home directory too.

There are two types of job. An interactive job (srun --pty) gives you a shell and you work in it, which is what an agent session is. A batch job (sbatch) runs a script without you and writes its output to a file, which is what you want for an overnight sweep.

First-time setup

Seven steps, once. Do them on the login node — they are all small, and installing software is exactly what the login node is for.

1. Get an account, and get into the CS2680 group. Access instructions are TBD and will be announced; the group membership is what lets you submit to the gpu-cs2680 partition.
2. Open a terminal. In the OnDemand portal, the shell is under the Clusters menu; it opens a terminal on the login node in a browser tab. Plain SSH to the login host works the same way and is nicer if you already live in a terminal — the SSH config below reduces it to ssh cs2680.
3. Register your SSH key. There is no upload form for this, so you install the key yourself through the OnDemand terminal you just opened, which is the one way onto the cluster that does not need a key already. On your laptop, generate a key if you do not have one, then print the public half:
   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 # skip if that file already exists cat ~/.ssh/id_ed25519.pub
   Copy the single line that prints. Note that the file to copy is id_ed25519.pub; the one without .pub is the private key, and it never leaves your laptop. Back in the OnDemand terminal, append that line to ~/.ssh/authorized_keys:
   mkdir -p ~/.ssh && chmod 700 ~/.ssh echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... you@laptop' >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
   Paste your own key between the quotes, in place of the example, and keep it on one line. Pasting into the browser terminal is Ctrl+Shift+V rather than Ctrl+V on Linux and Windows, and Cmd+V on a Mac. The two chmod lines are not optional: sshd ignores a key file that the group or the world can write, which is the usual reason a newly added key is refused without explanation. While you are in that terminal, run whoami and write the username down, because the SSH config below needs it. Then check the key from your laptop:
   ssh YOUR_USERNAME@academ-acade-iL73aitWT6xF-c83014867702e61e.elb.us-east-1.amazonaws.com
   A shell prompt on the login node means the key is registered. If you are asked for a password instead, or told Permission denied (publickey), re-open the OnDemand terminal and compare cat ~/.ssh/authorized_keys against cat ~/.ssh/id_ed25519.pub on your laptop, character for character. A key broken across two lines by the paste is the most common failure.
4. Install Claude Code into your home directory:
   curl -fsSL https://claude.ai/install.sh | bash
   This is the native installer, it puts the binary in ~/.local/bin, and it keeps itself updated. Because /home is shared, you are also installing it on every compute node at the same time.
5. Put it on your PATH.:
   echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc claude --version
6. Log in. Run claude and follow the prompts. In a browser-based terminal or over SSH you will get a code to paste back rather than a browser that returns to the terminal — that is expected, and the Claude Code page walks through it. Confirm with /status, then /exit.
7. Add the course telemetry block to ~/.claude/settings.json on this machine, per the Claude Code page. That file is per-machine, and your HPC home is a different machine from your laptop as far as it is concerned.

Interactive jobs: the everyday workflow

From an OnDemand terminal (or over SSH):

Wait for the node to come up. The first job of the day can take a few minutes, because idle nodes are powered down — sinfo shows them as idle~, and Slurm has to boot one before your shell appears. Nothing is wrong; you can type squeue -u $USER in another terminal to see the job pending reason.

When the prompt comes back you are on the compute node. Then:

That is the whole student workflow: allocate a shell on a compute node, run the agent there. Work normally, and exit when you are done, which ends the job and returns the node.

GPUs (Assignment 4, 5 and project)

The course has its own GPU partition:

Each node in it is 1× RTX PRO Server 6000.

Check the GPU is really yours, first thing inside the job:

nvidia-smi should list one card with almost no memory in use, and CUDA_VISIBLE_DEVICES should name the one device Slurm gave you. If nvidia-smi reports no devices, you are not using the correct partition.

One command from your laptop: ssh cs2680

Once the workflow above is familiar, you can collapse it into a single command. SSH will run srun for you on connect, so ssh cs2680 takes you from your laptop to a shell on a compute node with nothing typed in between. Put this in ~/.ssh/config on your own machine (C:\Users\you\.ssh\config on Windows), with your username and key path:

That gives you three names for the same machine:

• ssh cs2680 — a plain shell on the login node. This is where you install things, move files, and check the queue.
• ssh cs2680-cpu — a CPU compute node: 4 cores, 16 GB, 2 hours. Agent work goes here.
• ssh cs2680-gpu — a GPU compute node: the same, plus one RTX PRO 6000 Blackwell.

All three names authenticate with the key you installed in step 3 of first-time setup. If you skipped that step, do it now, because none of these aliases will connect until the public key is in ~/.ssh/authorized_keys on the cluster. You can also connect vscode to the cluster using the same SSH config, and it will work the same way as a terminal. Please remember to scancel JOB_ID if you do not need it anymore, otherwise it will occupy the resources for the whole time.

Surviving a closed tab: wrap it in tmux

Recommended for this class. An srun --pty shell belongs to the terminal that started it, so a closed browser tab, a laptop lid, or hotel wifi takes your agent with it — mid-edit, and the job dies too. Start the job inside tmux instead:

Now the agent is a process inside a tmux session on the compute node, and losing your terminal only detaches it. To get back:

Three things to keep straight:

• Tmux session lives on one specific node. You have to be on that node to attach, which is what squeue -u $USER tells you.
• tmux does not extend the allocation. When the walltime in -t runs out, Slurm ends the job and the tmux session with it. Nothing survives the lease.
• Detach on purpose with Ctrl-b d. That leaves everything running. Ctrl-b c opens another window in the same session, which is how you watch a log while the agent works. Please find a tmux tutorial that best suits your needs.

Long and unattended runs

Interactive jobs are for working. For an evaluation sweep — the same agent over fifty tasks, the kind of thing Assignment 3 onward asks for — you do not want to be present at all. Write a script:

Submit it with sbatch run_eval.sh, which prints a job id and returns immediately. squeue -u $USER tracks it, %j in the output path expands to the job id so concurrent runs do not overwrite each other, and everything the script prints lands in that file. Create the logs/ directory first — Slurm will not, and a job whose output file cannot be opened fails instantly for a reason that is hard to see.

Getting your code and data there

• Use git for code. Clone on the login node, commit and push from wherever you worked.
• Use rsync for everything else. From your laptop:
  rsync -avz ./data/ cs2680:~/cs2680/data/ # up rsync -avz cs2680:~/cs2680/results/ ./results/ # down
• OnDemand has a file browser for the one-off case — dragging a CSV in, pulling a plot out — without a terminal at all.
• Keep results out of the node's local disk. Write into your home directory or a shared filesystem. Anything a job leaves in /tmp on a compute node will not be available.

Sharing the HPC

We only have limited resources, please be mindful of the following and we may kill your job if you use excessive resources (especially before ddl) to make sure everyone has access to at least one server.

• Ask for what you need, we pay for the cost and each GPU costs $3.2/hr so please avoid resource waste.
• Use the resources responsibly. While we do not enforce per-student limit (yes, you can ask for many GPUs if you really need them when no one elses need them), please note that on average each student has < two servers.
• Bound every job with -t, and prefer a batch job over an interactive one you will not be watching.
• exit when you stop working. An idle allocation is invisible to you and expensive for everyone else, and the queue is longest in the week you most need it.
• Nothing heavy on the login node. It is shared by everyone at once, and the caps are there because someone always tries.

Command reference

Troubleshooting