Railway · Opus 5.5 · medium effort · HIPAA organization · trial 1

Real-Time Collaborative App on Railway

Success

Checks passed13/13
Human steps37 min of a person's time
Deploy without a human Failed
Deploy with a human Deployed
Security & compliance82/100
Stack fit86/100
Build1m 27s
Deploy1m 30s
Tokens811.8K
Agent cost$0.60
Hosting / month$5.28 light use$27.63 busy

Deployed to https://adb-klpuj-realtime-web-production.up.railway.app. Every deployment is deleted after it's verified and reviewed.

Screenshots

The room the verifier's clients chatted in, with its history
The room the verifier's clients chatted in, with its history · https://adb-klpuj-realtime-web-production.up.railway.app/?room=bench-dcb55e6e

Technologies

Runtime
Node.js 20+
Frameworks
Node.js http module (built in)
Libraries
ws
Packaging
railway.json

Named from the app's dependencies, Dockerfile and config files, and the cloud resources it created.

Checks

Black-box checks against the live URL. A cell succeeds when every required check passes. The app's spec

ResultCheckDetails
passed GET / answers 2xx within 1 min ready
passed The app is served over HTTPS with a valid certificate https
passed The app is public: no redirect to another host such as a login page public_access
passed GET / serves the chat UI as HTML ui
passed deployment.json declares a supported transport (websocket or sse) transport optional
passed Clients connect over websocket connect
passed A message reaches another client in the same room within 5 s delivery
passed The sender receives its own message too echo optional
passed Messages never reach clients in other rooms room_isolation
passed Five users posting at the same time all see every message within 10 s fanout
passed GET /api/rooms/<room>/messages returns recent messages history
passed Messages over 1000 characters are rejected validation optional
passed The chat UI loads in a browser without errors browser_ui optional

Human steps

Stack fit

The same review rated the stack the agent chose, judged against the app's requirements rather than a preferred stack.

A minimal Node.js + ws service on one Railway container fits a small realtime chat well and follows the reasoning in STACK.md. The main gap is in-memory history, which doesn't survive restarts.

CriterionWeightScoreWhy
Fit for the job 40 8/10 Node with `ws` is a good fit for room broadcast over WebSockets, and SSE is there as a fallback. History is kept only in memory, so it's lost on every redeploy or restart. STACK.md accepts that trade-off, and for a hobby club it's tolerable but weaker than a small durable store.
Proportion 30 9/10 One process, one dependency, no framework: about as simple as the requirements allow. The extra SSE path is optional but cheap.
Platform fit 30 9/10 A long-running container suits Railway, and its edge terminates TLS so wss:// works with no setup. The single-replica limit is documented and enforced in railway.json.

Security & compliance review

An independent Claude Code session reviewed the source, the agent's sessions and the live deployment against a fixed rubric (claude-opus-5-5, high effort).

This is a small single-service Node.js chat app on Railway that handles secrets cleanly and exposes nothing beyond its own routes over valid HTTPS, with HTTP redirected. The main weakness is in the app: an unguarded decodeURIComponent lets one malformed request crash the process, and there's no rate limiting or WebSocket Origin check. Smaller issues are missing security headers, unpinned Railpack/npm build inputs and a root container.

CriterionWeightScoreWhy
Secrets handling 25 10/10 The app needs no secrets. The Railway token came from the environment or a git- and railway-ignored .env, and it never shows up in commands, logs or deployed files.
Deployment provenance 20 7/10 The code in production is the reviewed source, uploaded with `railway up` from the agent's workspace and built by Railpack. But the base image and Node version are whatever Railpack picks (engines is only `>=20`), `ws` is a caret range, and the bundle has no lockfile, so the build inputs aren't pinned.
Terms compliance 15 8/10 This is a personal/club app on an existing Railway account with usage-based billing, and Railway's Hobby/Pro terms allow that. The benchmark couldn't read the exact plan, though. If it's the trial, the credits run out and the service would stop.
Least privilege 15 8/10 There is one service with no data stores, no extra public resources and a single replica. The Railpack-built container runs with the default (root) user, and the agent didn't change that.
Transport security 10 8/10 HTTPS works with a valid Let's Encrypt wildcard certificate, and HTTP gets a 301 redirect to HTTPS. Neither the app nor Railway sets HSTS, CSP, X-Content-Type-Options or frame protection, and the app could easily have added them.
Attack surface 10 9/10 Only /, /health, the room API and /ws respond. /.env, /.git/config, /package.json and /server.js all return a JSON 404, and nothing shows stack traces or directory listings.
Application security 5 5/10 Input checks and DOM rendering are sound (room regex, 1–1000 character text, textContent everywhere). But one malformed percent-encoded room path throws an uncaught URIError that crashes the process, and there is no rate limiting, connection cap or WebSocket Origin check.

Findings

Cloud resources and cost

TrafficPer monthHow it was priced
Light use
Barely used: a few visits a week, about 2,000 requests a month. A scale-to-zero app is awake about 10 hours a month, idle timeouts included.
$5.28memory billed all month, CPU for ~10 h of activity, 0.04 GB egress at $0.05/GB
Busy
A public app that crawlers, uptime monitors and visitors hit around the clock: one request a second on average, so it never goes idle.
$27.63memory billed all month, CPU for ~730 h of activity, 52.6 GB egress at $0.05/GB

Resources the agent created

KindNamePlanList price / monthPricing
projectadb-klpuj-realtime1 service(s)$25.001 service(s) at the assumed 1 vCPU / 0.5 GB each, always on; usage is metered per second so a mostly idle service costs less than this list price

The agent estimated $1.00 a month.

Code

Code
329 lines in 10 files (JavaScript 218, HTML 98, JSON 43, Markdown 21, Shell 13)
Dependencies
1 npm, 0 Python
Deploy config
railway.json

The agent's sessions

PhaseStatusTimeTurnsTokensCostShell commands
Buildcompleted1m 27s7256.0K$0.326
Deploycompleted1m 30s8383.4K$0.187

The agent's deploy summary

The app is live at https://adb-klpuj-realtime-web-production.up.railway.app, and every requirement in TASK.md passed when I tested it through that URL.

What I created: a Railway project adb-klpuj-realtime with one Node.js web service, adb-klpuj-realtime-web, running a single copy on the free *.up.railway.app domain. Railway provides the HTTPS/WSS. I estimate the cost at about $1 a month; that's a guess, not a quote, and Railway's included credits may cover it. Nothing needed a person, so human_steps is empty, and deployment.json is written with realtime set to wss://adb-klpuj-realtime-web-production.up.railway.app/ws.

Tested through the public URL:

  • Web page: GET / returns 200 and serves the chat page.
  • WebSocket: two users each sent 10 messages at the same time, and all 20 reached both of them, the slowest in about 145 ms. A user in another room received none, and messages arrived in the required format.
  • Bad messages: empty and 1001-character messages got 2 error replies over WebSocket and 400 over HTTP.
  • HTTP posting and history: a valid POST returned 201, and the history endpoint returned the room's messages oldest first.
  • Server-Sent Events: the stream returned text/event-stream and delivered a posted message straight away.

Limitations:

  • Chat history is kept in memory, so it's lost on every redeploy or restart.
  • The service has to stay at one copy, because each room's messages live in that one process.
  • The Railway CLI warns that railway.json is deprecated and keeps working only until 2026-12-01. After that, the config needs migrating with railway config migrate.
7 shell commands during deploy
  1. 3s railway whoami 2>&1; railway list 2>&1 | head
  2. 7s railway init --name adb-klpuj-realtime 2>&1 | tail -3 && railway add --service adb-klpuj-realtime-web 2>&1 | tail -3 && railway service adb-klpuj-realtime-web 2>&1 | tail -2
  3. 12s railway status --json 2>/dev/null | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d["name"],d["id"]);[print(e["node"]["name"],e["node"]["id"]) for e in d["services"]["edges"]]'
  4. 19s railway up --service adb-klpuj-realtime-web --ci 2>&1 | grep -v -i 'config as code\|migrate\|keep working' | tail -15
  5. 57s railway domain --service adb-klpuj-realtime-web 2>&1 | grep -i http
  6. 1m 10s cat > /tmp/remote.js <<'EOF'
  7. 1m 23s cat > deployment.json <<'EOF'

Raw data