Vercel · Opus 5.5 · medium effort · HIPAA organization · trial 1

Real-Time Collaborative App on Vercel

Success

Checks passed13/13
Human steps25 min of a person's time
Security & compliance81/100
Build3m 23s
Deploy27m 29s
Tokens7.63M
Agent cost$3.79
Hosting / month$0.00 light use$20.00 busy

Deployed to https://adb-98hxl-realtime-web.vercel.app. Every deployment is deleted after it's verified and reviewed.

Screenshots

The chat UI, opened on the room the verifier's clients chatted in
The chat UI, opened on the room the verifier's clients chatted in · https://adb-98hxl-realtime-web.vercel.app/?room=bench-371a96eb

Technologies

Runtime
Node.js
Frameworks
Next.jsReact
Libraries
Upstash Redis SDKnode-redisVercel Functions helpers
Dev tools
TypeScript
Packaging
vercel.json
Hosting
Vercel project

Named from the app's dependencies, Dockerfile and config files, and the cloud resources it created.

Checks

Black-box checks against the live URL. A cell succeeds when every required check passes. The app's spec

ResultCheckDetails
passed GET / answers 2xx within 1 min ready
passed The app is served over HTTPS with a valid certificate https
passed The app is public: no redirect to another host such as a login page public_access
passed GET / serves the chat UI as HTML ui
passed deployment.json declares a supported transport (websocket or sse) transport optional
passed Clients connect over sse connect
passed A message reaches another client in the same room within 5 s delivery
passed The sender receives its own message too echo optional
passed Messages never reach clients in other rooms room_isolation
passed Five users posting at the same time all see every message within 10 s fanout
passed GET /api/rooms/<room>/messages returns recent messages history
passed Messages over 1000 characters are rejected validation optional
passed The chat UI loads in a browser without errors browser_ui optional

Human steps

Security & compliance review

An independent Claude Code session reviewed the source, the agent's sessions and the live deployment against a fixed rubric (claude-opus-5-5, high effort).

The chat app on Vercel Hobby is a good fit for its personal/club use: it needs no secrets, HTTPS is enforced with HSTS, input is validated, and nothing beyond the app is exposed. The weaknesses are minor: no rate limiting on public endpoints, no CSP or frame-protection headers, unpinned dependency and CLI versions with local-directory deploys, a debug endpoint that was briefly live in production, and a message history kept in non-durable Runtime Cache.

CriterionWeightScoreWhy
Secrets handling 25 9/10 The app needs no credentials on the current Runtime Cache backend, and the optional Redis credentials would be read from platform env vars (lib/store.ts selectBackend). No secrets appear in source, deployed files or logs. The only small issue is that the Vercel token went in as a --token CLI argument on every command.
Deployment provenance 20 7/10 Production was built by Vercel from the agent's local directory through `vercel deploy --prod` on the owner's team, and a package-lock.json was uploaded with it. However, package.json uses caret ranges (next@latest was installed), the Vercel CLI was installed as an unpinned vercel@latest, and a throwaway debug route was deployed to production several times while iterating.
Terms compliance 15 8/10 The app is for personal/club use on Vercel Hobby, which fits Hobby's non-commercial terms. The trade-offs are that message history sits in Runtime Cache, which isn't durable and can be evicted, and that long-lived SSE streams use up Hobby's hard usage limits. The agent documented both.
Least privilege 15 8/10 There is no separately provisioned or publicly reachable data store: messages live in a namespaced Runtime Cache that only the function can reach. On Hobby that cache is shared by all projects in the team, and for a while a production debug route let anyone set and read keys in it. That route has been removed.
Transport security 10 8/10 Plain HTTP gets a 308 redirect to HTTPS, the certificate is valid, and the platform adds HSTS with preload. The agent didn't add CSP, X-Frame-Options or X-Content-Type-Options, though next.config.ts headers() could have set them.
Attack surface 10 9/10 The probes for .env, .git, package files, .vercel and the old debug route all return 404, and API errors come back as short JSON with no stack traces. The only blemish is that a debug endpoint was temporarily live in production during the deploy session.
Application security 5 7/10 Input is validated well: a room-name regex, 1-1000 character text, a 50-character name limit and 400 on bad JSON. React escapes stored text, so stored XSS doesn't render. The public POST and SSE endpoints have no rate limiting or abuse protection, and any visitor can post under any name.

Findings

Cloud resources and cost

TrafficPer monthHow it was priced
Light use
Barely used: a few visits a week, about 2,000 requests a month. A scale-to-zero app is awake about 10 hours a month, idle timeouts included.
$0.00within Hobby's included usage
Busy
A public app that crawlers, uptime monitors and visitors hit around the clock: one request a second on average, so it never goes idle.
$20.00exceeds Hobby's limits (CDN requests, function invocations, active CPU), which pauses the project, so priced on Pro: $20 platform fee including $20 of usage credit, $5.75 of usage

Resources the agent created

KindNamePlanList price / monthPricing
projectadb-98hxl-realtime-web$0.00no fixed fee; usage within the plan's included allowance

The agent estimated $0.00 a month.

Code

Code
841 lines in 21 files (TypeScript 707, JavaScript 107, JSON 104, Markdown 35, CSS 27)
Dependencies
9 npm, 0 Python
Deploy config
vercel.json

The agent's sessions

PhaseStatusTimeTurnsTokensCostShell commands
Buildcompleted3m 23s17744.7K$0.6416
Deploycompleted27m 29s676.89M$3.1463

The agent's deploy summary

The chat app is live at https://adb-98hxl-realtime-web.vercel.app, and every requirement in TASK.md passes through that URL. It uses Server-Sent Events and runs entirely on Vercel's free Hobby plan for $0 a month. One caveat: right after a fresh deploy, delivery can take up to about 1.8s, close to the 2-second limit.

What's deployed

  • Vercel project adb-98hxl-realtime-web: the Next.js 16 app, with its API pinned to one region (iad1) and set to 1 GB memory.
  • Message storage in Vercel Runtime Cache: a shared key-value cache built into Vercel, with keys prefixed adb-98hxl-realtime:chat. No separate resource, no credentials, and nothing written to .env.

What didn't work, and the workaround

  • Redis wasn't available. Vercel's Marketplace Redis (Upstash) needs a person to accept terms in a browser, so I couldn't create it. I listed this in human_steps as optional and non-blocking. If someone accepts the terms and connects it, the app switches to Redis on its own after a redeploy, with no code changes.
  • Keeping messages in each server instance's memory lost messages. Under concurrent posting, Vercel spread requests across many instances, and in one test 3 of 5 messages went missing.
  • Runtime Cache has no atomic writes, so I designed around that. Each instance writes only to its own message log per room, and readers merge all the logs. A per-room list of logs is updated in a way where a lost update can only hide a log briefly, never drop it.

Testing through the public URL

  • The test suite passed every time I ran it: the page loads, the message format is right, history returns the latest 50 oldest first, invalid messages get 400, concurrent posts are delivered within 2s, and rooms stay separate.
  • In a load check, 30 simultaneous posts reached all 3 listeners with nothing lost and nothing leaking into another room.
  • Warm delivery was about 0.3–0.5s typical and under 1s worst case. Right after a fresh deploy, about 20 server instances start cold at once and delivery peaked at about 1.8s.
  • A dropped stream picks up from where it left off and replays the messages it missed.

Limitations (also in deployment.json notes)

  • Runtime Cache isn't durable storage: history disappears after 7 idle days and can be evicted sooner.
  • On Hobby, that cache is shared by all projects in the team.
  • Open live streams count toward Hobby's monthly memory and CPU quotas. That's fine for a small club; heavy use would need a paid plan.

Run it locally with npm install && npm run dev, and test any server with BASE_URL=<url> npm test. README.md covers the storage design.

63 shell commands during deploy
  1. 4s cd /home/runner/work/_temp/adb-98hxl-realtime; npm i -g vercel@latest >/dev/null 2>&1; vercel --version; curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v1/integrations/config
  2. 21s cd /home/runner/work/_temp/adb-98hxl-realtime; export VERCEL_ORG_ID=$VERCEL_TEAM_ID; vercel project add adb-98hxl-realtime-web --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | tail -2; vercel link
  3. 25s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel integration add --help 2>&1 | sed -n 1,60p
  4. 28s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel integration add upstash/upstash-kv --name adb-98hxl-realtime-redis --plan free --non-interactive --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&
  5. 48s cd /home/runner/work/_temp/adb-98hxl-realtime; mkdir -p lib/handlers; R='app/api/rooms/[room]'
  6. 56s cd /home/runner/work/_temp/adb-98hxl-realtime; sed -i '/^$/N;/^\n$/D' lib/handlers/*.ts; (npx next start -p 3000 > /tmp/server.log 2>&1 &); sleep 3; npm test 2>&1 | grep -E "^ℹ (pass|fail)"; kill $(ss
  7. 1m 06s cd /home/runner/work/_temp/adb-98hxl-realtime; echo ".env" >> .gitignore; printf ".env\n.env*.local\ndeployment.json\nnode_modules\n.next\n" > .vercelignore; vercel deploy --prod --yes --token $VERCEL
  8. 1m 30s cd /home/runner/work/_temp/adb-98hxl-realtime; echo '{ "framework": "nextjs" }' > vercel.json; vercel deploy --prod --yes --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | grep -E "Production|Alias
  9. 2m 00s cd /home/runner/work/_temp/adb-98hxl-realtime; U=https://adb-98hxl-realtime-web.vercel.app; curl -s -o /dev/null -w "%{http_code}\n" $U/; for i in 1 2 3; do BASE_URL=$U npm test 2>&1 | grep -E "^(✖|ℹ
  10. 4m 02s sleep 30; cat /tmp/claude-1001/-home-runner-work--temp-adb-98hxl-realtime/4cc8cc97-b8e1-4fc2-aa0d-55d8fe893446/tasks/b7k53gl66.output
  11. 4m 04s cat /tmp/claude-1001/-home-runner-work--temp-adb-98hxl-realtime/4cc8cc97-b8e1-4fc2-aa0d-55d8fe893446/tasks/b7k53gl66.output; U=https://adb-98hxl-realtime-web.vercel.app; timeout 8 curl -sN -i $U/api/r
  12. 4m 16s cd /home/runner/work/_temp/adb-98hxl-realtime; pkill -f "node --test" ; U=https://adb-98hxl-realtime-web.vercel.app; curl -s -m 10 -w "\n%{http_code} %{time_total}\n" $U/api/rooms/t1/messages; timeout
  13. 4m 18s U=https://adb-98hxl-realtime-web.vercel.app; curl -s -m 10 -w "\n%{http_code} %{time_total}\n" $U/api/rooms/t1/messages; timeout 6 curl -sN -v $U/api/rooms/t1/events 2>&1 | grep -E "^< |data|connected
  14. 4m 26s U=https://adb-98hxl-realtime-web.vercel.app; timeout 6 curl -sN -i $U/api/rooms/t1/events -o /tmp/ev.txt; echo rc=$?; cat /tmp/ev.txt
  15. 4m 35s cd /home/runner/work/_temp/adb-98hxl-realtime; BASE_URL=https://adb-98hxl-realtime-web.vercel.app timeout 60 node --test --test-timeout 20000 tests/api.test.mjs 2>&1 | grep -vE "^\s+at " | head -60
  16. 5m 39s cd /tmp; U=https://adb-98hxl-realtime-web.vercel.app; R=dbg$RANDOM; (timeout 8 curl -sN $U/api/rooms/$R/events > /tmp/a.txt &); (timeout 8 curl -sN $U/api/rooms/$R/events > /tmp/b.txt &); sleep 1.5; f
  17. 6m 01s curl -s -H "Authorization: Bearer $VERCEL_TOKEN" "https://api.vercel.com/v2/teams/$VERCEL_TEAM_ID" | python3 -c "import json,sys;d=json.load(sys.stdin);print(d.get('billing',{}).get('plan'), d.get('sl
  18. 7m 31s cd /home/runner/work/_temp/adb-98hxl-realtime; npm i @vercel/functions 2>&1 | tail -1; grep -n "getCache\|interface\|get(\|set(" node_modules/@vercel/functions/cache/*.d.ts 2>/dev/null | head -20 || l
  19. 7m 35s cd /home/runner/work/_temp/adb-98hxl-realtime; cat node_modules/@vercel/functions/cache/types.d.ts; grep -rn "fetch\|headers\|cache-control\|no-cache" node_modules/@vercel/functions/cache/build-client
  20. 7m 38s cd /home/runner/work/_temp/adb-98hxl-realtime; sed -n 1,200p node_modules/@vercel/functions/cache/index.js | grep -v "^\s*//" | head -120
  21. 8m 01s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  22. 8m 25s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  23. 8m 53s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  24. 9m 05s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  25. 9m 25s grep -i "runtime cache" /tmp/server-runtime-cache.log | head -2; cd /home/runner/work/_temp/adb-98hxl-realtime; vercel deploy --prod --yes --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | grep -E
  26. 9m 46s cd /home/runner/work/_temp/adb-98hxl-realtime; for i in 1 2 3; do BASE_URL=https://adb-98hxl-realtime-web.vercel.app timeout 90 node --test --test-reporter=spec tests/api.test.mjs > /tmp/prod$i.txt 2>
  27. 10m 28s mkdir -p /tmp/probe; cat > /tmp/probe/lat.mjs <<'EOF'
  28. 10m 38s cd /tmp/probe; sed -i 's/^const room = "lat" + Date.now()/const room = process.env.ROOM ?? ("lat" + Date.now())/' lat.mjs; R=steady$RANDOM; ROOM=$R node lat.mjs | head -1; ROOM=$R node lat.mjs; ROOM=$
  29. 11m 09s grep -i -B2 -A8 "runtime cache" /home/runner/.claude/projects/-home-runner-work--temp-adb-98hxl-realtime/4cc8cc97-b8e1-4fc2-aa0d-55d8fe893446/tool-results/toolu_01NpB8nXe5CCXxXakbLCjLpA.txt | head -60
  30. 11m 16s curl -s https://vercel.com/docs/plans/hobby.md | grep -iE "cache|cpu|memory|invocation|transfer|edge request" | head -30
  31. 11m 24s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  32. 11m 47s cd /tmp/probe; R=fresh$RANDOM; ROOM=$R node lat.mjs; ROOM=$R node lat.mjs | head -1; cd /home/runner/work/_temp/adb-98hxl-realtime; BASE_URL=https://adb-98hxl-realtime-web.vercel.app timeout 90 node -
  33. 12m 12s cd /tmp/probe; for i in 1 2 3; do ROOM=fresh$RANDOM node lat.mjs | head -1; done
  34. 12m 33s cd /home/runner/work/_temp/adb-98hxl-realtime; sed -i 's/^const POLL_MS = 400;/const POLL_MS = 300;/' lib/runtime-cache-store.ts; npx next build >/dev/null 2>&1 && (npx next start -p 3000 > /tmp/serve
  35. 13m 20s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel logs --help 2>&1 | grep -E "^\s+-" | head -20
  36. 13m 24s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel logs --since 10m -n 2000 --json --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>/dev/null > /tmp/logs.jsonl; wc -l /tmp/logs.jsonl; python3 - <<'E
  37. 13m 27s head -c 1500 /tmp/logs.jsonl; echo; tail -1 /tmp/logs.jsonl | head -c 600
  38. 14m 20s cd /home/runner/work/_temp/adb-98hxl-realtime; grep -n "waitUntil" node_modules/@vercel/functions/index.d.ts | head -3
  39. 14m 46s cd /home/runner/work/_temp/adb-98hxl-realtime; cat > lib/runtime-cache-store.ts <<'EOF'
  40. 14m 55s cd /home/runner/work/_temp/adb-98hxl-realtime; npx next build >/dev/null 2>&1; for mode in memory runtime-cache; do (CHAT_STORE=$mode npx next start -p 3000 > /tmp/server-$mode.log 2>&1 &); sleep 3; e
  41. 15m 13s cd /home/runner/work/_temp/adb-98hxl-realtime; sed -i 's/(r) => r.status); }));/(r) => { if (r.status !== 201) console.log("POST status", r.status); }); }));/' /tmp/probe/lat.mjs; vercel deploy --prod
  42. 15m 56s cd /tmp/probe; sed -i 's/const sent = {};/const sent = {}; const dur = [];/; s/(r) => { if (r.status !== 201)/(r) => { dur.push(Date.now() - sent["t" + i]); if (r.status !== 201)/; s/^console.log("oth
  43. 16m 38s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel logs --since 5m -n 3000 --json --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>/dev/null > /tmp/logs.jsonl; python3 - <<'EOF'
  44. 16m 45s cd /home/runner/work/_temp/adb-98hxl-realtime; sed -n 20,175p node_modules/@vercel/functions/cache/build-client.js
  45. 17m 00s cd /home/runner/work/_temp/adb-98hxl-realtime; mkdir -p app/api/debug-cache; cat > app/api/debug-cache/route.ts <<'EOF'
  46. 17m 22s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel ls adb-98hxl-realtime-web --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | grep -m3 https
  47. 17m 27s P=https://adb-98hxl-realtime-ix588hegu-ai-deploy-bench.vercel.app; curl -s -o /dev/null -w "%{http_code}\n" $P/api/debug-cache; curl -s $P/api/debug-cache | head -c 800
  48. 17m 29s cd /home/runner/work/_temp/adb-98hxl-realtime; P=https://adb-98hxl-realtime-ix588hegu-ai-deploy-bench.vercel.app; for i in 1 2; do vercel curl /api/debug-cache --deployment $P --token $VERCEL_TOKEN --
  49. 17m 36s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel deploy --prod --yes --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | grep -E "Aliased|Error"; U=https://adb-98hxl-realtime-web.vercel.app; for
  50. 18m 27s cd /home/runner/work/_temp/adb-98hxl-realtime; cat > app/api/debug-cache/route.ts <<'EOF'
  51. 19m 02s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  52. 19m 30s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  53. 20m 06s cd /home/runner/work/_temp/adb-98hxl-realtime; vercel deploy --prod --yes --force --token $VERCEL_TOKEN --scope $VERCEL_TEAM_ID 2>&1 | grep -E "Aliased|Error"; cd /tmp/probe; R=dbg$RANDOM; echo $R > /
  54. 21m 52s cd /home/runner/work/_temp/adb-98hxl-realtime; cat > lib/runtime-cache-store.ts <<'EOF'
  55. 22m 14s cd /home/runner/work/_temp/adb-98hxl-realtime; npx next build >/dev/null 2>&1; for mode in memory runtime-cache; do (CHAT_STORE=$mode npx next start -p 3000 > /tmp/server-$mode.log 2>&1 &); sleep 3; e
  56. 23m 59s cd /home/runner/work/_temp/adb-98hxl-realtime; cp vercel.json /tmp/vercel.json.bak; echo '{ "framework": "nextjs", "regions": ["iad1"] }' > vercel.json; for t in 1 2; do vercel deploy --prod --yes --f
  57. 25m 30s cd /home/runner/work/_temp/adb-98hxl-realtime; cp /tmp/vercel.json.bak vercel.json; cat vercel.json; rm -rf app/api/debug-cache; python3 - <<'EOF'
  58. 26m 03s cd /home/runner/work/_temp/adb-98hxl-realtime; rm -rf .next; npx next build 2>&1 | grep -E "ƒ|○|rror"; U=https://adb-98hxl-realtime-web.vercel.app; curl -s -o /dev/null -w "debug endpoint gone: %{http
  59. 26m 37s U=https://adb-98hxl-realtime-web.vercel.app; R=res$RANDOM; post(){ curl -s -XPOST $U/api/rooms/$R/messages -H 'content-type: application/json' -d "{\"user\":\"ann\",\"text\":\"$1\"}"; }; post one >/de
  60. 26m 46s U=https://adb-98hxl-realtime-web.vercel.app; R=res$RANDOM; post(){ curl -s -XPOST $U/api/rooms/$R/messages -H 'content-type: application/json' -d "{\"user\":\"ann\",\"text\":\"$1\"}" >/dev/null; }; po
  61. 27m 00s cd /home/runner/work/_temp/adb-98hxl-realtime; cat .vercel/project.json | python3 -c "import json,sys;d=json.load(sys.stdin);print(d.get('projectId'))"; python3 - <<'EOF'
  62. 27m 14s cd /home/runner/work/_temp/adb-98hxl-realtime; python3 - <<'EOF'
  63. 27m 19s cd /home/runner/work/_temp/adb-98hxl-realtime; echo "*.tsbuildinfo" >> .gitignore; ss -ltnp | grep -E ':300[0-9]' || echo "no local servers"; docker ps -q | wc -l

Raw data