Railway Breach Puts API Keys at Risk Amid Major Outage

Railway, a cloud application hosting platform, has warned that an intruder may have accessed customer environment variables and decrypted some of their contents, putting API keys and database credentials at risk. Its 21 October security disclosure urges all customers to rotate stored secrets while the hosting platform investigates.

Editor’s note (24 October): Railway’s 23 October, 22:03 UTC notice says its support and community site, station.railway.com, remains unavailable and new signups are temporarily paused. The dashboard, API, builds and deployments are not affected. Customers needing help can contact support@railway.com. The article below reflects the earlier outage and security-disclosure updates.

A major outage has complicated that response. Railway initially paused every deployment; its 22 October, 09:28 UTC status update said builds were available for Pro and Enterprise workspaces, with elevated failures and queues. Hobby, Free and Trial builds were still awaiting restoration.

Environment variables are settings supplied to an application when it runs. They often contain credentials for connected services, so a Railway breach can also put an application's databases, cloud accounts and AI provider accounts at risk.

Rotate Secrets at Their Source

Rotate secrets stored in Railway environment variables, including API keys, and replace private container-registry credentials, following Railway's recommendation. Its initial assessment suggested only a subset of users might be affected, but it advised every customer to take precautions.

Revoke exposed credentials with the service that issued them. Removing a value from Railway cannot invalidate a copy held elsewhere. Railway's OpenAI recovery guide, for example, directs customers to delete the old key, create a replacement, update their application and inspect usage.

• Inventory the connections: identify secrets in each project's environments, shared variables and private-registry settings, including staging and background services.
• Revoke and replace: follow each provider's incident instructions, record the affected key identifiers and update every application that used them. Avoid copying secret values into incident notes.
• Verify the running application: once builds are available for your plan, deploy the replacement configuration and confirm the service can authenticate successfully.

Major Outage Complicates Recovery

Railway's status page records disruption to its API, dashboard, support site and build pipeline on 21 October. The deployment pause prevented customers from rolling out replacement credentials through their usual release process.

Revoking a key may interrupt an application until it can run with the replacement. Coordinate the change with the affected provider and the person leading your incident response. Updating a value in the Railway dashboard does not mean the running service is using it yet.

The PostgreSQL recovery guide explicitly requires redeploying every dependent service and checking that old credentials no longer connect. Its checks include successful application queries and authentication errors after rotation.

Recovery remained partial at the 22 October status checkpoint. Railway asked customers to submit each deployment once because repeated submissions add to the queue. Once builds resume, customers still need to verify replacement credentials and check for unauthorised access.

Railway Resets Tokens and Bucket Credentials

Railway invalidated all Railway API tokens, according to its 04:17 UTC status update on 22 October. Integrations using the public API need newly created tokens.

Its bucket and GitHub recovery guide says storage-bucket credentials were regenerated and GitHub connections reset. Services need redeployment to pick up bucket credentials, while customers must reconnect GitHub and check repository access.

Railway removed stored private GitHub Container Registry tokens, but those credentials can remain valid on GitHub until their owner revokes them. Revoke and replace your tokens at GitHub, then update each affected service's registry settings.

Check Provider Usage and Database Access

Check the external services accessible through credentials stored in Railway. Railway's OpenAI guide recommends reviewing usage by project and key, looking for unfamiliar requests and unexpected spending, and contacting provider support with evidence of misuse.

For AWS credentials, Railway's AWS response guide calls for deactivating affected keys and examining CloudTrail and billing. Investigate unfamiliar resources, changes to permissions and activity originating outside your infrastructure.

• AI and payment services: inspect provider-side usage, billing and audit records for actions your application did not generate.
• Databases and storage: check connection logs, permission changes, unfamiliar users and object access; preserve available evidence before routine log retention removes it.
• GitHub and registries: review account and organisation audit records, repository access and unexpected image publication alongside the credential reset.

The Breach Scope Remains Unconfirmed

Railway said it believed the incident formed part of attacks on platform companies aimed at compromising AI inference providers, and that it had notified upstream providers. That is Railway's assessment; its public disclosure does not identify the attacker or establish a technical link to other October incidents.

Railway's initial disclosure does not give a final affected-account count, explain how decryption occurred or quantify misuse of exposed credentials. Railway says it will contact confirmed affected customers directly and publish further updates. Customers can begin the recommended checks while that investigation continues.