The IMDS Saga: Five Bugs, One Wrong Parameter Name

The Iceberg sink, RisingWave writing Parquet to MinIO, took more debugging than any other part of Phronis. Multiple sessions, six related bugs, and the same misleading error the whole way down. The root cause was a parameter name with one extra word.

// 01 — THE SYMPTOM

The sink kept failing with a request to a URL I never configured:

error sending request for url (http://169.254.169.254/latest/api/token)

169.254.169.254 is the AWS EC2 instance metadata service (IMDS), the address a cloud SDK hits to find credentials when none are supplied. But this was MinIO, on my own machine. Why was it asking AWS for credentials?

// 02 — THE FALSE TRAILS

Because the error pointed at credentials, I chased credentials and fixed several real problems that weren’t the root cause:

• Added the missing AWS env vars to the RisingWave service, with AWS_EC2_METADATA_DISABLED=true to stop the IMDS fallthrough.
• Mounted a docker/aws/credentials file as a belt-and-suspenders fallback (reqsign’s credential-chain order varies by version).
• Later: a missing create_table_if_not_exists = 'true' (RisingWave v2.2 won’t auto-create the Iceberg table), a wrong s3a:// warehouse scheme (a Hadoop convention RisingWave doesn’t recognize; it wants s3://), and a missing s3.path.style.access = 'true' that MinIO requires.

Each was a genuine bug. None made the IMDS error go away.

// 03 — THE ROOT CAUSE

The credentials were in the config the whole time. They were being silently discarded, because the parameter names were wrong:

s3.access.key.id      →   s3.access.key
s3.secret.access.key  →   s3.secret.key

RisingWave v2.2+ uses the shorter names. The longer ones (which look correct, and match other tools) were simply unrecognized keys, so RisingWave ignored them, found no credentials, and fell through to IMDS. The misleading error was a symptom; the cause was two unrecognized config keys throwing my values on the floor.

// 04 — THE LESSON

Once the names were right, every other fix clicked into place and the sink worked. The whole saga was an exercise in a hard truth: a misleading error sends you debugging the symptom. “Can’t reach IMDS” screamed credentials missing, when the real story was credentials supplied under names nothing was reading.

TAKEAWAYS

• Unrecognized config keys are usually silent. A typo’d parameter name isn’t an error. It’s an ignored value, and the downstream failure rarely points back at it.
• When an error makes no sense for your environment (IMDS on a local box), suspect that your config never took effect, not that you configured it wrong.
• Verify parameter names against the exact version you’re running. s3.access.key.id vs s3.access.key cost multiple sessions.

NEXT

• Anomaly log: the circuit breaker that tripped before the demo even started.