# Troubleshooting Object Storage

Errors S3 clients and the console can show for a bucket, with their cause and fix.

## SignatureDoesNotMatch

**Cause:** the wrong secret, or a client that rewrites the host.

**Fix:** copy the key again; set the endpoint to `https://s3.<domain>` exactly.

## InvalidAccessKeyId

**Cause:** the key was deleted or is inactive ("The access key ID you provided does not exist in
our records").

**Fix:** activate the key, or create a new one (see [Create an access key](create-an-access-key.md)).

## AccessDenied "This key has expired" or "The key’s owner may no longer act in this organization"

**Cause:** the access key expired, or its IAM user was deactivated or the person left the
organization.

**Fix:** create a new key for an active user.

## AccessDenied on a bucket you can see in the console

**Cause:** the policies of the key's IAM user or service account do not grant the action on that
object. A key made with the bucket's **Create access key** in the last 30 seconds may not be seen
by S3 yet.

**Fix:** add the action for the bucket or prefix to the account's policies (see
[Access control](access-control.md)); the bucket's **Access** tab shows who reaches the bucket and
through which policy. A key for one bucket cannot list your buckets: name the bucket in every
command.

## AccessDenied with a legacy limited key

**Cause:** the key was made with a limit, and its account's policies do not cover what the limit
names. A limit only narrows the account; it never grants. The bucket's **Access** tab shows such a
key as **Does not work** or partly working.

**Fix:** migrate the key: **Identity** › **Access keys** › **Legacy limited keys** › **Migrate**
creates a service account with what the key can really do and a new key for it (see
[Keys with a limit](limit-a-key-to-one-bucket.md#keys-with-a-limit-legacy)). To give it more,
attach a policy for the bucket to the account.

## 404 NoSuchKey on ListObjectsV2, or 400 InvalidRequest "This address already names the bucket …"

**Cause:** the client sends path-style requests (the bucket name in the path) to the bucket's own
address, `https://<bucket>.s3.<domain>`, which already names the bucket in the host. Requests on
the bucket, such as a listing, answer `400` `InvalidRequest`; an object read answers `404`
`NoSuchKey` with the same hint.

**Fix:** set the client's endpoint to `https://s3.<domain>`, not the bucket's address, and give the
bucket name and the region `de1` separately. Or keep the bucket's address and turn path style off
in the client. The bucket's Overview shows the setup for each tool under **Connect a tool** (see
[S3 compatibility](s3-compatibility.md)).

## AccessDenied "This bucket is managed by the platform and is read-only"

**Cause:** a write to a bucket the platform manages, such as a database's backup bucket (`rds-…`).

**Fix:** only read it; restore a database through the database.

## RequestTimeTooSkewed

**Cause:** the client's clock is more than 15 minutes off.

**Fix:** synchronise the clock.

## AccessDenied on put-bucket-policy

**Cause:** the policy is public and Block public access is on.

**Fix:** allow public access on the bucket first (see [Make objects public](make-objects-public.md)).

## The website answers 403 for every page

**Cause:** nothing in the bucket is public; the console shows **On, not public**.

**Fix:** allow public access and add a public-read policy.

## A custom domain has no certificate

**Cause:** the name does not resolve to Deutsche Cloud yet, was not added, or the bucket's website is
off.

**Fix:** check the CNAME; turn the website on and add the domain on the bucket's **Website** tab (see
[Use a custom domain for a website](use-a-custom-domain.md)).

## ExpiredToken or InvalidToken

**Cause:** temporary credentials past their `expiresAt` (`ExpiredToken`); a session token missing,
wrong, or sent with a permanent key (`InvalidToken`).

**Fix:** create new temporary credentials; set `AWS_SESSION_TOKEN` for temporary credentials only.

## KeyTooLongError

**Cause:** an object key longer than 1024 bytes.

**Fix:** use a shorter key.

## AccessControlListNotSupported

**Cause:** an upload with an ACL that grants anyone else (`--acl public-read`, or any
`x-amz-grant-*` header); only `private` and `bucket-owner-full-control` are accepted.

**Fix:** leave out the ACL; publish with a bucket policy (see [Make objects public](make-objects-public.md)).

## NotImplemented on CreateBucket or DeleteBucket

**Cause:** buckets are not created or deleted over S3.

**Fix:** use the console, `dcloud s3 buckets create` or `dcloud s3 buckets delete`. For rclone,
set `no_check_bucket = true`.

## KMS.DisabledException

**Cause:** the bucket's KMS key is disabled. Disabling a key stops its buckets within 30 seconds.

**Fix:** enable the key; reads and writes work again with the next request.

## InsufficientStorage (507)

**Cause:** on a storage cluster, its fullest building reached its fill limit; on your own hosts,
the store is 95 % full.

**Fix:** delete objects (reads and deletes still work); ask for more capacity.

## QuotaExceeded (403) on own storage

**Cause:** your organization reached its `s3.stored-bytes` quota.

**Fix:** delete objects, or ask for a higher quota (see [Quotas](quotas.md)).

## ServiceUnavailable (503) on a write

**Cause:** a building of the storage cluster is down or cut off (`Retry-After: 10`), the cluster
is unavailable, or your hosts' object store is down.

**Fix:** retry with backoff; every S3 SDK does.

## SlowDown (503)

**Cause:** the object store kept asking to slow down after the gateway's own retries.

**Fix:** retry with backoff.

## Versioning or lifecycle is refused with 409 on a storage-cluster bucket

**Cause:** the bucket shares storage with others and already holds objects.

**Fix:** set it on an empty bucket, or copy the objects into a new bucket and set it there first
(see [Storage locations](storage-locations.md#versioning-and-lifecycle-on-a-storage-cluster)).

## A bucket shows entries storage cannot read

**Cause:** rarely, the storage underneath keeps listing an object whose data is gone, and every
request for it fails. The bucket shows "_N_ entries in this bucket cannot be
read or deleted." A delete that hits such an entry is never reported as done.

**Fix:** none on your side. Deutsche Cloud is notified and repairs the bucket by moving everything
readable to fresh storage; until then it cannot be deleted. Nothing else in it is affected.

## Related

- [S3 compatibility](s3-compatibility.md)
