Bucket policy

Bucket policies can be used to make buckets partially or completely public, they can also be used to limit access to public buckets. They can be used as some kind of firewall on your data.

Our Object store S3 API is provided by Ceph's Rados Gateway. Its supported policies are a subset of the AWS bucket policies.

Making a bucket public on a specific IP address range.

We will use the aws cli set policies. Make sure that is it properly configured for Juno.

This policy.json example will let objects from the bucket <private bucket> in project <project id> be listed (s3:ListBucket) and read (s3:GetObject) from the public IP < open IP> without the need for identification.

File : policy.json

{
    "Version": "2012-10-17",
    "Id": "S3-allow-from-my-ip",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3::<project id>:<private bucket>/*",
                "arn:aws:s3::<project id>:<private bucket>"
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "<open IP>/32"
                }
            }
        }
    ]
}

The policy on projects <project> is applied like this:

Get the local IP, this is the floating IP of a VM I have deployed on my project, but it could be any public IP or range.

$curl -4 ifconfig.co
198.168.189.175

Check that the bucket is private, it returns AccessDenied in a xml file.

$curl  https://objets.juno.calculquebec.ca/ad99d6c3087041bcb6c0fe5f2da54df9:truite/
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message></Message><BucketName>truite</BucketName><RequestId>tx00000a0656342bf1c6a6f-0069862e7e-122190171-default</RequestId><HostId>122190171-default-default</HostId></Error>

The xml formal is a pain we all have to deal with from time to time. You can install the yq cli to you environement to make them readable. The cli comes with yq a yaml parser but also with xq, an xml parser.

$pip install yq
$curl -s    https://objets.juno.calculquebec.ca/ad99d6c3087041bcb6c0fe5f2da54df9:truite  | xq  
{
  "Error": {
    "Code": "AccessDenied",
    "Message": null,
    "BucketName": "truite",
    "RequestId": "tx00000b4182e1d6e9932f2-0069863de1-122190231-default",
    "HostId": "122190231-default-default"
  }
}

Now, lets apply the policy and see if we can list the bucket and get the data

$aws --profile po-test s3api  put-bucket-policy --policy file://policy.json --bucket truite
$curl -s   https://objets.juno.calculquebec.ca/ad99d6c3087041bcb6c0fe5f2da54df9:truite  | xq .ListBucketResult.Contents.[].Key
"package.json/package.json"
"testdir/"
"testdir/package.json"
"vagabon.png"

For good mesure we go on another machine to make sure that the data is not available form there:

$curl -4 ifconfig.co
132.219.138.77
$curl -s   https://objets.juno.calculquebec.ca/ad99d6c3087041bcb6c0fe5f2da54df9:truite  | xq
{
  "Error": {
    "Code": "AccessDenied",
    "Message": null,
    "BucketName": "truite",
    "RequestId": "tx00000f13e03ed77f374bd-0069863f2b-122190379-default",
    "HostId": "122190379-default-default"
  }
}