1 Minute Tips

日: 2022年2月25日

  • About the concept of S3 directories

    Introduction

    Most of the tasks in aws can be operated with aws commands. The only command I couldn’t find in the official documentation was “create folder,” which should be on the screen.

    The concept of a folder in S3

    In fact, the concept of folders does not exist in S3. https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/UsingMetadata.html

    The data model of Amazon S3 is a flat structure. Buckets are created and objects are stored in those buckets. There is no hierarchy of sub-buckets or sub-folders. However, a logical hierarchy can be implied by using key name prefixes and delimiters, such as those used in the Amazon S3 console.

    So, for one bucket, there are no folders in it, and it is an image of a large number of files.

    So what do we do?

    I don’t have a directory, but I’ll just dump it in directly!

    aws s3 cp sample.txt s3://test/sample-folder/


    This will create a sample-folder in the test bucket. Copy the sample.txt file into it. Reference: Breaking the illusion of “folder” in Amazon S3 and revealing its reality https://dev.classmethod.jp/cloud/aws/amazon-s3-folders/

    If multiple environments are registered in the .aws directory, it is possible to specify the profile to be loaded with -profile role-a-profile.

    If you want to copy the whole directory, use

    --recursive
    option to copy the entire directory recursively.

  • What to do when /Var/Lib/Docker/Overlay2 gets bloated in Docker

    Introduction

    I ran out of storage on my docker build server. I need to delete unnecessary files, but there is a command to safely allocate space.

    What to do first

    grasp of the situation

    Identify what is putting pressure on the disk in docker.

    Use docker system df command, you can identify the disk usage related to docker.

    # docker system df
TYPE                    TOTAL      ACTIVE     SIZE       RECLAIMABLE
Images               9              1              2.014GB   1.962GB (97%)
Containers           2              0              0B              0B
Local Volumes        4              2              824.6MB   781.2MB (94%)
Build Cache          0              0              0B              0B

    cope

    It will remove stopped containers, unused volumes, unused networks, and unused images.

    docker system prune

    However, the option to delete unused volumes and images is required.

    # docker system prune --help

Usage:	docker system prune [OPTIONS]

Remove unused data

Options:
  -a, --all             Remove all unused images not just dangling ones
      --filter filter   Provide filter values (e.g. 'label=<key>=<value>')
  -f, --force           Do not prompt for confirmation
      --volumes         Prune volumes

    Delete Volumes

    I was able to reduce the size of the file by 355.4MB.

    553d15cd6b78ad0f21788a22e9ce16a7295c4bab97609973
deleted: sha256:7ae9338aed73a0f33568db53740431038d3a1f779c4dae40d27433984e1cd97c
deleted: sha256:b1be54c8cadff1e50b87b93559320a1ae57b8d0dd326507148f7ca81d707beed
deleted: sha256:86d78f10b9718618eaae056f5dfa1edae518949aee4578e4147268e9db2e75f0

Total reclaimed space: 355.4MB

    However, this does not remove the storage.

    use –volumes optiion. you can also delete storage.

    docker system prune --volumes
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all volumes not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y

    Deleting Images

    If you choose –all option. you can also delete images.

    # docker system prune --all
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N]

    After work

    When you finish deleting unnecessary files, it will be zero.

    # docker system df
TYPE    TOTAL   ACTIVE  SIZE    RECLAIMABLE
Images  0    0    0B   0B
Containers    0    0    0B   0B
Local Volumes 0    0    0B   0B
Build Cache   0    0    0B   0B