Reorganize files on the server

Useful commands to use when reorganizing large datasets to be prepped to be uploaded using datapack

This is run in the terminal

Basics

Moving around

cd directory_name

Go up one directory using ..

cd ..

Get the contents of the current directory

ls

Get the directory structure for this folder

• -d flag shows only the directories

tree

tree -d

Rename file based on file path

For example you have some files organized like this:

../../visitor/Lastname
└── RU_ALN_TR1_FL007R
│   └── rgb_images
|       └──image.png
└── RU_ALN_TR1_FL008B 
    └── micasense_preflight_calibration_images
        └──image.png

We want to rename the files.

1. Find all the files with the same file name

find . -name 'image.png' 

2. First extract the file name in parts ((.*)/(.*)/(.*)/(.*)) by using the. / as a separator.

• Each part is represented by $#. If you want the last part of the file name:$4. Rearrange this section as needed:$2/$3/$2-$3-$4
• Set the flag as -n while testing to see the result of the renaming first before committing to it using -v

rename -n -- 's|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|'

rename -v -- 's|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|'

3. Putting it together by taking the results from find and renaming those files

• | the pipe is like the %>% in R

find . -name 'image.png' | rename -v -- 's|(.*)/(.*)/(.*)/(.*)$|$2/$3/$2-$3-$4|'

Unzip files

unzip '*zip'

Zip shapefiles

1. For all the files in a directory, create a folder and move those files

for file in *; do dir=$(echo $file | cut -d. -f1); mkdir -p $dir; mv $file $dir; done

2. Zip the folders

for file in *;do zip -r $file $file; done

3. Remove the directories after zipping

-r means to recursively remove files

rm -r */