I have a csv with nearly a hundred of column.
I am writing a bash script to get only few of those column.
CSV:
0.132.XXXXX.1456836300,1456836300,1456814505,900,XXXXXX,762,0.1,3699244,363811201,3025019,482119186,3699204,363808801,3025019,482119186,0,0,0,0,3699153,194638692,3025031,301505084,886378,109026314,2812835,33759500,2138713,85552320,3025031,301505084,3699096,363753480,3699096,363753480,3699096,434036304,3699153,194638692,3699153,283418364,0,0,0,0,0,0,0,0,4291150,4970942734,4286035,4817376889,4291126,4970941294,4286035,4817376889,0,0,4898811554,4286035,4286035,4817376889,4286035,4697533512,4286035,4594668672,0,0,0,0,0,0,4291120,4700737486,4286035,4594668672,4291120,4700737486,3405068,4421499690,3399983,4418549966,4286035,4594668672,4291120,4970939408,0,0,0,0,0 0.132.YYYYYY.1456835400,1456835400,1456813605,900,YYYYYYYY,761,0.1,3642774,359185359,3002739,478806690,3642735,359183019,3002739,478806690,0,0,0,0,3642572,192544856,3002731,299528510,887300,109139720,2755332,33069464,2115491,84623440,3002731,299528510,3642629,359128287,3642629,359128287,3642629,428338238,3642572,192544856,3642572,279966584,0,0,0,0,0,0,0,0,4263398,4930723176,4259146,4779228782,4263375,4930721796,4259146,4779228782,0,0,4860152556,4259146,4259146,4779228782,4259126,4660142239,4259126,4557923215,0,0,0,0,0,0,4263381,4662299857,4259126,4557923215,4263381,4662299857,3376436,4384383822,3372181,4382438954,4259126,4557923215,4263369,4930719910,0,0,0,0,0
I did some research and found this:
while IFS="," read $col1 $col2 ... do echo "$col1|$col2" done < /home/xxx/Desktop/data.csv
But it requires to put ALL the column (92 in my case…) I can’t select want I want. So I try another solution:
cat /home/xxx/Desktop/data.csv | while read line do a=[ cut -d',' -f5 $line]; # get the source b=[ cut -d',' -f3 $line]; # get the received time echo "$a . $b" done
It cannot work because of the size of the $line. I get the error name too long or something like that. (and it does not look really optimized but anyway) I guess I am close to it. Any help?
Advertisement
Answer
You can use just cut
:
cut -d, -f3,18 --output-delimiter='|' data.csv
Note this only works if the comma is never escaped or quoted in the CSV, and no value can contain a newline.
To iterate over the lines and populate variables, you can do
cut -d, -f3,18 --output-delimiter=' ' data.csv | while read col3 col18 ; do echo "$col3 | $col18" done