How to gather IP and User Agent info and uniq them base IP address from nginx access log with AWK?

i have a sample log file

27.151.49.215 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1977 "http://localhost/" "Mozilla/5.0 (iPhone 6p; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6.1 Mobile/12B411 Safari/8536.25"
49.73.31.190 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1813 "http://localhost/" "Mozilla/5.0 (iPhone 5SGLOBAL; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6 Mobile/13B143 Safari/8536.25"
114.80.188.61 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 165 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
223.64.63.228 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2068 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13E238 QQ/6.2.0.427 Pixel/1080 NetType/WIFI Mem/48"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
221.204.176.30 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 165 "http://localhost/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
222.77.208.177 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2621 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Mobile/11D169 rabbit%2F1.0 baiduboxapp/0_0.0.8.6_enohpi_069_046/1.7_1C2%253enohPi/1099a/82840F498905C55D0EB7EBB0CF5DDC44BAF811E8FFCCOABNILE/1"
221.3.134.130 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1962 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1"
123.157.71.167 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2069 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13E238 UCBrowser/10.9.13.779 Mobile"
39.187.201.169 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2621 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13D15 UCBrowser/10.9.14.779 Mobile"

I want to collect all the ip and user agent info to a file, and uniq the same IP address, how can i do with awk ?

output like:

4 101.251.3.75 "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
1 27.151.49.215 "Mozilla/5.0 (iPhone 6p; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6.1 Mobile/12B411 Safari/8536.25"
1 49.73.31.190 "Mozilla/5.0 (iPhone 5SGLOBAL; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6 Mobile/13B143 Safari/8536.25"
1 114.80.188.61 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
1 223.64.63.228 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13E238 QQ/6.2.0.427 Pixel/1080 NetType/WIFI Mem/48"
1 221.204.176.30 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
1 22.77.208.177 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Mobile/11D169 rabbit%2F1.0 baiduboxapp/0_0.0.8.6_enohpi_069_046/1.7_1C2%253enohPi/1099a/82840F498905C55D0EB7EBB0CF5DDC44BAF811E8FFCCOABNILE/1"
1 221.3.134.130 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1"
1 123.157.71.167 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13E238 UCBrowser/10.9.13.779 Mobile"
1 39.187.201.169 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13D15 UCBrowser/10.9.14.779 Mobile"

Answer

With sed, sort and uniq :

sed 's/([^ ]*).* ("[^"]*")/1 2/' file | sort | uniq -c

Advertisement

Answer