I have a 300 GB text file that contains genomics data with over 250k records. There are some records with bad data and our genomics program ‘Popoolution’ allows us to comment out the “bad” records with an asterisk. Our problem is that we cannot find a text editor that will load the data so that we can comment out the bad records. Any suggestions? We have both Windows and Linux boxes.
UPDATE: More information
The program Popoolution (https://code.google.com/p/popoolation/) crashes when it reaches a “bad” record giving us the line number that we can then comment out. Specifically, we get a message from Perl that says “F#€%& Scaffolding”. The manual suggests we can just use an asterisk to comment out the bad line. Sadly, we will have to repeat this process many times…
One more thought… Is there an approach that would allow us to add the asterisk to the line without opening the entire text file at once. This could be very useful given that we will have to repeat the process an unknown number of times.
Advertisement
Answer
Based on your update:
One more thought… Is there an approach that would allow us to add the asterisk to the line without opening the entire text file at once. This could be very useful given that we will have to repeat the process an unknown number of times.
Here you have an approach: If you know the line number, you can add an asterisk in the beginning of that line saying:
sed 'LINE_NUMBER s/^/*/' file
See an example:
$ cat file aa bb cc dd ee $ sed '3 s/^/*/' file aa bb *cc dd ee
If you add -i, the file will be updated:
$ sed -i '3 s/^/*/' file $ cat file aa bb *cc dd ee
Even though I always think it’s better to do a redirection to another file
sed '3 s/^/*/' file > new_file
so that you keep intact your original file and save the updated one in new_file.