Python : How to read data from multiple files present in a directory

Question

I have many files in a directory that I want to use as inputs. Should I combine them first and then read or is there any other way to read data from those files?

Answer 1

You can combine all files and can create one file for the processing, however, there is another way to read data from all files present in a directory. You can create a list of filenames and then open/read one file at a time for the processing. Here is an example I used for the same scenario:

>>> filelist = [fl for fl in next(os.walk('classifier_outputs'))[2] if 'metavisits_probs' in fl[:len('metavisits_probs')]]
>>> filelist
['metavisits_probs1.txt', 'metavisits_probs11.txt', 'metavisits_probs21.txt', 'metavisits_probs31.txt', 'metavisits_probs41.txt', 'metavisits_probs51.txt', 'metavisits_probs61.txt', 'metavisits_probs71.txt', 'metavisits_probs81.txt', 'metavisits_probs91.txt', 'metavisits_probs101.txt', 'metavisits_probs111.txt']
>>> for filename in filelist:
...     filepath = wv.metavisits_probs_file.split('/')[0] + '/' + filename
...     with open(filepath) as mpf:
...             for line in mpf:
...                     print(filepath)
...                     break
...
classifier_outputs/metavisits_probs1.txt
classifier_outputs/metavisits_probs11.txt
classifier_outputs/metavisits_probs21.txt
classifier_outputs/metavisits_probs31.txt
classifier_outputs/metavisits_probs41.txt
classifier_outputs/metavisits_probs51.txt
classifier_outputs/metavisits_probs61.txt
classifier_outputs/metavisits_probs71.txt
classifier_outputs/metavisits_probs81.txt
classifier_outputs/metavisits_probs91.txt
classifier_outputs/metavisits_probs101.txt
>>>

You need to use 'import os' and replace 'print(filepath)' and 'break' with your statements.