Jak mogę podzielić plik w pythonie?

Question

Czy można podzielić plik? Na przykład masz ogromną listę słów, chcę ją podzielić tak, aby stała się ona więcej niż jednym plikiem. Jak to jest możliwe?

Answer 1

Oczywiście jest to możliwe:

open input file 
open output file 1 
count = 0 
for each line in file: 
    write to output file 
    count = count + 1 
    if count > maxlines: 
     close output file 
     open next output file 
     count = 0 

Answer 2

Ten plik dzieli się znakami nowej linii i zapisuje go z powrotem na zewnątrz. Możesz łatwo zmienić ogranicznik. Może to również obsługiwać nierówne ilości, jeśli nie masz wielu linii splitLen (20 w tym przykładzie) w pliku wejściowym.

splitLen = 20   # 20 lines per file 
outputBase = 'output' # output.1.txt, output.2.txt, etc. 

# This is shorthand and not friendly with memory 
# on very large files (Sean Cavanagh), but it works. 
input = open('input.txt', 'r').read().split('\n') 

at = 1 
for lines in range(0, len(input), splitLen): 
    # First, get the list slice 
    outputData = input[lines:lines+splitLen] 

    # Now open the output file, join the new slice with newlines 
    # and write it out. Then close the file. 
    output = open(outputBase + str(at) + '.txt', 'w') 
    output.write('\n'.join(outputData)) 
    output.close() 

    # Increment the counter 
    at += 1 

Answer 3

rozwiązanie do dzielenia plików binarnych na rozdziały .000, .001, itp .:

FILE = 'scons-conversion.7z' 

MAX = 500*1024*1024 # 500Mb - max chapter size 
BUF = 50*1024*1024*1024 # 50GB - memory buffer size 

chapters = 0 
uglybuf = '' 
with open(FILE, 'rb') as src: 
    while True: 
    tgt = open(FILE + '.%03d' % chapters, 'wb') 
    written = 0 
    while written < MAX: 
     if len(uglybuf) > 0: 
     tgt.write(uglybuf) 
     tgt.write(src.read(min(BUF, MAX - written))) 
     written += min(BUF, MAX - written) 
     uglybuf = src.read(1) 
     if len(uglybuf) == 0: 
     break 
    tgt.close() 
    if len(uglybuf) == 0: 
     break 
    chapters += 1 

Answer 4

Lepszym pętla na przykład SLI, a nie wyginanie pamięci:

splitLen = 20   # 20 lines per file 
outputBase = 'output' # output.1.txt, output.2.txt, etc. 

input = open('input.txt', 'r') 

count = 0 
at = 0 
dest = None 
for line in input: 
    if count % splitLen == 0: 
     if dest: dest.close() 
     dest = open(outputBase + str(at) + '.txt', 'w') 
     at += 1 
    dest.write(line) 
    count += 1 

Answer 5

def split_file(file, prefix, max_size, buffer=1024): 
    """ 
    file: the input file 
    prefix: prefix of the output files that will be created 
    max_size: maximum size of each created file in bytes 
    buffer: buffer size in bytes 

    Returns the number of parts created. 
    """ 
    with open(file, 'r+b') as src: 
     suffix = 0 
     while True: 
      with open(prefix + '.%s' % suffix, 'w+b') as tgt: 
       written = 0 
       while written < max_size: 
        data = src.read(buffer) 
        if data: 
         tgt.write(data) 
         written += buffer 
        else: 
         return suffix 
       suffix += 1 


def cat_files(infiles, outfile, buffer=1024): 
    """ 
    infiles: a list of files 
    outfile: the file that will be created 
    buffer: buffer size in bytes 
    """ 
    with open(outfile, 'w+b') as tgt: 
     for infile in sorted(infiles): 
      with open(infile, 'r+b') as src: 
       while True: 
        data = src.read(buffer) 
        if data: 
         tgt.write(data) 
        else: 
         break 

Answer 6

import re 
PATENTS = 'patent.data' 

def split_file(filename): 
    # Open file to read 
    with open(filename, "r") as r: 

     # Counter 
     n=0 

     # Start reading file line by line 
     for i, line in enumerate(r): 

      # If line match with teplate -- <?xml --increase counter n 
      if re.match(r'\<\?xml', line): 
       n+=1 

       # This "if" can be deleted, without it will start naming from 1 
       # or you can keep it. It depends where is "re" will find at 
       # first time the template. In my case it was first line 
       if i == 0: 
        n = 0    

      # Write lines to file  
      with open("{}-{}".format(PATENTS, n), "a") as f: 
       f.write(line) 

split_file(PATENTS) 

W rezultacie otrzymasz:

patent.data-0

patent.data-1

patent.data-N

Answer 7

Można używać Użyj tej PyPI moduł filesplit.