Lectura de archivos de texto línea por línea, con informe de desplazamiento/posición exacta

Question

Mi requisito simple: leer un enorme archivo de prueba de línea (> un millón) (en este ejemplo supongo que es un CSV de algún tipo) y mantener una referencia al principio de esa línea para una búsqueda más rápida en el futuro (lea una línea, comenzando en X).

me trataron de la manera ingenua y sencilla en primer lugar, el uso de un StreamWriter y acceder a la subyacente BaseStream.Position. Por desgracia eso no funciona como era mi intención:

dado un archivo que contiene la siguiente

Foo 
Bar 
Baz 
Bla 
Fasel 

y este código muy simple

using (var sr = new StreamReader(@"C:\Temp\LineTest.txt")) { 
    string line; 
    long pos = sr.BaseStream.Position; 
    while ((line = sr.ReadLine()) != null) { 
    Console.Write("{0:d3} ", pos); 
    Console.WriteLine(line); 
    pos = sr.BaseStream.Position; 
    } 
} 

la salida es:

000 Foo 
025 Bar 
025 Baz 
025 Bla 
025 Fasel 

Me imagino que la transmisión intenta ser útil/eficiente y, probablemente, se lee en partes (grandes) cuando n nuevos datos son necesarios. Para mí esto es malo ...

La pregunta, finalmente: Cualquier forma de obtener el byte, char) compensar mientras lee un archivo línea por línea sin usar un flujo básico y jugando con \ r \ n \ r \ n y la codificación de cadena, etc. manualmente? No es gran cosa, realmente, simplemente no me gusta construir cosas que pudieran existir ya ..

Answer 1

Se puede crear un envoltorio TextReader, lo que realizar un seguimiento de la posición actual en la base TextReader:

public class TrackingTextReader : TextReader 
{ 
    private TextReader _baseReader; 
    private int _position; 

    public TrackingTextReader(TextReader baseReader) 
    { 
     _baseReader = baseReader; 
    } 

    public override int Read() 
    { 
     _position++; 
     return _baseReader.Read(); 
    } 

    public override int Peek() 
    { 
     return _baseReader.Peek(); 
    } 

    public int Position 
    { 
     get { return _position; } 
    } 
} 

a continuación, puede utilizar la siguiente manera:

string text = @"Foo 
Bar 
Baz 
Bla 
Fasel"; 

using (var reader = new StringReader(text)) 
using (var trackingReader = new TrackingTextReader(reader)) 
{ 
    string line; 
    while ((line = trackingReader.ReadLine()) != null) 
    { 
     Console.WriteLine("{0:d3} {1}", trackingReader.Position, line); 
    } 
} 

Answer 2

Que este trabajo:

using (var sr = new StreamReader(@"C:\Temp\LineTest.txt")) { 
    string line; 
    long pos = 0; 
    while ((line = sr.ReadLine()) != null) { 
    Console.Write("{0:d3} ", pos); 
    Console.WriteLine(line); 
    pos += line.Length; 
    } 
} 

Answer 3

Aunque la solución de Thomas Levesque funciona bien, aquí está la mía. Utiliza la reflexión por lo que será más lento, pero es independiente de la codificación. Además, agregué la extensión de búsqueda también.

/// <summary>Useful <see cref="StreamReader"/> extentions.</summary> 
public static class StreamReaderExtentions 
{ 
    /// <summary>Gets the position within the <see cref="StreamReader.BaseStream"/> of the <see cref="StreamReader"/>.</summary> 
    /// <remarks><para>This method is quite slow. It uses reflection to access private <see cref="StreamReader"/> fields. Don't use it too often.</para></remarks> 
    /// <param name="streamReader">Source <see cref="StreamReader"/>.</param> 
    /// <exception cref="ArgumentNullException">Occurs when passed <see cref="StreamReader"/> is null.</exception> 
    /// <returns>The current position of this stream.</returns> 
    public static long GetPosition(this StreamReader streamReader) 
    { 
     if (streamReader == null) 
      throw new ArgumentNullException("streamReader"); 

     var charBuffer = (char[])streamReader.GetType().InvokeMember("charBuffer", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null); 
     var charPos = (int)streamReader.GetType().InvokeMember("charPos", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null); 
     var charLen = (int)streamReader.GetType().InvokeMember("charLen", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null); 

     var offsetLength = streamReader.CurrentEncoding.GetByteCount(charBuffer, charPos, charLen - charPos); 

     return streamReader.BaseStream.Position - offsetLength; 
    } 

    /// <summary>Sets the position within the <see cref="StreamReader.BaseStream"/> of the <see cref="StreamReader"/>.</summary> 
    /// <remarks> 
    /// <para><see cref="StreamReader.BaseStream"/> should be seekable.</para> 
    /// <para>This method is quite slow. It uses reflection and flushes the charBuffer of the <see cref="StreamReader.BaseStream"/>. Don't use it too often.</para> 
    /// </remarks> 
    /// <param name="streamReader">Source <see cref="StreamReader"/>.</param> 
    /// <param name="position">The point relative to origin from which to begin seeking.</param> 
    /// <param name="origin">Specifies the beginning, the end, or the current position as a reference point for origin, using a value of type <see cref="SeekOrigin"/>. </param> 
    /// <exception cref="ArgumentNullException">Occurs when passed <see cref="StreamReader"/> is null.</exception> 
    /// <exception cref="ArgumentException">Occurs when <see cref="StreamReader.BaseStream"/> is not seekable.</exception> 
    /// <returns>The new position in the stream. This position can be different to the <see cref="position"/> because of the preamble.</returns> 
    public static long Seek(this StreamReader streamReader, long position, SeekOrigin origin) 
    { 
     if (streamReader == null) 
      throw new ArgumentNullException("streamReader"); 

     if (!streamReader.BaseStream.CanSeek) 
      throw new ArgumentException("Underlying stream should be seekable.", "streamReader"); 

     var preamble = (byte[])streamReader.GetType().InvokeMember("_preamble", BindingFlags.DeclaredOnly | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField, null, streamReader, null); 
     if (preamble.Length > 0 && position < preamble.Length) // preamble or BOM must be skipped 
      position += preamble.Length; 

     var newPosition = streamReader.BaseStream.Seek(position, origin); // seek 
     streamReader.DiscardBufferedData(); // this updates the buffer 

     return newPosition; 
    } 
} 

Answer 4

Después de buscar, probar y hacer algo loco, está mi código para resolver (actualmente estoy usando este código en mi producto).

public sealed class TextFileReader : IDisposable 
{ 

    FileStream _fileStream = null; 
    BinaryReader _binReader = null; 
    StreamReader _streamReader = null; 
    List<string> _lines = null; 
    long _length = -1; 

    /// <summary> 
    /// Initializes a new instance of the <see cref="TextFileReader"/> class with default encoding (UTF8). 
    /// </summary> 
    /// <param name="filePath">The path to text file.</param> 
    public TextFileReader(string filePath) : this(filePath, Encoding.UTF8) { } 

    /// <summary> 
    /// Initializes a new instance of the <see cref="TextFileReader"/> class. 
    /// </summary> 
    /// <param name="filePath">The path to text file.</param> 
    /// <param name="encoding">The encoding of text file.</param> 
    public TextFileReader(string filePath, Encoding encoding) 
    { 
     if (!File.Exists(filePath)) 
      throw new FileNotFoundException("File (" + filePath + ") is not found."); 

     _fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read); 
     _length = _fileStream.Length; 
     _binReader = new BinaryReader(_fileStream, encoding); 
    } 

    /// <summary> 
    /// Reads a line of characters from the current stream at the current position and returns the data as a string. 
    /// </summary> 
    /// <returns>The next line from the input stream, or null if the end of the input stream is reached</returns> 
    public string ReadLine() 
    { 
     if (_binReader.PeekChar() == -1) 
      return null; 

     string line = ""; 
     int nextChar = _binReader.Read(); 
     while (nextChar != -1) 
     { 
      char current = (char)nextChar; 
      if (current.Equals('\n')) 
       break; 
      else if (current.Equals('\r')) 
      { 
       int pickChar = _binReader.PeekChar(); 
       if (pickChar != -1 && ((char)pickChar).Equals('\n')) 
        nextChar = _binReader.Read(); 
       break; 
      } 
      else 
       line += current; 
      nextChar = _binReader.Read(); 
     } 
     return line; 
    } 

    /// <summary> 
    /// Reads some lines of characters from the current stream at the current position and returns the data as a collection of string. 
    /// </summary> 
    /// <param name="totalLines">The total number of lines to read (set as 0 to read from current position to end of file).</param> 
    /// <returns>The next lines from the input stream, or empty collectoin if the end of the input stream is reached</returns> 
    public List<string> ReadLines(int totalLines) 
    { 
     if (totalLines < 1 && this.Position == 0) 
      return this.ReadAllLines(); 

     _lines = new List<string>(); 
     int counter = 0; 
     string line = this.ReadLine(); 
     while (line != null) 
     { 
      _lines.Add(line); 
      counter++; 
      if (totalLines > 0 && counter >= totalLines) 
       break; 
      line = this.ReadLine(); 
     } 
     return _lines; 
    } 

    /// <summary> 
    /// Reads all lines of characters from the current stream (from the begin to end) and returns the data as a collection of string. 
    /// </summary> 
    /// <returns>The next lines from the input stream, or empty collectoin if the end of the input stream is reached</returns> 
    public List<string> ReadAllLines() 
    { 
     if (_streamReader == null) 
      _streamReader = new StreamReader(_fileStream); 
     _streamReader.BaseStream.Seek(0, SeekOrigin.Begin); 
     _lines = new List<string>(); 
     string line = _streamReader.ReadLine(); 
     while (line != null) 
     { 
      _lines.Add(line); 
      line = _streamReader.ReadLine(); 
     } 
     return _lines; 
    } 

    /// <summary> 
    /// Gets the length of text file (in bytes). 
    /// </summary> 
    public long Length 
    { 
     get { return _length; } 
    } 

    /// <summary> 
    /// Gets or sets the current reading position. 
    /// </summary> 
    public long Position 
    { 
     get 
     { 
      if (_binReader == null) 
       return -1; 
      else 
       return _binReader.BaseStream.Position; 
     } 
     set 
     { 
      if (_binReader == null) 
       return; 
      else if (value >= this.Length) 
       this.SetPosition(this.Length); 
      else 
       this.SetPosition(value); 
     } 
    } 

    void SetPosition(long position) 
    { 
     _binReader.BaseStream.Seek(position, SeekOrigin.Begin); 
    } 

    /// <summary> 
    /// Gets the lines after reading. 
    /// </summary> 
    public List<string> Lines 
    { 
     get 
     { 
      return _lines; 
     } 
    } 

    /// <summary> 
    /// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources. 
    /// </summary> 
    public void Dispose() 
    { 
     if (_binReader != null) 
      _binReader.Close(); 
     if (_streamReader != null) 
     { 
      _streamReader.Close(); 
      _streamReader.Dispose(); 
     } 
     if (_fileStream != null) 
     { 
      _fileStream.Close(); 
      _fileStream.Dispose(); 
     } 
    } 

    ~TextFileReader() 
    { 
     this.Dispose(); 
    } 
} 

Answer 5

Este es un problema realmente difícil. Después de una enumeración muy larga y exhaustiva de diferentes soluciones en Internet (incluidas las soluciones de este hilo, ¡gracias!) Tuve que crear mi propia bicicleta.

Había siguientes requisitos:

• Rendimiento - lectura debe ser muy rápido, por lo que la lectura de un carbón de leña en el momento o utilizando la reflexión no son aceptables, por lo que se requiere de amortiguación
• Streaming - el archivo puede ser enorme, por lo que no es aceptable leerlo en la memoria por completo
• Enlazando - file tailing debería estar disponible
• líneas largas - líneas puede ser muy largo, por lo que el almacenamiento intermedio no se pueden limitar

• Estable - fue inmediatamente visible durante el uso de error de un solo byte. Por desgracia para mí, varias implementaciones que encontré fueron con problemas de estabilidad

  public class OffsetStreamReader 
  { 
      private const int InitialBufferSize = 4096;  
      private readonly char _bom; 
      private readonly byte _end; 
      private readonly Encoding _encoding; 
      private readonly Stream _stream; 
      private readonly bool _tail; 
  
      private byte[] _buffer; 
      private int _processedInBuffer; 
      private int _informationInBuffer; 
  
      public OffsetStreamReader(Stream stream, bool tail) 
      { 
       _buffer = new byte[InitialBufferSize]; 
       _processedInBuffer = InitialBufferSize; 
  
       if (stream == null || !stream.CanRead) 
        throw new ArgumentException("stream"); 
  
       _stream = stream; 
       _tail = tail; 
       _encoding = Encoding.UTF8; 
  
       _bom = '\uFEFF'; 
       _end = _encoding.GetBytes(new [] {'\n'})[0]; 
      } 
  
      public long Offset { get; private set; } 
  
      public string ReadLine() 
      { 
       // Underlying stream closed 
       if (!_stream.CanRead) 
        return null; 
  
       // EOF 
       if (_processedInBuffer == _informationInBuffer) 
       { 
        if (_tail) 
        { 
         _processedInBuffer = _buffer.Length; 
         _informationInBuffer = 0; 
         ReadBuffer(); 
        } 
  
        return null; 
       } 
  
       var lineEnd = Search(_buffer, _end, _processedInBuffer); 
       var haveEnd = true; 
  
       // File ended but no finalizing newline character 
       if (lineEnd.HasValue == false && _informationInBuffer + _processedInBuffer < _buffer.Length) 
       { 
        if (_tail) 
         return null; 
        else 
        { 
         lineEnd = _informationInBuffer; 
         haveEnd = false; 
        } 
       } 
  
       // No end in current buffer 
       if (!lineEnd.HasValue) 
       { 
        ReadBuffer(); 
        if (_informationInBuffer != 0) 
         return ReadLine(); 
  
        return null; 
       } 
  
       var arr = new byte[lineEnd.Value - _processedInBuffer]; 
       Array.Copy(_buffer, _processedInBuffer, arr, 0, arr.Length); 
  
       Offset = Offset + lineEnd.Value - _processedInBuffer + (haveEnd ? 1 : 0); 
       _processedInBuffer = lineEnd.Value + (haveEnd ? 1 : 0); 
  
       return _encoding.GetString(arr).TrimStart(_bom).TrimEnd('\r', '\n'); 
      } 
  
      private void ReadBuffer() 
      { 
       var notProcessedPartLength = _buffer.Length - _processedInBuffer; 
  
       // Extend buffer to be able to fit whole line to the buffer 
       // Was  [NOT_PROCESSED] 
       // Become [NOT_PROCESSED  ] 
       if (notProcessedPartLength == _buffer.Length) 
       { 
        var extendedBuffer = new byte[_buffer.Length + _buffer.Length/2]; 
        Array.Copy(_buffer, extendedBuffer, _buffer.Length); 
        _buffer = extendedBuffer; 
       } 
  
       // Copy not processed information to the begining 
       // Was [PROCESSED NOT_PROCESSED] 
       // Become [NOT_PROCESSED   ] 
       Array.Copy(_buffer, (long) _processedInBuffer, _buffer, 0, notProcessedPartLength); 
  
       // Read more information to the empty part of buffer 
       // Was [ NOT_PROCESSED     ] 
       // Become [ NOT_PROCESSED NEW_NOT_PROCESSED ] 
       _informationInBuffer = notProcessedPartLength + _stream.Read(_buffer, notProcessedPartLength, _buffer.Length - notProcessedPartLength); 
  
       _processedInBuffer = 0; 
      } 
  
      private int? Search(byte[] buffer, byte byteToSearch, int bufferOffset) 
      { 
       for (int i = bufferOffset; i < buffer.Length - 1; i++) 
       { 
        if (buffer[i] == byteToSearch) 
         return i; 
       } 
       return null; 
      } 
  }