Microsoft.IO.RecyclableMemoryStream
Manages pools of RecyclableMemoryStream objects.
There are two pools managed in here. The small pool contains same-sized buffers that are handed to streams
as they write more data.
For scenarios that need to call GetBuffer(), the large pool contains buffers of various sizes, all
multiples/exponentials of LargeBufferMultiple (1 MB by default). They are split by size to avoid overly-wasteful buffer
usage. There should be far fewer 8 MB buffers than 1 MB buffers, for example.
ETW events for RecyclableMemoryStream
Static log object, through which all events are written.
Type of buffer
Small block buffer
Large pool buffer
The possible reasons for discarding a buffer
Buffer was too large to be re-pooled
There are enough free bytes in the pool
Logged when a stream object is created.
A unique ID for this stream.
A temporary ID for this stream, usually indicates current usage.
Requested size of the stream
Logged when the stream is disposed
A unique ID for this stream.
A temporary ID for this stream, usually indicates current usage.
Logged when the stream is disposed for the second time.
A unique ID for this stream.
A temporary ID for this stream, usually indicates current usage.
Call stack of initial allocation.
Call stack of the first dispose.
Call stack of the second dispose.
Note: Stacks will only be populated if RecyclableMemoryStreamManager.GenerateCallStacks is true.
Logged when a stream is finalized.
A unique ID for this stream.
A temporary ID for this stream, usually indicates current usage.
Call stack of initial allocation.
Note: Stacks will only be populated if RecyclableMemoryStreamManager.GenerateCallStacks is true.
Logged when ToArray is called on a stream.
A unique ID for this stream.
A temporary ID for this stream, usually indicates current usage.
Call stack of the ToArray call.
Length of stream
Note: Stacks will only be populated if RecyclableMemoryStreamManager.GenerateCallStacks is true.
Logged when the RecyclableMemoryStreamManager is initialized.
Size of blocks, in bytes.
Size of the large buffer multiple, in bytes.
Maximum buffer size, in bytes.
Logged when a new block is created.
Number of bytes in the small pool currently in use.
Logged when a new large buffer is created.
Requested size
Number of bytes in the large pool in use.
Logged when a buffer is created that is too large to pool.
Size requested by the caller
A temporary ID for this stream, usually indicates current usage.
Call stack of the requested stream.
Note: Stacks will only be populated if RecyclableMemoryStreamManager.GenerateCallStacks is true.
Logged when a buffer is discarded (not put back in the pool, but given to GC to clean up).
Type of the buffer being discarded.
A temporary ID for this stream, usually indicates current usage.
Reason for the discard.
Logged when a stream grows beyond the maximum capacity.
The requested capacity.
Maximum capacity, as configured by RecyclableMemoryStreamManager.
A temporary ID for this stream, usually indicates current usage.
Call stack for the capacity request.
Note: Stacks will only be populated if RecyclableMemoryStreamManager.GenerateCallStacks is true.
Generic delegate for handling events without any arguments.
Delegate for handling large buffer discard reports.
Reason the buffer was discarded.
Delegate for handling reports of stream size when streams are allocated
Bytes allocated.
Delegate for handling periodic reporting of memory use statistics.
Bytes currently in use in the small pool.
Bytes currently free in the small pool.
Bytes currently in use in the large pool.
Bytes currently free in the large pool.
Default block size, in bytes
Default large buffer multiple, in bytes
Default maximum buffer size, in bytes
Initializes the memory manager with the default block/buffer specifications.
Initializes the memory manager with the given block requiredSize.
Size of each block that is pooled. Must be > 0.
Each large buffer will be a multiple of this value.
Buffers larger than this are not pooled
blockSize is not a positive number, or largeBufferMultiple is not a positive number, or maximumBufferSize is less than blockSize.
maximumBufferSize is not a multiple of largeBufferMultiple
Initializes the memory manager with the given block requiredSize.
Size of each block that is pooled. Must be > 0.
Each large buffer will be a multiple/exponential of this value.
Buffers larger than this are not pooled
Switch to exponential large buffer allocation strategy
blockSize is not a positive number, or largeBufferMultiple is not a positive number, or maximumBufferSize is less than blockSize.
maximumBufferSize is not a multiple/exponential of largeBufferMultiple
The size of each block. It must be set at creation and cannot be changed.
All buffers are multiples/exponentials of this number. It must be set at creation and cannot be changed.
Use multiple large buffer allocation strategy. It must be set at creation and cannot be changed.
Use exponential large buffer allocation strategy. It must be set at creation and cannot be changed.
Gets the maximum buffer size.
Any buffer that is returned to the pool that is larger than this will be
discarded and garbage collected.
Number of bytes in small pool not currently in use
Number of bytes currently in use by stream from the small pool
Number of bytes in large pool not currently in use
Number of bytes currently in use by streams from the large pool
How many blocks are in the small pool
How many buffers are in the large pool
How many bytes of small free blocks to allow before we start dropping
those returned to us.
How many bytes of large free buffers to allow before we start dropping
those returned to us.
Maximum stream capacity in bytes. Attempts to set a larger capacity will
result in an exception.
A value of 0 indicates no limit.
Whether to save callstacks for stream allocations. This can help in debugging.
It should NEVER be turned on generally in production.
Whether dirty buffers can be immediately returned to the buffer pool. E.g. when GetBuffer() is called on
a stream and creates a single large buffer, if this setting is enabled, the other blocks will be returned
to the buffer pool immediately.
Note when enabling this setting that the user is responsible for ensuring that any buffer previously
retrieved from a stream which is subsequently modified is not used after modification (as it may no longer
be valid).
Causes an exception to be thrown if ToArray is ever called.
Calling ToArray defeats the purpose of a pooled buffer. Use this property to discover code that is calling ToArray. If this is
set and stream.ToArray() is called, a NotSupportedException will be thrown.
Removes and returns a single block from the pool.
A byte[] array
Returns a buffer of arbitrary size from the large buffer pool. This buffer
will be at least the requiredSize and always be a multiple/exponential of largeBufferMultiple.
The minimum length of the buffer
The tag of the stream returning this buffer, for logging if necessary.
A buffer of at least the required size.
Returns the buffer to the large pool
The buffer to return.
The tag of the stream returning this buffer, for logging if necessary.
buffer is null
buffer.Length is not a multiple/exponential of LargeBufferMultiple (it did not originate from this pool)
Returns the blocks to the pool
Collection of blocks to return to the pool
The tag of the stream returning these blocks, for logging if necessary.
blocks is null
blocks contains buffers that are the wrong size (or null) for this memory manager
Retrieve a new MemoryStream object with no tag and a default initial capacity.
A MemoryStream.
Retrieve a new MemoryStream object with no tag and a default initial capacity.
A unique identifier which can be used to trace usages of the stream.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and a default initial capacity.
A tag which can be used to track the source of the stream.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and a default initial capacity.
A unique identifier which can be used to trace usages of the stream.
A tag which can be used to track the source of the stream.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and at least the given capacity.
A tag which can be used to track the source of the stream.
The minimum desired capacity for the stream.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and at least the given capacity.
A unique identifier which can be used to trace usages of the stream.
A tag which can be used to track the source of the stream.
The minimum desired capacity for the stream.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and at least the given capacity, possibly using
a single contiguous underlying buffer.
Retrieving a MemoryStream which provides a single contiguous buffer can be useful in situations
where the initial size is known and it is desirable to avoid copying data between the smaller underlying
buffers to a single large one. This is most helpful when you know that you will always call GetBuffer
on the underlying stream.
A unique identifier which can be used to trace usages of the stream.
A tag which can be used to track the source of the stream.
The minimum desired capacity for the stream.
Whether to attempt to use a single contiguous buffer.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and at least the given capacity, possibly using
a single contiguous underlying buffer.
Retrieving a MemoryStream which provides a single contiguous buffer can be useful in situations
where the initial size is known and it is desirable to avoid copying data between the smaller underlying
buffers to a single large one. This is most helpful when you know that you will always call GetBuffer
on the underlying stream.
A tag which can be used to track the source of the stream.
The minimum desired capacity for the stream.
Whether to attempt to use a single contiguous buffer.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and with contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
A unique identifier which can be used to trace usages of the stream.
A tag which can be used to track the source of the stream.
The byte buffer to copy data from.
The offset from the start of the buffer to copy from.
The number of bytes to copy from the buffer.
A MemoryStream.
Retrieve a new MemoryStream object with the contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
The byte buffer to copy data from.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and with contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
A tag which can be used to track the source of the stream.
The byte buffer to copy data from.
The offset from the start of the buffer to copy from.
The number of bytes to copy from the buffer.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and with contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
A unique identifier which can be used to trace usages of the stream.
A tag which can be used to track the source of the stream.
The byte buffer to copy data from.
A MemoryStream.
Retrieve a new MemoryStream object with the contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
The byte buffer to copy data from.
A MemoryStream.
Retrieve a new MemoryStream object with the given tag and with contents copied from the provided
buffer. The provided buffer is not wrapped or used after construction.
The new stream's position is set to the beginning of the stream when returned.
A tag which can be used to track the source of the stream.
The byte buffer to copy data from.
A MemoryStream.
Triggered when a new block is created.
Triggered when a new block is created.
Triggered when a new large buffer is created.
Triggered when a new stream is created.
Triggered when a stream is disposed.
Triggered when a stream is finalized.
Triggered when a stream is finalized.
Triggered when a user converts a stream to array.
Triggered when a large buffer is discarded, along with the reason for the discard.
Periodically triggered to report usage statistics.
MemoryStream implementation that deals with pooling and managing memory streams which use potentially large
buffers.
This class works in tandem with the RecyclableMemoryStreamManager to supply MemoryStream
objects to callers, while avoiding these specific problems:
1. LOH allocations - since all large buffers are pooled, they will never incur a Gen2 GC
2. Memory waste - A standard memory stream doubles its size when it runs out of room. This
leads to continual memory growth as each stream approaches the maximum allowed size.
3. Memory copying - Each time a MemoryStream grows, all the bytes are copied into new buffers.
This implementation only copies the bytes when GetBuffer is called.
4. Memory fragmentation - By using homogeneous buffer sizes, it ensures that blocks of memory
can be easily reused.
The stream is implemented on top of a series of uniformly-sized blocks. As the stream's length grows,
additional blocks are retrieved from the memory manager. It is these blocks that are pooled, not the stream
object itself.
The biggest wrinkle in this implementation is when GetBuffer() is called. This requires a single
contiguous buffer. If only a single block is in use, then that block is returned. If multiple blocks
are in use, we retrieve a larger buffer from the memory manager. These large buffers are also pooled,
split by size--they are multiples/exponentials of a chunk size (1 MB by default).
Once a large buffer is assigned to the stream the small blocks are NEVER again used for this stream. All operations take place on the
large buffer. The large buffer can be replaced by a larger buffer from the pool as needed. All blocks and large buffers
are maintained in the stream until the stream is disposed (unless AggressiveBufferReturn is enabled in the stream manager).
All of these blocks must be the same size
This list is used to store buffers once they're replaced by something larger.
This is for the cases where you have users of this class that may hold onto the buffers longer
than they should and you want to prevent race conditions which could corrupt the data.
This is only set by GetBuffer() if the necessary buffer is larger than a single block size, or on
construction if the caller immediately requests a single large buffer.
If this field is non-null, it contains the concatenation of the bytes found in the individual
blocks. Once it is created, this (or a larger) largeBuffer will be used for the life of the stream.
Unique identifier for this stream across its entire lifetime
Object has been disposed
A temporary identifier for the current usage of this stream.
Object has been disposed
Gets the memory manager being used by this stream.
Object has been disposed
Callstack of the constructor. It is only set if MemoryManager.GenerateCallStacks is true,
which should only be in debugging situations.
Callstack of the Dispose call. It is only set if MemoryManager.GenerateCallStacks is true,
which should only be in debugging situations.
Allocate a new RecyclableMemoryStream object.
The memory manager
Allocate a new RecyclableMemoryStream object.
The memory manager
A unique identifier which can be used to trace usages of the stream.
Allocate a new RecyclableMemoryStream object
The memory manager
A string identifying this stream for logging and debugging purposes
Allocate a new RecyclableMemoryStream object
The memory manager
A unique identifier which can be used to trace usages of the stream.
A string identifying this stream for logging and debugging purposes
Allocate a new RecyclableMemoryStream object
The memory manager
A string identifying this stream for logging and debugging purposes
The initial requested size to prevent future allocations
Allocate a new RecyclableMemoryStream object
The memory manager
A unique identifier which can be used to trace usages of the stream.
A string identifying this stream for logging and debugging purposes
The initial requested size to prevent future allocations
Allocate a new RecyclableMemoryStream object
The memory manager
A unique identifier which can be used to trace usages of the stream.
A string identifying this stream for logging and debugging purposes
The initial requested size to prevent future allocations
An initial buffer to use. This buffer will be owned by the stream and returned to the memory manager upon Dispose.
The finalizer will be called when a stream is not disposed properly.
Failing to dispose indicates a bug in the code using streams. Care should be taken to properly account for stream lifetime.
Returns the memory used by this stream back to the pool.
Whether we're disposing (true), or being called by the finalizer (false)
Equivalent to Dispose
Gets or sets the capacity
Capacity is always in multiples of the memory manager's block size, unless
the large buffer is in use. Capacity never decreases during a stream's lifetime.
Explicitly setting the capacity to a lower value than the current value will have no effect.
This is because the buffers are all pooled by chunks and there's little reason to
allow stream truncation.
Writing past the current capacity will cause Capacity to automatically increase, until MaximumStreamCapacity is reached.
Object has been disposed
Gets the number of bytes written to this stream.
Object has been disposed
Gets the current position in the stream
Object has been disposed
Whether the stream can currently read
Whether the stream can currently seek
Always false
Whether the stream can currently write
Returns a single buffer containing the contents of the stream.
The buffer may be longer than the stream length.
A byte[] buffer
IMPORTANT: Doing a Write() after calling GetBuffer() invalidates the buffer. The old buffer is held onto
until Dispose is called, but the next time GetBuffer() is called, a new buffer from the pool will be required.
Object has been disposed
Asynchronously reads all the bytes from the current position in this stream and writes them to another stream.
The stream to which the contents of the current stream will be copied.
This parameter is ignored.
The token to monitor for cancellation requests.
A task that represents the asynchronous copy operation.
is .
Either the current stream or the destination stream is disposed.
The current stream does not support reading, or the destination stream does not support writing.
Returns a sequence containing the contents of the stream.
A ReadOnlySequence of bytes
IMPORTANT: Doing a Write(), Dispose(), or Close() after calling GetReadOnlySequence() invalidates the sequence.
Object has been disposed
Returns an ArraySegment that wraps a single buffer containing the contents of the stream.
An ArraySegment containing a reference to the underlying bytes.
Always returns true.
GetBuffer has no failure modes (it always returns something, even if it's an empty buffer), therefore this method
always returns a valid ArraySegment to the same buffer returned by GetBuffer.
Returns a new array with a copy of the buffer's contents. You should almost certainly be using GetBuffer combined with the Length to
access the bytes in this stream. Calling ToArray will destroy the benefits of pooled buffers, but it is included
for the sake of completeness.
Object has been disposed
The current RecyclableStreamManager object disallows ToArray calls.
Reads from the current position into the provided buffer
Destination buffer
Offset into buffer at which to start placing the read bytes.
Number of bytes to read.
The number of bytes read
buffer is null
offset or count is less than 0
offset subtracted from the buffer length is less than count
Object has been disposed
Reads from the specified position into the provided buffer
Destination buffer
Offset into buffer at which to start placing the read bytes.
Number of bytes to read.
Position in the stream to start reading from
The number of bytes read
buffer is null
offset or count is less than 0
offset subtracted from the buffer length is less than count
Object has been disposed
Reads from the current position into the provided buffer
Destination buffer
The number of bytes read
Object has been disposed
Reads from the specified position into the provided buffer
Destination buffer
Position in the stream to start reading from
The number of bytes read
Object has been disposed
Writes the buffer to the stream
Source buffer
Start position
Number of bytes to write
buffer is null
offset or count is negative
buffer.Length - offset is not less than count
Object has been disposed
Writes the buffer to the stream
Source buffer
buffer is null
Object has been disposed
Returns a useful string for debugging. This should not normally be called in actual production code.
Writes a single byte to the current position in the stream.
byte value to write
Object has been disposed
Reads a single byte from the current position in the stream.
The byte at the current position, or -1 if the position is at the end of the stream.
Object has been disposed
Reads a single byte from the specified position in the stream.
The position in the stream to read from
The byte at the current position, or -1 if the position is at the end of the stream.
Object has been disposed
Sets the length of the stream
value is negative or larger than MaxStreamLength
Object has been disposed
Sets the position to the offset from the seek location
How many bytes to move
From where
The new position
Object has been disposed
offset is larger than MaxStreamLength
Invalid seek origin
Attempt to set negative position
Synchronously writes this stream's bytes to the argument stream.
Destination stream
Important: This does a synchronous write, which may not be desired in some situations
stream is null
Synchronously writes this stream's bytes, starting at offset, for count bytes, to the argument stream.
Destination stream
Offset in source
Number of bytes to write
stream is null
Offset is less than 0, or offset + count is beyond this stream's length.
Release the large buffer (either stores it for eventual release or returns it immediately).