Configure storage and readers
To read a dataset, Packata needs to find the data and choose a reader for its format. Both the CLI and .NET library can read database tables as well as files. A contract supplies the location and format settings; a database read also needs a compatible connector or provider and connection credentials where required.
An endpoint describes a file location or database connection. An asset’s binding identifies the path or table that holds that dataset. If an asset has several bindings, choose the one you intend to read rather than relying on a default. Loading the contract alone does not open any of its data.
Choose where to load the contract file
The contract file and the data it describes can live in different places. For example, a local contract.yaml can describe a table in a remote PostgreSQL database. Packata loads that file to find the table and connection settings, then connects to the database when you preview its records.
The table below lists where the CLI can load the contract file. It does not restrict the dataset to files:
| Task | Accepted descriptor locations |
|---|---|
| Validate, inspect, diff, assess compatibility, or export metadata | Local files, HTTP(S) URLs, or local and remote ZIP, TAR, TAR.GZ and TAR.BR archives |
| Preview records | Local or HTTP(S) descriptors, or local and remote ZIP, TAR, TAR.GZ and TAR.BR packages containing CSV-family, Excel or Parquet resources |
| Infer a schema | Local or HTTP(S) descriptors, including extensionless files, or local and remote ZIP, TAR, TAR.GZ and TAR.BR archives |
Packata normally detects the contract standard from its contents. If you need to select it explicitly, use --input-format auto, odcs or datapackage. Selecting a format does not expand the locations a command accepts. See the CLI reference for command-specific details.
For example, the Quickstart sample can be previewed with:
packata data preview datapackage.json --asset customers --limit 2
This opens the CSV referenced by the customers asset and displays up to two rows. The data guide explains the output and the additional preview options.
Access remote files and archives from the CLI
The CLI uses some Packata.Storages components internally, but it does not expose the library’s full storage configuration. Available access depends on whether you are loading a contract or reading the data it describes:
| Storage | Load a contract in the CLI | Read dataset records with CLI preview |
|---|---|---|
| Local files | Yes | Yes, for supported formats |
| HTTP(S) | Yes | Yes, for supported formats at accessible HTTP(S) URLs |
| Local or HTTP(S) ZIP, TAR, TAR.GZ and TAR.BR archives | Yes, for validation, inspection, diff, compatibility analysis, metadata export, preview and inference | Yes, for CSV-family, Excel and Parquet resources inside the package |
| Native S3 storage | No | No |
| Native Azure storage | No | No |
All archive formats use the same descriptor selection: exactly one datapackage.json, datapackage.yaml, datapackage.yml, contract.yaml or contract.yml, optionally in a subdirectory. TAR containers materialize bundled entries under the default limits described below. Metadata commands do not parse the datasets or fetch their referenced locations. Schema inference reads selected local, HTTP(S), or archive-relative CSV-family, Excel and Parquet resources in both standards.
Preview resolves relative data paths against the contract’s location. A contract served over HTTP(S) can therefore refer to data files alongside it. A local contract can also refer to an absolute HTTP(S) data URL. Schema inference uses the same resource resolution. Non-seekable Excel/Parquet streams use bounded temporary-file staging; see inference transport and output policy for limits, HTTP authentication/error behavior and standalone-output restrictions.
For ZIP, TAR, TAR.GZ and TAR.BR packages, preview reads CSV-family, Excel and Parquet entries without requiring manual extraction. The archive must contain exactly one descriptor named datapackage.json, datapackage.yaml, datapackage.yml, contract.yaml or contract.yml. Resource paths resolve relative to that descriptor inside the archive, so a descriptor in package/ can reference data/customers.csv at package/data/customers.csv. Select a resource with --asset; a package can contain several file resources. See the data guide for an example.
An S3 or Azure object exposed at an accessible HTTP(S) URL can be read through that URL, for a supported data format. That uses HTTP access, not an S3 or Azure backend. The CLI has no options to register those backends or configure their cloud credentials. Use the .NET storage APIs below when you need that integration.
Stage Excel and Parquet resources
Both data preview and schema infer (including its infer alias) use the same transport-independent staging adapter for ODCS and Data Package bindings. It passes readable seekable inputs through without reading or copying them. Non-seekable Excel and Parquet inputs are copied to unique temporary files before the existing parser runs. CSV-family readers remain streaming. No new format parser or cloud authentication is added.
| Limit | Default | Scope and precedence |
|---|---|---|
--staging-maximum-bytes | 268435456 bytes (256 MiB) | Each non-seekable Excel/Parquet data resource; the invocation value replaces the default |
Inference --maximum-bytes for Excel | 67108864 bytes (64 MiB) | Complete workbook, including seekable files; the smaller workbook/staging limit applies during copying |
| Parquet inference footer limit | 16777216 bytes (16 MiB) | Serialized footer metadata, independent of total transfer size; field/depth limits also remain unchanged |
| HTTP descriptor/archive transfer and non-seekable CSV reads | 256 MiB | Existing fixed transport/read limit; unaffected by the staging option |
| TAR materialization | Separate limits below | TAR entries may already be seekable; staging does not replace archive limits |
Known oversized content is rejected before copying when a length is available. Every copy also counts actual bytes, so missing or understated length metadata cannot bypass the cap. A staging-limit error identifies the staging byte limit (CLI132 in preview); an Excel workbook-limit failure identifies MaximumBytes. Neither changes Parquet’s footer limit. Seekable local Parquet files can exceed the staging cap because they do not need a full copy. Externally compressed Excel/Parquet resources remain unsupported.
packata data preview contract.yaml --asset measurements --limit 5 --staging-maximum-bytes 134217728
packata schema infer contract.yaml --asset measurements --staging-maximum-bytes 134217728
Ordinary non-seekable HTTP access may download the entire file before the parser starts, even for metadata-only Parquet inference. Inference still reads only footer/schema metadata after staging and does not materialize rows. Preview uses its normal reader; the current Parquet reader eagerly materializes all row groups. A preview row limit therefore bounds output rows, not staging transfer or provider memory. HTTP range access is not implemented.
Temporary files use the system temporary directory and unique create-new names. Provide enough free disk space for all concurrent copies plus any archive materialization. Files are opened with delete-on-close, with user-only Unix permissions or inherited Windows temporary-directory permissions. Data, signed query strings and temporary filenames are not included in staging diagnostics. Requests and saved contracts retain original resource identifiers; relative paths are resolved before acquisition.
The CLI owns acquired streams. A staged input closes its upstream stream after copying; the reader/inferrer owns the resulting temporary stream. A seekable input has one owner and is closed once. Success, parser/transfer/limit failure, cancellation and early stopping all release acquired resources. Cleanup failures during acquisition or parser construction do not replace the original failure. Delete-on-close follows platform handle semantics; deletion can be delayed while a handle remains open, and abrupt termination or storage failure can require operating-system cleanup.
In .NET, Packata.Storages.SeekableStaging.AcquireAsync returns a SeekableInput lease. Its default leaveOpen: true borrows caller-owned input. Disposing that lease closes only a staged copy; a seekable original remains open at its original position. With leaveOpen: false, ownership transfers even if acquisition fails. OpenOwnedAsync returns one owned seekable stream for factory integrations; CopyAsync always copies and leaves its input open. These adapters do not assign resource identities or select formats.
Cancellation is checked before and throughout copying and passed into asynchronous I/O. Parquet preview and inference pass it into supported parser operations. Excel and CSV have synchronous parser calls, so cancellation is checked between supported operations and cannot always interrupt a parser already running on staged/local input.
Choose how to read the dataset
CLI preview includes readers for delimited files, inline data, Excel, Parquet and databases. With a database-bound asset, data preview reads table records and schema infer discovers table metadata. Follow the database connection guide for connector setup, runtime credentials and examples of both commands.
In .NET, CSV, TSV, PSV and inline-data readers are built in. Other formats require an additional package and registration. Fixed-width, NDJSON, key-value and web-log readers are available in .NET but are not currently registered for CLI preview.
| Format | Package | Registration |
|---|---|---|
| CSV, TSV, PSV; inline data | Packata.ResourceReaders | Built in |
| Excel | Packata.ResourceReaders.Excel | AddExcel() |
| Parquet | Packata.ResourceReaders.Parquet | AddParquet() |
| Fixed-width | Packata.ResourceReaders.FixedWidth | AddFixedWidth(); also configure layout resolution |
| NDJSON / JSON Lines | Packata.ResourceReaders.Ndjson | AddNdjson() |
| LTSV / logfmt | Packata.ResourceReaders.KeyValue | AddKeyValueReaders() |
| Common / W3C logs | Packata.ResourceReaders.WebLogs | AddWebLogs() |
| Database | Packata.ResourceReaders.Database | AddDatabase(); also configure the driver or connector |
This table describes .NET package registration. The CLI already configures its database reader; you do not need to write .NET registration code to preview a database table from the command line.
Install the packages for the formats your application reads, then add those providers to a reader factory. For example, an application that reads Excel and Parquet uses:
using Packata.ResourceReaders;
using Packata.ResourceReaders.Excel;
using Packata.ResourceReaders.Parquet;
var readers = ResourceReaderFactory.Create(options => options
.AddExcel()
.AddParquet());
This factory can also read the built-in formats. Follow the .NET Quickstart to open an asset through the factory and iterate over its records.
Packata selects a reader using the explicit format, then the media type and file extension. An unknown declared format causes an error instead of silently being read as CSV. For an extensionless delimited file, an explicit delimiter can identify the format.
If the format is recognized but the provider is missing, Packata throws ReaderProviderUnavailableException. Its message names the package and registration method to add.
Read compressed delimited resources
The built-in delimited reader supports CSV, TSV and PSV resources compressed with Brotli (.br) or Gzip (.gz and .gzip), including HTTP(S) URLs with query strings. It preserves the underlying format and delegates decompression to PocketCsvReader.
For paths without a compression suffix, set canonical format metadata explicitly:
var format = new DataFormat("csv", Compression: "brotli");
// "br" is also accepted, as is the compound format new DataFormat("csv.br").
DataFormat is in Packata.Core.Contracts. Its explicit Compression value takes precedence over compression inferred from the compound format, media type or path. A compound format such as csv.br takes precedence over the media type and path. Compression identifiers are case-insensitive and may have a leading dot.
This configuration compresses an individual resource. Whole-package .tar.br archives use the storage configuration below. HTTP Content-Encoding: br describes transport encoding and is separate from resource compression.
Resolve file locations in a .NET application
Resolve relative data paths against the directory containing the descriptor. If you resolve them against the application’s working directory instead, the same contract can stop working when you launch the program elsewhere. The .NET Quickstart includes this path resolution before it opens the reader.
For local directories, HTTP(S), ZIP and TAR archives, S3 or Azure storage, use Packata.Storages. Register location schemes and archive wrappers with DataPackageLocatorBuilder, and register resource handlers with StorageProviderBuilder. Your application supplies cloud credentials.
These are library configuration points. Adding a storage backend to an application does not make its URLs available to every CLI command. If you need custom stream access through the shared contract model, implement IEndpointStreamResolver.
Read a TAR package in .NET
Register the archive suffixes your application accepts. TAR bundles files without compression; .tar.gz and .tar.br apply Gzip or Brotli to the whole TAR stream. All three use the same container implementation. The CLI registers all three TAR variants alongside ZIP for the metadata commands described above; .NET applications can configure their own registrations and limits.
using Packata.Storages;
var limits = new TarArchiveOptions
{
MemoryThresholdBytes = 8L << 20,
MaxEntryBytes = 1L << 30,
MaxTotalBytes = 4L << 30,
MaxArchiveBytes = 5L << 30,
MaxEntries = 10_000
};
var locator = new DataPackageLocatorBuilder()
.Register("file", b => b.UseLocalFileSystem())
.Register(new[] { "http", "https" }, b => b.UseHttp())
.Register("tar", b => b.UseTarArchive(options: limits))
.Register("tar.gz", b => b.UseTarArchive(ArchiveCompression.Gzip, limits))
.Register("tar.br", b => b.UseTarArchive(ArchiveCompression.Brotli, limits))
.Build();
var handle = await locator.LocateAsync(new Uri(Path.GetFullPath("package.tar.gz")));
using var container = handle.Container;
await handle.ValidateAsync();
using var descriptor = await container.OpenAsync(handle.DescriptorPath);
using var customers = await container.OpenAsync("data/customers.csv");
A minimal bundle has this layout:
datapackage.json
data/customers.csv
Its descriptor can be:
{"resources":[{"name":"customers","path":"data/customers.csv"}]}
For example, create the uncompressed bundle with tar -cf package.tar datapackage.json data/customers.csv, or a Gzip bundle with tar -czf package.tar.gz datapackage.json data/customers.csv. Packata reads these archives; it does not provide archive creation APIs.
Suffix matching is case-insensitive and selects the longest registered suffix from the URI path. URL queries and fragments do not affect detection, and HTTP queries are retained when downloading the archive. Standalone .gz and .br files are not TAR packages. HTTP content decoding and compression of individual resources inside an archive are separate from this outer compression layer.
TAR input is consumed once during LocateAsync, including for non-seekable HTTP streams. Entry order does not matter. Small entry contents share the configured memory budget; larger entries spill to randomly named temporary files. With TarArchiveOptions.TemporaryDirectory left null, Packata creates a private Packata/archives directory under the current user’s local application data. It uses an owner-only Windows ACL or Unix mode 0700. Spill files are deleted when released; the empty application directory remains available for later reads. To choose another directory, set TemporaryDirectory to an existing private, writable directory. Publicly accessible directories and symbolic links/reparse points are rejected. On Windows, access rules may also grant access to SYSTEM and local Administrators. If local application data is unavailable, configure a private directory explicitly. Resource streams are independent, so a resource can be opened repeatedly or alongside another resource. Entry paths are case-sensitive, normalize . and separators, and reject parent traversal, absolute paths and duplicates. Directories are skipped; links and other special entry types are rejected.
The options shown above are the defaults. MaxTotalBytes counts file contents; MaxArchiveBytes bounds the complete decoded TAR stream, including headers, metadata and padding. Initialization closes the input stream and cleans up stored entries if reading fails. Dispose the returned container to release storage; close its resource streams before disposal to ensure temporary files are removed promptly. ZIP wrappers also run through LocateAsync when registered with UseZipArchive().
Patched Brotli runtimes
Brotli decoding uses .NET’s built-in BrotliStream, rather than a separate Brotli NuGet package or PocketCsvReader. Use .NET 8.0.33, 9.0.22, 10.0.14 or later patched releases. These updates fix the Brotli decoder vulnerability CVE-2026-111497. Packata’s build runtime versions and CI SDKs select these patched releases. Applications consuming the library must also deploy a patched runtime: update and restart framework-dependent applications; rebuild and republish self-contained applications and container images. Installing the library alone does not patch the runtime of a consuming application.
Configure how values are parsed
A delimited file’s dialect describes its delimiter, quote character, line terminator and header. Use IDelimitedDialectResolver when those settings come from a catalog or sidecar file. A fixed-width reader instead needs a layout resolver to supply column widths. The provider packages expose their own options; see the .NET reference for entry points.
Check the chosen provider’s support for encoding, compression, repeated headers, field matching, missing values and numeric conventions. A contract can preserve a setting even when a particular reader cannot apply it.
Temporal parsing uses the declared format. An offset-bearing strptime format preserves the offset; a value parsed with an offset-free format remains offset-free. The temporal concepts explain these distinctions.
Add a custom reader
Implement IDataEndpointReaderProvider and register it with AddProvider. Custom providers are tried before built-ins, in registration order. Registering the same provider type twice is rejected, which helps catch accidental duplicate configuration.
Release resources when a read finishes
Dispose the streams, readers and containers your application owns, including when reading fails or stops early. A database connection service can be shared across readers, but each operation still owns its connection. The database guide explains which resources belong to the service and which belong to the caller.