Skip to content

PyTorch

rddac.pytorch.RDDACDataset is a torch.utils.data.IterableDataset over a Croissant view. It is available only when the [torch] extra is installed (see Installation).

RDDACDataset

RDDACDataset

Bases: DDACSDataset

Streaming PyTorch dataset for a single RDDAC Croissant view.

Yields a dict[str, numpy.ndarray] per experiment. Field selection is derived from the Croissant view + field-map; sharding across DataLoader workers and DDP ranks is decided inside __iter__, so the same instance works under num_workers=0, num_workers=N and DDP.

Views must source field-map (HDF5) fields only; for views that include process-parameters metadata columns use rddac.streaming.iter_view, or build the view with with_metadata=False.

Parameters:

Name Type Description Default
view str

Name of the RecordSet to stream (e.g. "force-curve").

 required
source str | Path | None

Override the manifest URL / path.

 None
data_dir str | Path | None

Local data directory (default "./data"). Pass None to skip local-file discovery.

 DEFAULT_DATA_DIR
dataset

A pre-loaded mlcroissant.Dataset (e.g. one mutated by rddac.add_view) — the way to stream a custom view.

 None
sim_ids list[int] | None

Explicit allowlist of experiment ids (name kept for drop-in DDACS compatibility). Requested ids that cannot be served warn via rddac.streaming.MissingDataWarning.

 None
where Callable[[Series], bool] | None

Predicate applied to each process_parameters.csv row before any zip is opened.

 None
shuffle bool

Per-shard seeded shuffle; call set_epoch between epochs.

 False
seed int

Base seed for the per-shard shuffle.

 0
Source code in rddac/pytorch.py 
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
class RDDACDataset(DDACSDataset):
    """Streaming PyTorch dataset for a single RDDAC Croissant view.

    Yields a `dict[str, numpy.ndarray]` per experiment. Field selection is
    derived from the Croissant view + field-map; sharding across DataLoader
    workers and DDP ranks is decided inside `__iter__`, so the same instance
    works under `num_workers=0`, `num_workers=N` and DDP.

    Views must source `field-map` (HDF5) fields only; for views that include
    `process-parameters` metadata columns use `rddac.streaming.iter_view`, or
    build the view with `with_metadata=False`.

    Args:
        view: Name of the RecordSet to stream (e.g. "force-curve").
        source: Override the manifest URL / path.
        data_dir: Local data directory (default "./data"). Pass `None` to skip
            local-file discovery.
        dataset: A pre-loaded `mlcroissant.Dataset` (e.g. one mutated by
            `rddac.add_view`) — the way to stream a custom view.
        sim_ids: Explicit allowlist of experiment ids (name kept for drop-in
            DDACS compatibility). Requested ids that cannot be served warn via
            `rddac.streaming.MissingDataWarning`.
        where: Predicate applied to each `process_parameters.csv` row before
            any zip is opened.
        shuffle: Per-shard seeded shuffle; call `set_epoch` between epochs.
        seed: Base seed for the per-shard shuffle.
    """

    def __init__(
        self,
        view: str,
        source: str | Path | None = None,
        data_dir: str | Path | None = DEFAULT_DATA_DIR,
        dataset=None,
        sim_ids: list[int] | None = None,
        where: Callable[[pd.Series], bool] | None = None,
        shuffle: bool = False,
        seed: int = 0,
    ):
        super().__init__(
            view,
            source=source,
            data_dir=data_dir,
            dataset=dataset,
            sim_ids=sim_ids,
            where=where,
            shuffle=shuffle,
            seed=seed,
            spec=RDDAC_SPEC,
        )

__init__(view: str, source: str | Path | None = None, data_dir: str | Path | None = DEFAULT_DATA_DIR, dataset=None, sim_ids: list[int] | None = None, where: Callable[[pd.Series], bool] | None = None, shuffle: bool = False, seed: int = 0)

Source code in rddac/pytorch.py 
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def __init__(
    self,
    view: str,
    source: str | Path | None = None,
    data_dir: str | Path | None = DEFAULT_DATA_DIR,
    dataset=None,
    sim_ids: list[int] | None = None,
    where: Callable[[pd.Series], bool] | None = None,
    shuffle: bool = False,
    seed: int = 0,
):
    super().__init__(
        view,
        source=source,
        data_dir=data_dir,
        dataset=dataset,
        sim_ids=sim_ids,
        where=where,
        shuffle=shuffle,
        seed=seed,
        spec=RDDAC_SPEC,
    )