Upload

Upload files to bucket.

iter_files(input_dir: Path, exclude: list[str] | None = None) → Iterator[Path]

Return an iterator of Path

upload(input_dir: str | PathLike, exclude: list[str] | None = None, prescient_client: PrescientClient | None = None, overwrite: bool = True) → None

Upload files from input directory to the location defined by PRESCIENT_UPLOAD_BUCKET

Parameters:

• input_dir (str | os.PathLike) – Input directory containing file(s) to be uploaded. By default will upload all files contained in input directory. This can be an absolute or relative path, the final path component will be included as part of the object key e.g. /path/to/data_dir -> s3://bucket/data_dir/file.txt. When input_dir is a relative path, this should be relative to the current working directory used to execute this function.

• exclude (Optional[list[str]]) – A list of glob patterns to exclude from uploading. For example `exclude=[”.txt”, “.csv”] would skip any matched files that end with a .txt or .csv suffix. If not provided by default all files will be uploaded.

• prescient_client (Optional[PrescientClient]) – A PrescientClient instance. If not provided a default PrescientClient instance will be created.

• overwrite (bool) – Whether to overwrite objects if they already exist. If False, upload is skipped. Useful for continuing an upload that was started previously. Defaults to True.