Skip to main content
The run_profiler.sh script enables processing and profiling of image datasets for the Visual Layer platform. It supports multiple data sources, including local directories, S3 buckets, HTTP/HTTPS URLs, and file lists in .txt, .csv, or .parquet formats. With flexible execution modes and broad data compatibility, the script adapts to a wide range of environments and workflows. The rest of this article outlines script usage syntax and provides detailed explanations for each command-line parameter.

​
Usage

Following is the command-line syntax for running the script:

​
Parameters

Following are the command-line parameters supported by the script:

​
Dataset Path (-p, required)

Specifies the source location of your dataset. Multiple formats are supported:
​
string
required

​
Dataset Name (-n, required)

A human-readable name for your dataset. This name will be used for identification in logs and results.
​
string
required

​
Execution Mode (-e)

Determines how the processing pipeline is executed. default: compose
​
string
default:"compose"

​
Reduce Disk Space (-r)

Enables reduced disk space consumption mode. Useful for large datasets or environments with limited storage. default: false When enabled, activates serve_mode=reduce_disk_space for optimized storage usage.
​
boolean
default:"false"

​
Help (-h)

Displays usage information and example commands.
​
boolean

​
Examples

​
Execution Modes

The script supports two execution modes for processing datasets:

​
Compose Mode (Default)

Recommended for most users. Uses Docker Compose to run the processing pipeline with the Visual Layer API.
API Endpoint: POST http://localhost:2080/api/v1/process Parameters sent to API:

​
Local Mode

Advanced users only. Runs the pipeline directly using Python virtual environment.
  • Executes pipeline.controller.controller module directly
  • Requires local Python virtual environment setup
  • Uses manual flow configuration (MANUAL_FLOW=yes)
  • Configures device settings based on hardware type (CPU/GPU)
Environment Variables Set:
  • MANUAL_FLOW=yes
  • FASTDUP_PRODUCTION=1
  • PREFECT_LOGGING_SETTINGS_PATH=./.vl/prefect-logging.yaml
  • Device-specific settings for CPU mode

​
Data Source Support

​
Local Paths

  • Converts relative paths to absolute paths using realpath
  • For file inputs (lists), copies to .vl/ directory for container access
  • Validates path existence before processing

​
Remote Paths

​
Supported File Formats

  • Image Directories: Any directory containing image files
  • File Lists:
    • .txt: Plain text file with one file path per line
    • .csv: CSV file with file paths
    • .parquet: Parquet file containing file path data

​
Error Handling

The script includes comprehensive error handling:

​
Dependencies

​
Output

​
Compose Mode

  • Success: Displays dataset processing confirmation with response
  • Failure: Shows API error message in red text

​
Local Mode

  • Runs pipeline with full logging output
  • Returns to original shell environment on completion

​
Performance Considerations

​
Reduced Disk Space Mode (-r)

  • Activates serve_mode=reduce_disk_space parameter
  • Optimizes storage usage during processing
  • Recommended for large datasets or limited storage environments

​
Hardware Configuration

  • CPU Mode: Automatically configures all processing devices to use CPU
  • GPU Mode: Uses default GPU acceleration when available

​
Integration

This script integrates with the core components of the Visual Layer platform:

​
Troubleshooting

Path not found errors
  • Verify local paths exist and are accessible
  • Check S3 credentials and permissions for S3 paths
  • Ensure HTTP/HTTPS URLs are accessible