Data Pipeline Toolkit

The problem

jq is the standard tool for reshaping JSON, and it's also the subject of a well-known "I can't stand jq" complaint thread — its syntax turns unreadable fast once you go past simple field access. Address normalization has the opposite problem: libpostal is the library everyone reaches for, and its own package page admits "installation is complex," especially across conda/pyenv/system Python — people write entire tutorials just to get it installed before writing any real logic. And "I just want to run SQL against this CSV" is common enough that at least half a dozen tools exist solely to avoid standing up a database for one query. This toolkit is the boring-but-essential middle layer — a dozen narrow, single-purpose Actors for exactly these jobs — without the install step or the syntax cliff.

Who it's for

• Pipeline builders who need a transformation step between two Actors without writing custom code
• Data engineers cleaning and normalizing scraped data (addresses, phone numbers, ZIP codes)
• Anyone chaining Apify Actors who needs jq, SQL, or regex in the middle of a run
• Developers who need a quick UUID, password, or thumbnail generated on demand

What's inside

• jq HelperRun jq programs over inline JSON or a linked Apify dataset.
• DuckDB HelperRun a DuckDB SQL query over CSV, JSON, Parquet, Excel, or SQLite input.
• Regex HelperApply named regular expressions to a list of strings and extract structured matches.

• URL WranglerJoin relative URLs, decompose URLs into a node/edge tree, strip or replace query params.
• ZIP Code HelperResolve US ZIP codes into city, state, county, and other location data.
• Postal Address NormaliserParse and normalize arbitrary postal address strings with libpostal.
• Phone Number WranglerValidate, format, and parse phone numbers with Google's libphonenumber.

• UUID GeneratorBulk UUIDs (v1, v3, v4, v5, v7) on demand.
• Secure Password & Passphrase GeneratorCryptographically secure passwords and diceware passphrases per NIST guidance.
• Thumbnail MakerGenerate thumbnails from a list of image URLs.

• Katana Web CrawlerCrawl websites with ProjectDiscovery Katana, streamed as structured JSONL.
• ProjectDiscovery NotifyStream text or dataset records to Slack, Discord, Telegram, Email, and more.

Why run them together

None of these Actors are meant to run alone — each is a five-second step you'd otherwise write as throwaway code, or lose an afternoon installing a native dependency for. Drop jq or DuckDB after a scraper to reshape its output, normalize addresses and phone numbers before they hit your database, generate the IDs a downstream system needs, and fire a Notify alert when the whole pipeline finishes. The value is in composition, not any single Actor.

FAQ