Full lesson
scrape-breakout-solana
Scrape a decentralized Scraping Hub is a Chrome extension on Solana that incentivizes users to share bandwidth for AI-optimized web scraping.
What it does
From the project README:
Scrape is a decentralized scraping hub that delivers model-ready data for AI builders while rewarding users for contributing their IPs. Built on Solana, we bypass bot detection, cut costs, and save 4+ hours weekly, enabling scalable, GDPR-compliant data collection.
- Original
- N-45div/scrape-breakout-solana
- SuperStack fork
- Blockchains/scrape-breakout-solana @
4ec330e - Stars
- 2
- Last push (original)
- 2025-05-19
- Languages
- TypeScript, Rust, CSS, MDX, JavaScript
- Detected stack
- AnchorNext.jsweb3.js
- Categories
- AI & AgentsDePIN
- Licence
- MIT
- Homepage
- https://scrapit.fun
Architecture
68 source files. By extension: .rs ×32, .tsx ×24, .ts ×10, .js ×2.
| Top-level directory | Source files |
|---|---|
programs | 32 |
scrape-landing | 19 |
scrape-extension | 13 |
scrape-backend | 2 |
migrations | 1 |
tests | 1 |
On-chain programs: programs/scrape
Dapp: deploying
This fork is queued for its own live dapp on Sol.College, with a Devnet/Mainnet switch. It appears here, at /dapps/scrape-breakout-solana/, once it is deployed and checked on devnet. Until then, run it locally with the commands below.
Key code, explained
Excerpts chosen automatically where the code uses the most Solana concepts, pinned to the fork's commit.
programs/scrape/src/instructions/endpoint_node/close.rs L1–24 @ 4ec330e · MIT · originaluse crate::state::endpoint_node::EndpointNode;use anchor_lang::prelude::*;use crate::error::ErrorCode; #[derive(Accounts)]pub struct CloseEndpointNodeContext<'info> { #[account( mut, close = signer, seeds = [EndpointNode::PREFIX.as_bytes(), signer.key().as_ref()], bump )] pub endpoint_node: Box<Account<'info, EndpointNode>>, #[account(mut)] pub signer: Signer<'info>,} #[inline(never)]pub fn close(ctx: Context<CloseEndpointNodeContext>) -> Result<()> { let endpoint_node = &ctx.accounts.endpoint_node; let signer = &ctx.accounts.signer; // Verify the signer is the owner of the EndpointNode require!(endpoint_node.owner == signer.key(), ErrorCode::UnauthorizedNode);- Line 9
close = signer,: Closes the account at the end of the instruction and refunds its lamports. - Line 10
seeds = [EndpointNode::PREFIX.as_bytes(), signer.key().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match. - Line 11
bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve. - Line 15
pub signer: Signer<'info>,: Requires this account to have signed the transaction. - Line 24
require!(endpoint_node.owner == signer.key(), ErrorCode::UnauthorizedNode);: An explicit validation that aborts the instruction with an error.
programs/scrape/src/instructions/task/create.rs L97–120 @ 4ec330e · MIT · original task.dataset_size = 0; // Update token vault token_vault.bandwidth_paid += reward; msg!("Task created: ID {}, URL: {}", task.id, task.url); Ok(())} #[derive(Accounts)]pub struct CloseTaskContext<'info> { #[account( mut, close = signer, seeds = [Task::PREFIX.as_bytes(), task.owner.as_ref(), task.id.to_le_bytes().as_ref()], bump, constraint = task.owner == signer.key() @ ErrorCode::UnauthorizedNode )] pub task: Box<Account<'info, Task>>, #[account(mut)] pub signer: Signer<'info>,} pub fn close_task(ctx: Context<CloseTaskContext>) -> Result<()> {- Line 102
msg!("Task created: ID {}, URL: {}", task.id, task.url);: Logs to the program log, which is visible in explorers and the test output. - Line 110
close = signer,: Closes the account at the end of the instruction and refunds its lamports. - Line 111
seeds = [Task::PREFIX.as_bytes(), task.owner.as_ref(), task.id.to_le_bytes().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match. - Line 112
bump,: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve. - Line 117
pub signer: Signer<'info>,: Requires this account to have signed the transaction.
programs/scrape/src/instructions/node_registry/init.rs L1–24 @ 4ec330e · MIT · originaluse anchor_lang::prelude::*;use crate::state::node_registry::NodeRegistry; #[derive(Accounts)]pub struct NodeRegistryInitializeContext<'info> { #[account( init, payer = signer, space = NodeRegistry::calculate_size(0), // Start with space for 0 nodes seeds = [NodeRegistry::PREFIX.as_bytes()], bump )] pub node_registry: Account<'info, NodeRegistry>, #[account(mut)] pub signer: Signer<'info>, pub system_program: Program<'info, System>,} pub fn noderegistryinitialize(ctx: Context<NodeRegistryInitializeContext>) -> Result<()> { let node_registry = &mut ctx.accounts.node_registry; node_registry.bump = ctx.bumps.node_registry; node_registry.nodes = Vec::new(); Ok(())}- Line 7
init,: Creates and pays for a new account. Space and payer must be specified. - Line 10
seeds = [NodeRegistry::PREFIX.as_bytes()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match. - Line 11
bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve. - Line 15
pub signer: Signer<'info>,: Requires this account to have signed the transaction.
programs/scrape/src/instructions/dataset/download.rs L13–35 @ 4ec330e · MIT · original #[account( seeds = [Client::PREFIX.as_bytes(), signer.key().as_ref()], bump )] pub client: Box<Account<'info, Client>>, pub signer: Signer<'info>,} #[inline(never)]pub fn download(ctx: Context<DownloadDatasetContext>) -> Result<()> { let task = &ctx.accounts.task; let client = &ctx.accounts.client; let signer = &ctx.accounts.signer; // Verify the task is completed require!(task.status == TaskStatus::Completed, ErrorCode::TaskNotAssigned); // Verify the signer is the client who owns the task require!(task.owner == signer.key(), ErrorCode::UnauthorizedNode); require!(task.owner == client.owner, ErrorCode::UnauthorizedNode); // Verify the dataset is available (IPFS hash exists) require!(task.ipfs_hash.is_some(), ErrorCode::TaskNotAssigned);- Line 14
seeds = [Client::PREFIX.as_bytes(), signer.key().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match. - Line 15
bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve. - Line 18
pub signer: Signer<'info>,: Requires this account to have signed the transaction. - Line 28
require!(task.status == TaskStatus::Completed, ErrorCode::TaskNotAssigned);: An explicit validation that aborts the instruction with an error.
How to run, build and test
These commands come from the repo's own manifests (package.json scripts, Anchor.toml, Cargo, go.mod, Makefile). Check the README for prerequisites and environment variables.
| Directory | Command | Runs |
|---|---|---|
. | npm run lint | prettier */*.js "*/**/*{.js,.ts}" --check |
scrape-backend | npm run start | node dist/index.js |
scrape-backend | npm run build | tsc |
scrape-extension | npm run dev | plasmo dev |
scrape-extension | npm run build | plasmo build |
scrape-landing | npm run dev | next dev --turbo |
scrape-landing | npm run start | next start |
scrape-landing | npm run build | next build |
. | anchor build | compile programs and generate IDLs |
. | anchor test | local validator + test suite |
. | anchor run test | yarn run ts-mocha -p ./tsconfig.json -t 1000000 tests/**/*.ts |
. | cargo build | Rust workspace build |
. | cargo test | Rust tests |
What you could build from it
- Fork the SuperStack copy and extend it within the AI & Agents gaps listed on the idea page.
- Learn the pattern first: AI agents on Solana
- Learn the pattern first: Building and sending transactions from a client
- Learn the pattern first: Blinks and Solana Actions