Sol.College

Full lesson

scrape-breakout-solana

Scrape a decentralized Scraping Hub is a Chrome extension on Solana that incentivizes users to share bandwidth for AI-optimized web scraping.

What it does

From the project README:

Scrape is a decentralized scraping hub that delivers model-ready data for AI builders while rewarding users for contributing their IPs. Built on Solana, we bypass bot detection, cut costs, and save 4+ hours weekly, enabling scalable, GDPR-compliant data collection.
Original
N-45div/scrape-breakout-solana
SuperStack fork
Blockchains/scrape-breakout-solana @ 4ec330e
Stars
2
Last push (original)
2025-05-19
Languages
TypeScript, Rust, CSS, MDX, JavaScript
Detected stack
AnchorNext.jsweb3.js
Categories
AI & AgentsDePIN
Licence
MIT
Homepage
https://scrapit.fun

Architecture

68 source files. By extension: .rs ×32, .tsx ×24, .ts ×10, .js ×2.

Top-level directorySource files
programs32
scrape-landing19
scrape-extension13
scrape-backend2
migrations1
tests1

On-chain programs: programs/scrape

Dapp: deploying

This fork is queued for its own live dapp on Sol.College, with a Devnet/Mainnet switch. It appears here, at /dapps/scrape-breakout-solana/, once it is deployed and checked on devnet. Until then, run it locally with the commands below.

Key code, explained

Excerpts chosen automatically where the code uses the most Solana concepts, pinned to the fork's commit.

N-45div/scrape-breakout-solana · programs/scrape/src/instructions/endpoint_node/close.rs L1–24 @ 4ec330e · MIT · original
use crate::state::endpoint_node::EndpointNode;use anchor_lang::prelude::*;use crate::error::ErrorCode; #[derive(Accounts)]pub struct CloseEndpointNodeContext<'info> {    #[account(        mut,        close = signer,        seeds = [EndpointNode::PREFIX.as_bytes(), signer.key().as_ref()],        bump    )]    pub endpoint_node: Box<Account<'info, EndpointNode>>,    #[account(mut)]    pub signer: Signer<'info>,} #[inline(never)]pub fn close(ctx: Context<CloseEndpointNodeContext>) -> Result<()> {    let endpoint_node = &ctx.accounts.endpoint_node;    let signer = &ctx.accounts.signer;     // Verify the signer is the owner of the EndpointNode    require!(endpoint_node.owner == signer.key(), ErrorCode::UnauthorizedNode);
  1. Line 9 close = signer,: Closes the account at the end of the instruction and refunds its lamports.
  2. Line 10 seeds = [EndpointNode::PREFIX.as_bytes(), signer.key().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match.
  3. Line 11 bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve.
  4. Line 15 pub signer: Signer<'info>,: Requires this account to have signed the transaction.
  5. Line 24 require!(endpoint_node.owner == signer.key(), ErrorCode::UnauthorizedNode);: An explicit validation that aborts the instruction with an error.
N-45div/scrape-breakout-solana · programs/scrape/src/instructions/task/create.rs L97–120 @ 4ec330e · MIT · original
    task.dataset_size = 0;     // Update token vault    token_vault.bandwidth_paid += reward;     msg!("Task created: ID {}, URL: {}", task.id, task.url);    Ok(())} #[derive(Accounts)]pub struct CloseTaskContext<'info> {    #[account(        mut,        close = signer,        seeds = [Task::PREFIX.as_bytes(), task.owner.as_ref(), task.id.to_le_bytes().as_ref()],        bump,        constraint = task.owner == signer.key() @ ErrorCode::UnauthorizedNode    )]    pub task: Box<Account<'info, Task>>,    #[account(mut)]    pub signer: Signer<'info>,} pub fn close_task(ctx: Context<CloseTaskContext>) -> Result<()> {
  1. Line 102 msg!("Task created: ID {}, URL: {}", task.id, task.url);: Logs to the program log, which is visible in explorers and the test output.
  2. Line 110 close = signer,: Closes the account at the end of the instruction and refunds its lamports.
  3. Line 111 seeds = [Task::PREFIX.as_bytes(), task.owner.as_ref(), task.id.to_le_bytes().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match.
  4. Line 112 bump,: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve.
  5. Line 117 pub signer: Signer<'info>,: Requires this account to have signed the transaction.
N-45div/scrape-breakout-solana · programs/scrape/src/instructions/node_registry/init.rs L1–24 @ 4ec330e · MIT · original
use anchor_lang::prelude::*;use crate::state::node_registry::NodeRegistry; #[derive(Accounts)]pub struct NodeRegistryInitializeContext<'info> {    #[account(        init,        payer = signer,        space = NodeRegistry::calculate_size(0), // Start with space for 0 nodes        seeds = [NodeRegistry::PREFIX.as_bytes()],        bump    )]    pub node_registry: Account<'info, NodeRegistry>,    #[account(mut)]    pub signer: Signer<'info>,    pub system_program: Program<'info, System>,} pub fn noderegistryinitialize(ctx: Context<NodeRegistryInitializeContext>) -> Result<()> {    let node_registry = &mut ctx.accounts.node_registry;    node_registry.bump = ctx.bumps.node_registry;    node_registry.nodes = Vec::new();    Ok(())}
  1. Line 7 init,: Creates and pays for a new account. Space and payer must be specified.
  2. Line 10 seeds = [NodeRegistry::PREFIX.as_bytes()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match.
  3. Line 11 bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve.
  4. Line 15 pub signer: Signer<'info>,: Requires this account to have signed the transaction.
N-45div/scrape-breakout-solana · programs/scrape/src/instructions/dataset/download.rs L13–35 @ 4ec330e · MIT · original
    #[account(        seeds = [Client::PREFIX.as_bytes(), signer.key().as_ref()],        bump    )]    pub client: Box<Account<'info, Client>>,    pub signer: Signer<'info>,} #[inline(never)]pub fn download(ctx: Context<DownloadDatasetContext>) -> Result<()> {    let task = &ctx.accounts.task;    let client = &ctx.accounts.client;    let signer = &ctx.accounts.signer;     // Verify the task is completed    require!(task.status == TaskStatus::Completed, ErrorCode::TaskNotAssigned);     // Verify the signer is the client who owns the task    require!(task.owner == signer.key(), ErrorCode::UnauthorizedNode);    require!(task.owner == client.owner, ErrorCode::UnauthorizedNode);     // Verify the dataset is available (IPFS hash exists)    require!(task.ipfs_hash.is_some(), ErrorCode::TaskNotAssigned);
  1. Line 14 seeds = [Client::PREFIX.as_bytes(), signer.key().as_ref()],: Declares the PDA seeds. Anchor re-derives the address and rejects the account if it doesn't match.
  2. Line 15 bump: Uses the bump seed, the extra byte that pushes the derived address off the ed25519 curve.
  3. Line 18 pub signer: Signer<'info>,: Requires this account to have signed the transaction.
  4. Line 28 require!(task.status == TaskStatus::Completed, ErrorCode::TaskNotAssigned);: An explicit validation that aborts the instruction with an error.

How to run, build and test

These commands come from the repo's own manifests (package.json scripts, Anchor.toml, Cargo, go.mod, Makefile). Check the README for prerequisites and environment variables.

DirectoryCommandRuns
.npm run lintprettier */*.js "*/**/*{.js,.ts}" --check
scrape-backendnpm run startnode dist/index.js
scrape-backendnpm run buildtsc
scrape-extensionnpm run devplasmo dev
scrape-extensionnpm run buildplasmo build
scrape-landingnpm run devnext dev --turbo
scrape-landingnpm run startnext start
scrape-landingnpm run buildnext build
.anchor buildcompile programs and generate IDLs
.anchor testlocal validator + test suite
.anchor run testyarn run ts-mocha -p ./tsconfig.json -t 1000000 tests/**/*.ts
.cargo buildRust workspace build
.cargo testRust tests

What you could build from it

Used in lessons